`llmman serve` never checks credentials. Apus still needs a non-empty `apiKey` on the provider entry so the provider counts as configured. This page uses `LLMMAN_API_KEY=llmman-local` with `apiKey: "${LLMMAN_API_KEY}"`, mirroring the `OLLAMA_API_KEY=ollama-local` convention. A literal `apiKey: "llmman-local"` works too.

For `llmman.hybrid/...` and `llmman.provider/...` refs, llmman forwards the bearer Apus presents to the hosted provider as that provider's API key. The one exception is the literal placeholder `llmman`, which tells the daemon to use its own key. See [Hybrid inference](#hybrid-inference) for both patterns. A marker such as `llmman-local` would be sent to the hosted provider and rejected there.

A daemon started with `LLMMAN_HOST=0.0.0.0` in its environment binds every interface with no auth. Point Apus at a LAN host only inside a network you trust; there is no credential Apus can send that llmman would enforce. A daemon reachable off loopback also refuses to spend its own hosted-provider key for callers that presented none.

`${LLMMAN_API_KEY}` resolves from the Gateway process environment or `~/.apus/.env`. If the variable is missing, Apus logs a config warning and treats the provider as unavailable, so put the export where the Gateway can read it. See [Environment](/help/environment).

```bash
llmman pull qwen3.8
llmman serve
echo 'LLMMAN_API_KEY=llmman-local' >> ~/.apus/.env
apus models set llmman/qwen3.8
```

Use the [Full config example](#full-config-example) for the provider entry, and set `LLMMAN_CONTEXT_LENGTH=65536` in the daemon's environment if you want the server context to match it exactly.

The [Hybrid config](#hybrid-config) above: `llmman.hybrid/qwen3.8,openai/gpt-5.6-luna` as `primary`, `LLMMAN_API_KEY` set to the OpenAI key, `LLMMAN_CONTEXT_LENGTH` matching the pair's `contextWindow`. Add `openai/gpt-5.6-luna` to `fallbacks` so a stopped daemon does not block replies.

Local models served through a custom `openai-completions` provider use structured [Tool Search](/tools/tool-search) automatically when unset. The explicit setting below pins that surface, keeping optional capabilities available while loading their schemas only when needed. Cap the context to what the host can run with `LLMMAN_CONTEXT_LENGTH=32768` in the daemon's environment:

```json5
{
  agents: {
    defaults: {
      model: { primary: "llmman/qwen3.8" },
    },
  },
  tools: {
    toolSearch: { mode: "tools" },
  },
  models: {
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [
          {
            id: "qwen3.8",
            name: "Qwen3.8 (llmman)",
            reasoning: true,
            input: ["text", "image"],
            contextWindow: 32768,
            contextTokens: 32768,
            maxTokens: 4096,
          },
        ],
      },
    },
  },
}
```

Use `compat.supportsTools: false` only when the model or server reliably fails on tool schemas; it disables tool use entirely. For a deliberately narrower agent, prefer `tools.profile` or a per-agent tool policy.

Custom provider ids when running more than one daemon; each gets its own host, models, and timeout:

```json5
{
  models: {
    providers: {
      "llmman-fast": {
        baseUrl: "http://mini.local:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [{ id: "qwen3.5:9b", name: "qwen3.5:9b", input: ["text"], contextWindow: 32768 }],
      },
      "llmman-large": {
        baseUrl: "http://gpu-box.local:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        timeoutSeconds: 420,
        models: [{ id: "qwen3.8", name: "qwen3.8", reasoning: true, input: ["text", "image"], contextWindow: 131072 }],
      },
    },
  },
  agents: {
    defaults: {
      model: {
        primary: "llmman-fast/qwen3.5:9b",
        fallbacks: ["llmman-large/qwen3.8"],
      },
    },
  },
}
```

llmman can also pool several daemons itself: `llmman config set aggregation.peers <host>,<host>` (or `LLMMAN_PEERS`) makes one daemon forward to peers, so Apus sees a single endpoint whose model list spans the group.

Model ids are whatever llmman resolves. Pull with the full reference and use the same string as the Apus model id:

```bash
llmman pull hf.co/unsloth/Qwen3.5-0.8B-GGUF
llmman pull ghcr.io/myorg/private-model:v1
```

```json5
{
  models: {
    providers: {
      llmman: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
        api: "openai-completions",
        models: [
          { id: "hf.co/unsloth/Qwen3.5-0.8B-GGUF", name: "Qwen3.5 0.8B", input: ["text"] },
          { id: "ghcr.io/myorg/private-model:v1", name: "Private model", input: ["text"] },
        ],
      },
    },
  },
}
```

The agent ref is then `llmman/hf.co/unsloth/Qwen3.5-0.8B-GGUF`. Run `llmman login <registry>` first for private registries.

`LLMMAN_CONTEXT_LENGTH` is the server-side context (there is no flag). Semantics by backend:

- **llama-server (GGUF):** set, it is passed as `--ctx-size` for generation models; `0` means the trained context. Unset, llmman uses `262144` or the model's trained context if smaller, and on out-of-memory retries with the context halved down to a 16,384 floor.
- **vLLM (safetensors):** a positive value becomes `--max-model-len`; unset uses the vLLM default.
- **mlx_lm.server:** not forwarded.

`LLMMAN_NUM_PARALLEL` scales `--ctx-size` up by that factor so each slot keeps the full context.

On the Apus side, `contextWindow` declares the model's window and `contextTokens` caps active input. Keep `contextWindow` at or below the server value; Apus derives compaction and preflight thresholds from it. Apus's `contextWindow` does not change llmman's hybrid byte budget; configure the daemon separately as described in [Hybrid config](#hybrid-config).

```json5
{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "qwen3.8",
            contextWindow: 65536,
            contextTokens: 49152,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}
```

Qwen3.8 thinks by default; llmman returns the reasoning as `reasoning_content`, which Apus's `openai-completions` adapter separates from the final text. Requests are proxied to `llama-server`, so `chat_template_kwargs` passes through. To turn thinking off for agent turns with a local Qwen model:

```json5
{
  agents: {
    defaults: {
      models: {
        "llmman/qwen3.8": {
          params: {
            chat_template_kwargs: { enable_thinking: false },
          },
        },
      },
    },
  },
}
```

For per-run or session control, declare the local Qwen model's thinking format:

```json5
{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "Qwen3.8 (llmman)",
            reasoning: true,
            input: ["text", "image"],
            compat: { thinkingFormat: "qwen-chat-template" },
          },
        ],
      },
    },
  },
}
```

With this declaration, `apus agent --model llmman/qwen3.8 --thinking off`, `/think off`, and `apus infer model run --local --model llmman/qwen3.8 --thinking off --prompt "Reply with exactly: pong" --json` map the thinking setting to `chat_template_kwargs.enable_thinking`. Without it, the generic proxy defaults do not send this control or `reasoning_effort`.

The lean `infer model run` path does not read the agent-level `params` recipe above; use the compatibility declaration and `--thinking off` for that check. Do not combine a fixed `enable_thinking` agent param with per-run control, since the fixed param overrides the generated value. Apply Qwen-specific controls to a hybrid ref only if both its local and hosted backends accept them.

Models load on their first request, each in its own backend subprocess, and unload after five idle minutes. Tune with:

| Variable                   | Meaning                                                                                  |
| -------------------------- | ---------------------------------------------------------------------------------------- |
| `LLMMAN_KEEP_ALIVE`        | Idle unload timer (default `5m`); `0` unloads right after each request                    |
| `LLMMAN_MAX_LOADED_MODELS` | Cap on concurrently loaded models; idle ones are evicted LRU, busy ones return `503`      |
| `LLMMAN_MAX_QUEUE`         | Pending-request cap before `503` (default `512`)                                          |
| `LLMMAN_LOAD_TIMEOUT`      | Load stall timeout (default `10m`)                                                       |

`llmman ps` shows loaded models with their context and expiry; `llmman stop <model>` unloads one now. A first request after startup or an idle unload pays the load cost, so set `timeoutSeconds` on the provider and raise `LLMMAN_KEEP_ALIVE` to keep the daemon warm for chat surfaces.

llmman checks CUDA, ROCm, Vulkan (Linux/Windows) or Metal (macOS) and downloads a matching `llama-server` release if none is on `PATH`. Override with `LLMMAN_LLM_LIBRARY`: `cpu`, `cuda`, `cuda13`, `rocm`, `vulkan`, or `metal`. Other knobs: `LLMMAN_FLASH_ATTENTION` (`on`/`off`/`auto`), `LLMMAN_KV_CACHE_TYPE` (`f16`, `q8_0`, `q4_0`), `LLMMAN_SCHED_SPREAD` for multi-GPU layer splitting, `LLMMAN_IGPU_ENABLE` to count integrated GPUs. On Linux, `llmman serve --ociman docker|podman` runs `llama-server` from the `ghcr.io/ggml-org/llama.cpp` images instead of a local binary. `LLMMAN_DEBUG=1` prints the check result.

llmman serves `/v1/embeddings` for GGUF embedding models, so [memory search](/concepts/memory) can use it through the generic `openai-compatible` embedding provider:

```bash
llmman pull embeddinggemma
```

```json5
{
  memory: {
    search: {
      provider: "openai-compatible",
      model: "embeddinggemma",
      remote: {
        baseUrl: "http://127.0.0.1:17434/v1",
        apiKey: "${LLMMAN_API_KEY}",
      },
    },
  },
}
```

Embedding models are capped to their trained context regardless of `LLMMAN_CONTEXT_LENGTH`. See [Memory config](/reference/memory-config#remote-endpoint-config) for the remaining fields.

llmman also implements Ollama's native `/api/chat`, `/api/tags`, `/api/show`, and `/api/ps`, and `OLLAMA_HOST=127.0.0.1:17434 ollama run <model>` works against it. Prefer `api: "openai-completions"` from Apus anyway: llmman's `/api/show` reports only `completion` and `vision` capabilities, so the bundled Ollama plugin's discovery would mark every llmman model `compat.supportsTools: false`. If you do point the Ollama plugin at `http://127.0.0.1:17434` (no `/v1`), list models explicitly instead of relying on discovery.

llmman forwards message content and tool schemas to the backend without normalizing them, so compatibility depends on the selected engine and model. Structured content parts (text + image) and Apus's full tool schema work with `qwen3.8` on `llama-server`. If a different backend or model rejects them:

- `messages[].content: invalid type: sequence, expected a string` → set `compat.requiresStringContent: true` on the model entry. Apus then flattens pure text content parts into plain strings.
- `400 JSON schema conversion failed` → `llama-server` could not compile a tool schema into its grammar subset. Update Apus first; if a third-party tool or MCP server contributes the offending schema, disable it for that agent, and use `compat.supportsTools: false` only as a last resort.

```json5
{
  models: {
    providers: {
      llmman: {
        models: [
          {
            id: "qwen3.8",
            name: "qwen3.8",
            compat: {
              requiresStringContent: true,
            },
          },
        ],
      },
    },
  },
}
```

Because llmman is a non-native `openai-completions` endpoint, Apus treats it as a proxy route: no `service_tier`, no Responses `store`, no prompt-cache hints, no OpenAI reasoning-compat payload shaping, no hidden Apus attribution headers, and `compat.supportsDeveloperRole` is forced to `false`. Vendor-specific fields can be merged into the request body with `agents.defaults.models["llmman/<model>"].params.extra_body`.

Local llmman models are free, so set all costs to `0`. The hosted half of a hybrid or `llmman.provider/...` ref is billed by that provider; `llmman list --provider openai` shows its per-million-token prices.

`llmman serve` is not running or is not reachable at the configured address. The default is `127.0.0.1:17434`; if you set `LLMMAN_HOST`, update the Apus `baseUrl` and `healthUrl` to match.

```bash
llmman serve
curl http://127.0.0.1:17434/api/version
```

`models.providers.llmman.apiKey: *** env var "LLMMAN_API_KEY"` in the config warnings means the substitution found no value. Add `LLMMAN_API_KEY=llmman-local` to `~/.apus/.env`, or replace `"${LLMMAN_API_KEY}"` with the literal `"llmman-local"`.

The model is not pulled, or the id in config does not match what llmman resolves. Compare against `llmman list`; a short name maps to `docker.io/ai/<name>:latest`, and an `owner/repo` name maps to `hf.co/owner/repo`.

```bash
llmman pull qwen3.8
llmman list
```

Large models can take minutes to load, especially on the first request after an idle unload. Raise `models.providers.llmman.timeoutSeconds`, warm the model with a first `apus infer model run`, and consider a longer `LLMMAN_KEEP_ALIVE` on the daemon.

llmman routed the request to the hosted half and found no usable key. Either the bearer Apus sent was a marker (`llmman-local`) rather than a real key, or you used the `llmman` placeholder without giving the daemon its own `OPENAI_API_KEY`, or the daemon is bound off loopback and refuses to spend its own key. See [Hosted-provider key](#hosted-provider-key).

Check the daemon log; every routed request records the side and the reason. Routing is by request body size against `LLMMAN_CONTEXT_LENGTH` (4 bytes per token) unless `x-llmman-route` is set. Lower `LLMMAN_HYBRID_LOCAL_BYTES` to overflow sooner, raise it to stay local longer, or pin a side with a provider `headers` entry as in [Pinning a side](#pinning-a-side).

Both checks are tool-free, so `compat.supportsTools` cannot change this failure. Check the configured base URL, model id, and `LLMMAN_API_KEY`, inspect the daemon and backend logs, and compare the two request payloads.

The agent turn adds a larger prompt and tool schemas. A `400 JSON schema conversion failed` is `llama-server` rejecting a tool schema; update Apus and check third-party tools or MCP servers. Otherwise enable [Tool Search](/tools/tool-search) to defer schemas, confirm the server's actual context allocation, and use `compat.supportsTools: false` only as a last resort. See [Smaller or stricter backends](/gateway/local-models#smaller-or-stricter-backends).

If schema errors are gone but the spawned `llama-server` still crashes on larger turns, treat it as an upstream `llama.cpp` or model limitation. Lower `LLMMAN_CONTEXT_LENGTH`, set `LLMMAN_KV_CACHE_TYPE=q8_0` to reduce memory, or switch the backend or model.

Confirm the model's chat template supports tool calling and that the request reached `/v1/chat/completions` (not the Ollama or Anthropic surfaces via a proxy). If the model only calls tools when forced, set `params.extra_body.tool_choice: "required"` on that model ref as described in [Local models](/gateway/local-models#other-openai-compatible-local-proxies).