The llama-cpp plugin provides one llama-cpp model provider. Apus can manage a local llama-server or connect to one that you operate. Both choices use llama-cpp/<model> references and the OpenAI-compatible transport.

apus plugins install @openclaw/llama-cpp-provider
apus onboard

Choose server ownership

models.providers.llama-cpp.localService is the ownership discriminator. If it exists, Apus manages the process. Without it, baseUrl identifies an existing endpoint. Switching choices rewrites ownership-specific state on the same provider. It never creates another provider namespace.

Managed local server

Choose Managed local server when Apus should install, start, and stop the server. Setup reads the Gateway host's hardware and recommends a model from its available memory, GPU capability, and free disk space. A browser connected to a remote Gateway installs and runs the model on that Gateway, not on the browser's computer.

Review the named host, execution backend, model, and download size, then confirm the download. Setup verifies pinned model files and the llama.cpp build, prepares a loopback endpoint, and checks inference before saving the new default. Guided activation also asks the model to read a temporary file through an Apus tool and return its contents. The tool check uses an isolated workspace without your agent's bootstrap instructions. A plain text reply alone does not pass that check. Each verification check has a 90-second deadline. Changing agents.defaults.timeoutSeconds does not extend setup verification. Failures identify whether the response check or tool-use check timed out.

If the configured server executable is missing from Apus's current managed installation, chat and local embeddings restore the same verified build before starting it. Custom server paths and installations from older releases still require manual repair or rerunning setup.

Managed local models automatically use structured Tool Search unless you have explicitly configured it. Optional capabilities remain available. Their schemas load as needed, reducing the input the model must process before replying. Setup does not enable lean mode. Normal chats still include your agent's instructions. On CPU-only hosts, the first reply can take several minutes even after setup verification succeeds.

Model recommendations

Setup prefers these text-only recipes as memory permits. Each uses a 65,536-token context and supports tools:

Qwen3.8 is the first recommendation when its memory budget fits. Muse has a smaller context-cache budget and can fit a 24 GiB NVIDIA card where Qwen3.8 does not. CPU recommendations stop at Qwen3.5 9B. Gemma 4 E2B, E4B, and 26B A4B remain in the catalog for existing routes and cached downloads. The recommendation order is a product default, not a claim that one model wins every task.

These are selection floors, not guaranteed fit or speed. Setup reserves memory for the operating system, context cache, runtime, and default embedding model. It accounts for current memory pressure and container memory limits, and may recommend a smaller model when RAM, GPU memory, or disk space is limited. Separate NVIDIA cards are not added together to assume a model will fit. Existing Gemma 4 E4B configurations and cached custom models remain supported.

The chat download also includes your configured local embedding model, or EmbeddingGemma by default (about 0.3 GB). Leave additional disk space for the runtime and download staging. Setup checks this before offering a recommended model. When the cache and runtime use independent volumes, setup checks each volume's free space separately. Shared storage pools and volumes whose independence cannot be established use a combined reserve. Custom embedding models may need more memory and disk space than these budgets.

Execution backends

The verified Windows CUDA build requires NVIDIA driver 551.78 or newer and compute capability 5.0 or newer. Setup checks the driver and installed runtime's device discovery. When an NVIDIA GPU has no compatible managed CUDA build, setup explains the limitation and names CPU execution in the confirmation. For other acceleration backends, run a compatible server yourself and choose Existing llama-server.

Runtime validation allows up to two minutes per executable version check so macOS security assessment and Metal initialization can finish after installation or when reusing a runtime. Setup remains cancellable during validation.

The verified macOS builds require macOS 13.3 or later, and setup stops before downloading on older releases. To keep managed chat and local embeddings there, build llama-server on that Mac and set models.providers.llama-cpp.localService.command to its absolute path; see Local model services.

If no recommendation fits, setup explains whether to free memory, free disk space, or fix cache-directory permissions. Cancelling or failing guided verification leaves the previous default model selected. A setup candidate has its own server preset, so verification does not rewrite an existing managed server's preset. Downloaded files may remain cached for a retry. Setup verifies and reuses cached recommendations, and charges disk space only for missing model and runtime files.

Managed router presets retain configured chat models in deterministic order and remove model sections outside that inventory. Chat and embedding preparation update their owned settings while preserving the header, [*] defaults, comments, and additional options on retained models. Embedding-only setup uses a fresh preset.

Managed embeddings default to one server slot (parallel = 1), a 2,048-token context, and a 2,048-token physical batch (ubatch-size). Memory indexing reserves four positions for tokenizer overhead and the server's context boundary. These defaults also apply to custom GGUFs and legacy EmbeddingGemma cache paths. One slot bounds llama.cpp's large embedding output buffer to one input; the context bound avoids reserving a custom model's full training context for short memory chunks.

Existing context, batch, and slot settings in the embedding section, [*], or the managed service's arguments or environment remain authoritative. Apus adds the context bound only when no slot count or per-slot context is configured, because llama.cpp can divide total context across slots. If you choose a different slot count, size its context too. Long queries or direct embedding requests may need larger native capacity settings on models that support them. Missing defaults are added on the next managed preset preparation.

Set up only local embeddings

When memory.search.provider is local and chat setup cannot proceed or is declined, Apus offers a separate embedding-only setup. It installs only the managed server and the configured embedding model after explicit consent. It does not add a llama.cpp chat model or change the current chat model. Setup discovery remains read-only and never installs or downloads anything.

If no recommended chat model fits your memory budget, enable local memory search and retry setup to get the embedding-only offer (about 0.3 GB for the default embedding model):

apus config set memory.search.provider local
apus models auth login --provider llama-cpp --method local

If you use --profile, use the same profile for both commands.

If the llama.cpp provider has any configured chat models, embedding-only setup leaves it unchanged. Move any chat routes to another provider and remove those model entries before retrying. An existing external llama.cpp server config must also be removed before Apus can manage embeddings.

Use another managed GGUF

Add a model under models.providers.llama-cpp.models, select its llama-cpp/<id> reference, and run managed setup again:

{
  id: "my-local-model",
  name: "My local GGUF",
  reasoning: false,
  input: ["text"],
  cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
  contextWindow: 65536,
  maxTokens: 2048,
  params: {
    modelPath: "~/Models/my-model.Q4_K_M.gguf",
    contextSize: 65536,
  },
  compat: { supportsTools: true },
}

modelPath accepts local paths, cache-relative filenames, full hf: file URIs, and HTTPS GGUF URLs that publish a SHA-256 response digest. The default cache is ~/.apus/models/llama.cpp. A configured modelCacheDir remains authoritative for managed setup.

Setup resolves model aliases and account-qualified references to the selected authored model, and keeps that model even when it is not cached. It shows the download size from the pinned recipe or source metadata when available and asks before downloading. Invalid sources produce a model-specific error instead of a different recommendation. Declining leaves the configured routes unchanged. The hardware recommendation applies when no managed chat models are configured; custom models remain subject to the source's checksum and runtime verification.

Existing llama-server

Choose Existing llama-server when another terminal, container, service manager, or machine owns the process.

Give the model a stable alias:

```bash
llama-server \
  --model /path/to/model.gguf \
  --alias my-model \
  --host 127.0.0.1 \
  --port 8080
```

Run `apus onboard`, choose **Existing llama-server**, and enter the
endpoint. Enable API-key authentication only when the server or proxy
requires it.

The URL prompt accepts HTTP or HTTPS endpoints and host shorthand such as
`localhost:8080`. Invalid URLs and embedded credentials are rejected inline
so you can correct the endpoint without restarting setup.

```bash
apus models list --provider llama-cpp
apus models set llama-cpp/my-model
```

Apus reads /health, /models (falling back to /v1/models), and /props. Router property checks use autoload=false. Discovery never loads, wakes, unloads, downloads, or reloads models.

For discovered models, Apus advertises reasoning and effort controls only when /props sets chat_template_caps.supports_reasoning_effort to true. Missing or false values leave those capabilities unadvertised. Explicit configured context and tool values remain authoritative over discovery.

Setup saves context limits and tool support only when live /props verifies them. Unloaded router models remain in the configured inventory with those fields unset. Runtime discovery uses a 128,000-token context fallback and keeps tools disabled while those facts are unknown; it does not save these defaults as model overrides. After loading a model yourself, refresh its live facts:

apus models list --provider llama-cpp --refresh

Refresh fills unset context and tool fields in the runtime catalog from live /props, without changing your explicit overrides or loading other models. Older setup versions may have saved contextWindow: 128000, contextTokens: 128000, and compat.supportsTools: false for unloaded models. If those were generated by setup rather than values you intended to pin, remove the affected model row from models.providers.llama-cpp.models, then refresh or rerun Existing llama-server setup. Load the model first to discover its actual limits. Rerunning setup alone preserves existing overrides.

Refreshing a configured external server reports authentication rejection or unavailability when discovery fails. Previously discovered models remain visible only while their endpoint and credentials are unchanged. A successful empty list removes discovered rows. Explicit configured models remain. Restore the server or correct its credentials, then refresh again to recover the live inventory.

Authentication and endpoint replacement

Existing endpoints support no auth, API keys, SecretRefs, auth profiles, and explicit authorization headers. An explicit Authorization header wins over ambient API-key discovery unless setup receives a new key. Choosing no API key removes the default llama.cpp auth profile and stale inline key fields while preserving an explicit Authorization header and unrelated headers. Endpoint URLs containing a username or password are rejected.

export LLAMA_SERVER_API_KEY="<API_KEY>"
apus onboard

When the endpoint changes, setup does not send the old endpoint's environment, profile, configured key, or header credentials to the replacement. Switching from managed mode also removes localService, managed model/cache parameters, and the managed request timeout before discovery.

For non-interactive setup:

apus onboard \
  --non-interactive \
  --accept-risk \
  --auth-choice llama-cpp-existing-server \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id my-model

Use --llama-server-api-key <API_KEY> when a replacement endpoint requires a new credential. LLAMA_SERVER_API_KEY remains available for initial setup and unchanged endpoints.

Manual configuration

Guided setup is recommended because it verifies discovery. The minimal manual shape is:

{
  models: {
    mode: "merge",
    providers: {
      "llama-cpp": {
        baseUrl: "http://127.0.0.1:8080/v1",
        api: "openai-completions",
        request: { allowPrivateNetwork: true },
        models: [],
      },
    },
  },
}

Custom provider IDs may also point at llama-server through the generic OpenAI-compatible path. They remain custom providers and should declare the llamacpp tool-schema profile explicitly. See custom provider capability declarations.

Requests and local embeddings

Both ownership choices use Apus's normal chat, image, streaming, and tool transport. The llama.cpp compatibility family cleans unsupported tool-schema constraints. Agent turns and standalone completions also map thinking-off requests to the server's chat-template flag and adapt JSON Schema requests for older llama-server builds.

Local memory embeddings require managed mode:

{
  memory: {
    search: {
      provider: "local",
      local: {
        modelPath: "hf:ggml-org/embeddinggemma-300m-qat-q8_0-GGUF/embeddinggemma-300m-qat-Q8_0.gguf",
      },
    },
  },
}

The default EmbeddingGemma model automatically receives different task prefixes for search queries and indexed documents. Existing indexes built without these prefixes rebuild once on the next search or sync, without --force. Cached raw embeddings are not reused, and keyword search remains available during recovery. Run apus memory status --index after intentionally changing the embedding model.

Troubleshooting

• Managed setup: run apus doctor and apus memory status --deep.
• Existing server: inspect /health, /models, and /props. HTTP 503 means the model is still loading.
• Missing tools: verify both tool capability flags in /props and use a tool-capable Jinja chat template.
• Managed Linux builds require glibc 2.34 on x64 or 2.38 on arm64. Windows builds require the Microsoft Visual C++ 2015-2022 Redistributable.
• A model that replies to a simple prompt but fails the setup tool check is not selected as the default. Retry after reviewing its tool support or choose another model.
• Platforms without a verified managed build should use an existing server.

Apus does not auto-select ROCm, SYCL, OpenVINO, or Vulkan archives.

Related

• Local model services
• Model providers
• LM Studio
• Llama Cpp plugin reference — manifest and config reference for the managed and external llama.cpp servers