Apus can install and manage a local model or connect to a server you already run. For a hardware-aware recommendation, install the llama.cpp plugin, run apus onboard, and choose Managed local server. Setup shows the Gateway host, model, download size, and execution backend before downloading, then verifies a real tool call before changing the default model. LM Studio and Ollama remain options when you want to manage the model separately.

This page also covers larger local stacks and custom OpenAI-compatible servers. Local models do not provide hosted providers' safety filters. Keep tool permissions and prompt-injection defenses appropriate for the model and task.

For local servers that should start only when a selected model needs them, see Local model services.

Hardware floor

Memory requirements depend on the model weights, context size, runtime, and other work on the host. Managed llama.cpp setup checks available RAM, supported GPU memory, and disk space instead of assuming a particular machine. Its curated recipes use a 64K context. The smallest has an 8 GiB host-memory floor, while larger recipes need more memory. These floors do not guarantee fit or speed. See model recommendations for the current catalog.

For custom servers, leave room for the full Apus prompt, tools, history, and model output. A model that loads or answers a short prompt may still fail an agent turn. Test actual tasks before making it your default, and review local-model security.

Pick a backend

Backend Use when
ds4 Local DeepSeek V4 Flash on macOS Metal with OpenAI-compatible tool calls
LiteLLM / OAI-proxy / custom OpenAI-compatible proxy You front another model API and need Apus to treat it as OpenAI
llama.cpp Hardware-aware model selection, verified downloads, and an Apus-managed server
llmman OCI-registry model pulls, upstream llama.cpp/vLLM/MLX engines, hybrid local + hosted routing
LM Studio First-time local setup, GUI loader, native Responses API
MLX / vLLM / SGLang High-throughput self-hosted serving with an OpenAI-compatible HTTP endpoint
Ollama CLI workflow, model library, hands-off systemd service

Use api: "openai-responses" when the backend supports it (LM Studio does). Otherwise use api: "openai-completions". If api is omitted on a custom provider with a baseUrl, Apus defaults to openai-completions.

WSL2 + Ollama + NVIDIA/CUDA: the official Ollama Linux installer enables a systemd service with Restart=always. On WSL2 GPU setups, autostart can reload the last model during boot and pin host memory, causing repeated VM restarts. See WSL2 crash loop.

LM Studio + large local model (Responses API)

For a separately managed local server, load a model that fits your hardware in LM Studio. Enable the local server (default http://127.0.0.1:1234). Use the Responses API to keep reasoning separate from final text.

{
  agents: {
    defaults: {
      model: { primary: "lmstudio/my-local-model" },
      models: {
        "anthropic/claude-opus-4-6": { alias: "Opus" },
        "lmstudio/my-local-model": { alias: "Local" },
      },
    },
  },
  models: {
    mode: "merge",
    providers: {
      lmstudio: {
        baseUrl: "http://127.0.0.1:1234/v1",
        apiKey: "lmstudio",
        api: "openai-responses",
        models: [
          {
            id: "my-local-model",
            name: "Local Model",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 196608,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}

Setup checklist:

Hybrid config: hosted primary, local fallback

{
  agents: {
    defaults: {
      model: {
        primary: "anthropic/claude-sonnet-4-6",
        fallbacks: ["lmstudio/my-local-model", "anthropic/claude-opus-4-6"],
      },
      models: {
        "anthropic/claude-sonnet-4-6": { alias: "Sonnet" },
        "lmstudio/my-local-model": { alias: "Local" },
        "anthropic/claude-opus-4-6": { alias: "Opus" },
      },
    },
  },
  models: {
    mode: "merge",
    providers: {
      lmstudio: {
        baseUrl: "http://127.0.0.1:1234/v1",
        apiKey: "lmstudio",
        api: "openai-responses",
        models: [
          {
            id: "my-local-model",
            name: "Local Model",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 196608,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}

For local-first with a hosted safety net, swap primary/fallbacks order and keep the same providers block and models.mode: "merge".

Apus fallbacks switch models per turn on provider errors. For per-request routing that keeps small prompts on the local model and sends only oversized ones to a hosted model, see Hybrid inference with llmman.

Regional hosting / data routing

Hosted MiniMax/Kimi/GLM variants also exist on OpenRouter with region-pinned endpoints (for example, US-hosted). Pick the regional variant to keep traffic in your chosen jurisdiction while keeping models.mode: "merge" for Anthropic/OpenAI fallbacks. Local-only is still the strongest privacy path. Hosted regional routing is the middle ground when you need provider features but want control over data flow.

Other OpenAI-compatible local proxies

MLX (mlx_lm.server), vLLM, SGLang, LiteLLM, OAI-proxy, or any custom gateway works if it exposes an OpenAI-style /v1/chat/completions endpoint. Use openai-completions unless the backend explicitly documents /v1/responses support.

{
  agents: {
    defaults: {
      model: { primary: "local/my-local-model" },
    },
  },
  models: {
    mode: "merge",
    providers: {
      local: {
        baseUrl: "http://127.0.0.1:8000/v1",
        apiKey: "sk-local",
        api: "openai-completions",
        timeoutSeconds: 300,
        models: [
          {
            id: "my-local-model",
            name: "Local Model",
            reasoning: false,
            input: ["text"],
            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
            contextWindow: 120000,
            maxTokens: 8192,
          },
        ],
      },
    },
  },
}

Custom/local provider entries trust their exact configured baseUrl origin for guarded model requests, including loopback, LAN, tailnet, and private DNS hosts. Metadata, link-local, and local-use NAT64 (64:ff9b:1::/48) origins remain blocked without explicit opt-in. Requests to other private origins still need models.providers.<id>.request.allowPrivateNetwork: true. Set the trust flag to false to opt out of exact-origin trust.

models.providers.<id>.models[].id is provider-local - do not include the provider prefix. For an MLX server started with mlx_lm.server --model mlx-community/Qwen3-30B-A3B-6bit:

Set input: ["text", "image"] on local or proxied vision models so image attachments get injected into agent turns. Interactive custom-provider onboarding infers common vision model IDs and only asks about unknown names. Non-interactive onboarding uses the same inference, with --custom-image-input / --custom-text-input to override it.

Use models.providers.<id>.timeoutSeconds for slow local/remote model servers before raising agents.defaults.timeoutSeconds. The provider timeout covers connect, headers, body streaming, and the total guarded-fetch abort for model HTTP requests only. If the agent or run timeout is lower, raise that too. The provider timeout cannot extend the whole run.

For custom OpenAI-compatible providers, a non-secret local marker such as apiKey: "ollama-local" is accepted when baseUrl resolves to loopback, a private LAN, .local, or a bare hostname. Apus treats it as a valid local credential instead of reporting a missing key. Use a real value for any provider that accepts a public hostname.

Behavior notes for local/proxied /v1 backends:

Compat declarations are only for the custom endpoint described by this provider row. Catalog-known routes use provider-owned capabilities instead. See the custom-provider capability guide.

Compat overrides for stricter OpenAI-compatible backends:

Smaller or stricter backends

If the model loads cleanly but full agent turns misbehave, check transport first, then inspect tool use and the context budget.

  1. Check the local model responds - no tools, no agent context:

    apus infer model run --local --model <provider/model> --prompt "Reply with exactly: pong" --json

  2. Check Gateway routing - sends only the prompt. It skips transcript, AGENTS bootstrap, context-engine assembly, tools, and bundled MCP servers. It still exercises Gateway routing, auth, and provider selection:

    apus infer model run --gateway --model <provider/model> --prompt "Reply with exactly: pong" --json

  3. Check Tool Search if both checks pass but real agent turns fail with malformed tool calls or oversized prompts. Local Ollama models, LM Studio, and managed local services automatically use structured Tool Search when tools.toolSearch is unset. Other backends can enable it with tools.toolSearch: { mode: "tools" }. This defers schemas while preserving policy-approved capabilities. Leave localModelLean unset or set it to false so optional tools remain available. Check the server's actual context allocation and memory use as well.

  4. Disable tools entirely as a last resort by setting models.providers.<provider>.models[].compat.supportsTools: false for that model - the agent then runs without tool calls.

  5. Inspect the failing request and server logs. Check the chat template, context window, memory pressure, and server errors. A successful text-only check does not prove that the model can reliably complete a multi-step agent task.

Troubleshooting

Local model lean mode

Configure lean mode in Settings → Agent Defaults → Agents with advanced settings shown, or use the config examples below. The retained experimental.localModelLean key remains supported.

Lean mode is an advanced troubleshooting override that explicitly restricts capabilities. Local inference normally uses Tool Search to defer schemas while preserving capabilities, so leave lean mode off unless you deliberately want a smaller tool set.

agents.defaults.experimental.localModelLean: true removes optional tools before catalog construction: browser, automations, message, image_generate, music_generate, video_generate, tts, and pdf. These removed tools cannot be found through Tool Search. Explicitly allowed or delivery-required tools remain available, though Tool Search may catalog them instead of exposing them directly. Lean mode also defaults catalogs to structured Tool Search (tool_search, tool_describe, tool_call) when tools.toolSearch is not already set. Use agents.entries.*.experimental.localModelLean to scope this to one agent.

Setup no longer writes this flag. For older installations, apus doctor --fix removes an onboarding-owned true when its ownership marker still matches the default model. Explicit settings and settings with stale ownership markers are preserved. Set a retained flag to false to restore optional capabilities; automatic Tool Search still applies to local routes.

If you already tune Tool Search globally, Apus leaves that config alone. Set tools.toolSearch: false to opt out of the lean-mode Tool Search default.

In structured tools mode, lean runs keep exec directly visible beside the Tool Search controls so coding-tuned local models can still choose their familiar shell path. This changes schema visibility only: normal tool policy, sandboxing, and exec approvals still apply. Explicit code and directory modes keep their normal compaction behavior.

Why these tools

These tools have the largest descriptions, broadest parameter shapes, or highest chance of distracting a small model from the normal coding and conversation path. On a small-context or stricter OpenAI-compatible backend that is the difference between:

The model still has read, write, edit, exec, apply_patch, image understanding, web search/fetch (when configured), memory, and session/agent tools. Remaining catalog tools stay reachable through Tool Search unless you set tools.toolSearch: false; explicit tool allows can restore a capability removed by lean mode.

When to turn it on

Enable lean mode once you have proved the model can talk to the Gateway but full agent turns misbehave:

  1. apus infer model run --gateway --model <ref> --prompt "Reply with exactly: pong" succeeds.
  2. A normal agent turn fails with malformed tool calls, oversized prompts, or the model ignoring its tools.
  3. Toggling localModelLean: true clears the failure.

When to leave it off

Leave lean mode unset or set agents.defaults.experimental.localModelLean: false to retain the full policy-approved tool set. Setup preserves explicit choices and never enables lean mode automatically.

Lean mode does not replace tools.profile, tools.allow/tools.deny, or the model compat.supportsTools: false escape hatch. For a permanent narrower tool surface on a specific agent, prefer those stable knobs.

Enable

{
  agents: {
    defaults: {
      experimental: {
        localModelLean: true,
      },
    },
  },
}

For one agent only:

{
  agents: {
    entries: {
      local: {
        model: "lmstudio/gemma-4-e4b-it",
        experimental: {
          localModelLean: true,
        },
      },
    },
  },
}

Restart the Gateway after changing the flag in the config file. Lean filtering removes browser, automations, message, image_generate, music_generate, video_generate, tts, and pdf unless you explicitly preserve them with tools.allow or tools.alsoAllow; Tool Search may still catalog preserved tools instead of exposing them directly.

Related