Apus can summarize inbound media (image/audio/video) before the reply pipeline runs, so command parsing and routing work off short text instead of raw bytes. Understanding auto-detects local tools or provider keys, or you can configure explicit models. In the Apus-managed reply path, successfully described images are represented by text instead of being attached again as native images. Failed, disabled, or unselected images remain eligible for normal native image delivery; understanding errors do not block the reply.

Vendor plugins register capability metadata (which provider supports which media type, default model, priority). Apus core owns the shared tools.media config, fallback order, and reply-pipeline integration.

How it works

Collect ordered inbound media facts (`path`, `url`, `contentType`, and `kind`).

For each enabled capability (image/audio/video), select attachments per the `attachments` policy (default: first attachment only).

Pick the first eligible model entry (size + capability + auth available).

If a model errors, times out, or the media exceeds `maxBytes`, try the next entry.

`Body` becomes an `[Image]`, `[Audio]`, or `[Video]` block. Audio also sets `{{Transcript}}`; command parsing uses caption text when present, otherwise the transcript. Captions are preserved as `User text:` inside the block.

Config

tools.media holds one capability-tagged model list plus small per-capability controls:

{
  tools: {
    media: {
      concurrency: 2, // max concurrent capability runs (default)
      models: [
        { provider: "openai", model: "gpt-4o-mini-transcribe", capabilities: ["audio"] },
        { provider: "google", model: "gemini-3-flash-preview", capabilities: ["image", "video"] },
      ],
      image: { preferredModel: "google/gemini-3-flash-preview" },
      audio: { enabled: true },
      video: { enabled: true },
    },
  },
}

Per-capability (image/audio/video) keys:

Prompts, limits, language hints, request overrides, and provider options can be set as capability defaults or overridden on individual tools.media.models[] entries. Capability defaults also cover auto-detected providers when no explicit model is configured. A model list selects which entries run; it does not hide other enabled providers from provider discovery.

Model entries

Each models[] entry is a provider entry (default) or a CLI entry:

```json5
{
  type: "provider", // default if omitted
  provider: "openai",
  model: "gpt-6-astra",
  prompt: "Describe the image in <= 500 chars.",
  maxChars: 500,
  maxBytes: 10485760,
  timeoutSeconds: 60,
  capabilities: ["image"],
  profile: "vision-profile",
  preferredProfile: "vision-fallback",
}
```

```json5
{
  type: "cli",
  command: "gemini",
  args: [
    "-m",
    "gemini-3-flash",
    "--allowed-tools",
    "read_file",
    "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
  ],
  maxChars: 500,
  maxBytes: 52428800,
  timeoutSeconds: 120,
  capabilities: ["video", "image"],
}
```

Placeholders are substituted literally into each argument, and an unset one becomes an **empty argument** rather than being dropped. `{{Language}}` is only populated for audio, when a language is resolved from the request, the entry's `language`, or `tools.media.audio.language`; with none set, `["--language", "{{Language}}"]` runs as `--language ""`. Transcribers that reject an empty language should omit the flag and its value together and rely on autodetection. `{{Prompt}}` falls back to the built-in default prompt, so it is normally populated. The same empty substitution applies to any other placeholder without a value, such as `{{AttachmentContentType}}` for an attachment with no MIME type.

CLI templates can also use `{{AttachmentUrl}}`, `{{AttachmentContentType}}`, `{{AttachmentDir}}`, `{{AttachmentIndex}}`, `{{OutputDir}}` (scratch dir created for this run), and `{{OutputBase}}` (scratch file base path, no extension). The `{{Attachment*}}` names replaced `{{MediaPath}}`, `{{MediaUrl}}`, `{{MediaType}}`, and `{{MediaDir}}` in 2026.8.1. The older names remain deprecated compatibility aliases under the `media-legacy-projection` record: its approved `removeAfter` date is 2026-10-01, with removal additionally gated on a clean published-plugin artifact sweep. Migrate before that date - see [Media legacy projection](/plugins/sdk-migration/compatibility-policy#media-legacy-projection).

CLI entries need a nonblank command and a nonempty args list. Arguments remain literal strings with optional template interpolation; existing literal file paths and custom wrapper arguments are supported. Pass the attachment through a template such as {{AttachmentPath}} or your command's existing input contract. Empty argument lists are not supported because Apus does not feed attachments to CLI stdin. apus doctor reports missing commands or args with the exact config path and a manual fix; it does not invent commands or rewrite these entries. At runtime, an incomplete entry records a failure without launching the binary, and the next configured model is tried. If none succeeds, the attachment gets a failure outcome and a warning is logged. Config validation remains permissive for these fields so an existing config can still start the Gateway after an update.

Remote attachments are staged as temporary files only when a CLI needs a file path. The media-understanding run owns those files and removes them after processing; a failed staging attempt leaves other staged attachments available. Existing local attachments are read in place and retained.

Provider credentials

Provider media understanding uses the same auth resolution as normal model calls: auth profiles, environment variables, then models.providers.<providerId>.apiKey. tools.media.models[] entries do not accept an inline apiKey field.

{
  models: {
    providers: {
      openai: { apiKey: "<OPENAI_API_KEY>" },
      moonshot: { apiKey: "<MOONSHOT_API_KEY>" },
    },
  },
}

See Tools and custom providers for profiles, env vars, and custom base URLs.

Rules and behavior

• Media exceeding maxBytes skips that model and tries the next one.
• Audio files under 1024 bytes are treated as empty/corrupt and skipped before transcription; the agent gets a deterministic placeholder transcript instead.
• If the active primary image model already supports vision natively, Apus skips the [Image] summary block and passes the original image into the model directly. MiniMax is an exception: minimax, minimax-cn, minimax-portal, and minimax-portal-cn always route image understanding through the plugin-owned MiniMax-VL-01 media provider, even if legacy MiniMax M2.x chat metadata claims image input (only MiniMax-M3 and later are treated as natively vision-capable).
• If a Gateway/WebChat primary model is text-only, image attachments are preserved as offloaded media://inbound/* refs so image/PDF tools or a configured image model can still inspect them instead of losing the attachment.
• Explicit apus infer image describe --file <path> --model <provider/model> (alias: apus capability image describe) runs that image-capable provider/model directly, including Ollama refs such as ollama/qwen2.5vl:7b when a matching image-capable model is configured under models.providers.ollama.models[].
• If <capability>.enabled is not false but no models are configured, Apus tries the active reply model when its provider supports the capability.

Auto-detect (default)

When tools.media.<capability>.enabled is not false and no models are configured, Apus tries these in order and stops at the first working option:

`agents.defaults.imageModel` primary/fallback refs, unless the active reply model already supports vision natively. Prefer `provider/model` refs; bare refs are qualified from configured image-capable provider model entries only when the match is unique.

The active reply model, when its provider supports the capability.

Configured `models.providers.*` entries that support audio are tried before local CLIs. Bundled provider priority order (ties break alphabetically by provider id): Groq/OpenAI → xAI → Deepgram → OpenRouter → Google/SenseAudio → Deepinfra/ElevenLabs → Mistral.

Ready local binaries become an ordered fallback list:
- `whisper-cli` first only after an earlier model invocation in the current process observed Metal or CUDA
- CPU-default `sherpa-onnx-offline` (requires `SHERPA_ONNX_MODEL_DIR` with `tokens.txt`/`encoder.onnx`/`decoder.onnx`/`joiner.onnx`)
- `whisper-cli` when acceleration is merely build-capable or unobserved
- `parakeet-mlx` on Apple Silicon (MLX-capable, device use unobserved)
- `whisper` (Python CLI; defaults to the `turbo` model, downloads automatically)

Backend capability inspection is cached and does not load a model. Build capability, requested backend flags, and backend observed from a real invocation remain separate. Auto-detected whisper.cpp leaves model-run logs enabled so the upstream selected-backend line can be recorded. Explicit CLI entries keep their configured order, backend flags, and output flags.

Configured `models.providers.*` entries that support the capability are tried before the bundled fallback order. Image-only config providers with an image-capable model auto-register for media understanding even when they are not a bundled vendor plugin.

Bundled provider priority order (ties break alphabetically by provider id):
- Image: Anthropic/OpenAI → Google → MiniMax → Deepinfra → MiniMax Portal → Z.AI
- Video: Google → Qwen → Moonshot

To disable auto-detection for a capability:

{
  tools: {
    media: {
      audio: {
        enabled: false,
      },
    },
  },
}

Proxy support (audio/video provider calls)

Provider-based audio and video understanding honors standard outbound proxy environment variables, including NO_PROXY/no_proxy bypass rules: HTTPS_PROXY, HTTP_PROXY, ALL_PROXY, https_proxy, http_proxy, all_proxy. Lowercase vars take precedence over uppercase. If none are set, media understanding uses direct egress; if the proxy value is malformed, Apus logs a warning and falls back to direct fetch. Image understanding does not go through this proxy path.

Capabilities

Set capabilities on a models[] entry to restrict it to specific media types. For shared lists, Apus infers defaults per bundled provider:

CLI entries require explicit capabilities; entries without valid capability tags are skipped. Provider entries without valid explicit tags use their registered capability metadata.

Provider support matrix

Model selection guidance

• Prefer the strongest current-generation model for each media capability when quality and safety matter.
• For tool-enabled agents handling untrusted inputs, avoid older/weaker media models.
• Keep at least one fallback per capability for availability (quality model + faster/cheaper model).
• CLI fallbacks (whisper-cli, whisper, gemini) help when provider APIs are unavailable.
• Known file-output modes are authoritative: an empty or missing inferred transcript file produces no transcript instead of falling back to CLI progress output.
• parakeet-mlx: use --output-format txt (or all) with --output-dir and the default {filename} output template. The upstream PARAKEET_OUTPUT_FORMAT and PARAKEET_OUTPUT_TEMPLATE environment variables are also honored. Apus reads <output-dir>/<media-basename>.txt; the default srt format, other formats, and custom output templates continue to use stdout.

Attachment policy

Per-capability attachments controls which attachments are processed:

When mode: "all", outputs are labeled [Image 1/2], [Audio 2/2], etc.

Local attachments stay within the session's allowed media roots. Directory aliases such as macOS /tmp and /private/tmp are accepted when the opened file remains inside those roots; they do not grant access to sibling sandbox workspaces.

File-attachment extraction

• Every inbound document attachment ends in a model-visible file block. Attachments routed to image, audio, or video understanding are outside this contract; those stages own their outcomes.
• Extracted file text is wrapped as untrusted external content before it's appended to the media prompt, using boundary markers like <<<EXTERNAL_UNTRUSTED_CONTENT id="...">>> / <<<END_EXTERNAL_UNTRUSTED_CONTENT id="...">>> plus a Source: External metadata line.
• This path intentionally omits the one-line data-boundary note to keep the media prompt short; the boundary markers and metadata still apply.
• Unsupported files saved on local disk get self-serve guidance only when the reply runtime proves it can read host-local paths (currently non-sandboxed embedded sessions). The path is fenced as untrusted external metadata; the trusted guidance tells the agent to extract the file with its own tools, and modern Office files get an unzip hint. Generic ACP backends, URL-only attachments, and sandboxed sessions keep the plain [Unsupported document format: <mime>. PDF and plain-text attachments can be read.] marker.
• Files rejected by an operator-configured allowlist never include the self-serve path; a policy rejection must not coach the agent around the operator's decision.
• Files rejected by an operator-configured allowedMimes list get [Attachment type not allowed: <mime>] instead, so the prompt never claims support the active configuration disables.
• Read failures get [Attachment could not be read].
• URL attachments get [Attachment skipped: URL file sources are disabled] when URL file sources are disabled.
• A file with no extractable text, including an empty local text file, gets [No extractable text] and does not consume the skip-marker budget.
• At most five skip markers render per message; further skipped attachments collapse into one reason-neutral [<n> more attachments skipped] summary so junk attachments cannot grow the prompt without bound. File and image, audio, or video markers share this five-marker budget.
• If a PDF falls back to rendered page images, Apus forwards those images to vision-capable reply models and keeps the placeholder [PDF content rendered to images] in the file block.
• If page, text, or image limits make document extraction partial, the file block starts with a bounded [Partial document: ...] marker so the reply model does not mistake the visible prefix for the complete attachment.
• Image, audio, and video decisions record one closed disposition for every attachment candidate: handled, handed to native vision, not selected after the attachment limit, disabled, missing a model, denied by chat scope, or failed.
• Unhandled media gets a bounded model-visible marker. Images handed to native vision do not add markers. When a native harness owns the turn and Apus runs only audio preprocessing, failed or skipped audio still gets a marker; image, video, and document inputs remain owned by the harness. Too-small audio keeps its placeholder transcript without a duplicate marker.

Vision models and image replay cost

In the Apus-managed reply path, a vision-capable reply model normally receives native images without a separate description pass. Images retained in later requests can contribute to input cost; the amount depends on the provider and the active runtime's context handling. See Session pruning for the applicable cleanup behavior.

For identification or extraction tasks that do not need continued pixel access, an image-capable entry in tools.media.models[] opts into image understanding even when the reply model supports vision. For each selected image that is successfully described, the reply receives description text in place of that native image. This is not a guarantee that every attached image is converted or that later requests contain no images.

{
  tools: {
    media: {
      models: [{ provider: "google", model: "gemini-3-flash-preview", capabilities: ["image"] }],
    },
  },
}

Tradeoffs and boundaries:

• Selection defaults to the first matching image only. Use the existing tools.media.image.attachments policy to select more, with an explicit maxAttachments cap. Failed or unselected images can still be delivered natively.
• A successful description replaces that image's pixels on the current reply turn. Avoid this when the reply model needs to inspect pixels directly, such as screenshots, UI review, or visual debugging.
• An explicit image-capable tools.media.models[] entry overrides the native-vision skip, subject to the existing enablement, scope, size, and attachment limits. tools.media.image.preferredModel alone does not override that skip.
• tools.media.image.enabled: false disables image understanding; it does not disable native image delivery.
• This describes Apus-managed preprocessing. A native harness that owns image handling has its own delivery and context rules; do not assume this setting controls its image replay or billing.

Config examples

```json5
{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-6-astra", capabilities: ["image"] },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["image", "audio", "video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image", "video"],
        },
      ],
      audio: {
        attachments: { mode: "all", maxAttachments: 2 },
      },
      video: {
        maxChars: 500,
      },
    },
  },
}
```

```json5
{
  tools: {
    media: {
      models: [
        {
          provider: "openai",
          model: "gpt-4o-mini-transcribe",
          capabilities: ["audio"],
        },
        {
          type: "cli",
          command: "whisper",
          args: ["--model", "base", "{{AttachmentPath}}"],
          capabilities: ["audio"],
        },
        {
          provider: "google",
          model: "gemini-3-flash-preview",
          capabilities: ["video"],
        },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["video"],
        },
      ],
      audio: {
        enabled: true,
      },
      video: {
        enabled: true,
        maxChars: 500,
      },
    },
  },
}
```

```json5
{
  tools: {
    media: {
      models: [
        { provider: "openai", model: "gpt-6-astra", capabilities: ["image"] },
        { provider: "anthropic", model: "claude-opus-5", capabilities: ["image"] },
        {
          type: "cli",
          command: "gemini",
          args: [
            "-m",
            "gemini-3-flash",
            "--allowed-tools",
            "read_file",
            "Read the media at {{AttachmentPath}} and describe it in <= {{MaxChars}} characters.",
          ],
          capabilities: ["image"],
        },
      ],
      image: {
        enabled: true,
        maxBytes: 10485760,
        maxChars: 500,
      },
    },
  },
}
```

```json5
{
  tools: {
    media: {
      models: [
        {
          provider: "google",
          model: "gemini-3.1-pro-preview",
          capabilities: ["image", "video", "audio"],
        },
      ],
    },
  },
}
```

Status output

When media understanding runs, /status includes a per-capability summary line:

📎 Media: image ok (openai/gpt-6-astra) · audio ok (whisper-cli observed=metal)

Auto-detected local audio tools report their resolved executable path as the result's model, so status and verbose summaries can include that path alongside the tool family and backend. Explicit CLI entries retain their authored command; preflight inventory keeps the logical tool names.

For preflight inventory, run apus capability audio providers. Local rows show the local fallback winner separately from global provider selection, readiness, and separate capable/requested/observed backend fields. The same local selection is available as an informational doctor finding:

apus doctor --lint --only core/doctor/local-audio-acceleration --severity-min info

Notes

• Understanding is best-effort. Errors do not block replies.
• Attachments are still passed to models even when understanding is disabled.
• Use scope to limit where understanding runs (for example, only DMs).

Related

• Configuration
• Image & media support
• Audio and voice notes
• Camera capture
• Media playback
• Talk mode
• Voice wake
• Mistral — Voxtral audio transcription, one of the audio-capable providers above
• Media overview — how the media tools fit together