Apus generates images, videos, and music, understands inbound media (images, audio, video), and speaks replies aloud with text-to-speech. All media capabilities are tool-driven: the agent decides when to use them based on the conversation, and each tool only appears when at least one backing provider is configured.

Live speech uses the Talk session contract instead of the one-shot media tool path. Talk has three modes: provider-native realtime, local or streaming stt-tts, and transcription for observe-only speech capture. Those modes share provider catalogs, event envelopes, and cancellation semantics with telephony, meetings, browser realtime, and native push-to-talk clients.

Capabilities

Create and edit images from text prompts or reference images via
`image_generate`. Async in chat sessions — runs in the background and
posts the result when ready.

Text-to-video, image-to-video, and video-to-video via `video_generate`.
Async — runs in the background and posts the result when ready.

Generate music or audio tracks via `music_generate`. Async in chat
sessions on the shared media-generation task lifecycle.

Convert outbound replies to spoken audio via the `tts` tool plus
`tts` config. Synchronous.

Summarize inbound images, audio, and video using vision-capable model
providers and dedicated media-understanding plugins.

Transcribe inbound voice messages through batch STT or Voice Call
streaming STT providers.

Play assistant audio and video inline across the Control UI and native
apps, with managed access and portable playback renditions.

Local media files

view_image, pdf, and reference inputs for image_generate, music_generate, and video_generate use the task's working directory for relative paths. A task running in a Git worktree can read media from that worktree even when the agent's default workspace is elsewhere.

Workspace-only access follows the session's approved filesystem root, which can include parent directories of the current working directory. Host tools can also read the Gateway's media store, including browser screenshots and staged inbound attachments. Paths and symlinks outside these roots are rejected. Sandboxed tools read through the sandbox filesystem; selecting a host worktree does not grant access outside the sandbox.

Provider capability matrix

Async vs synchronous

For async tools, Apus submits the request to the provider, returns a task id immediately, and tracks the job in the media runtime. The agent continues responding to other messages while the job runs. When the provider finishes, Apus wakes the agent with the generated media paths so it can tell the user through the session's normal visible-reply mode: automatic final reply delivery when configured, or message(action="send") when the session requires the message tool. If the requester session is inactive or its active wake fails, and some generated media is still missing from the completion reply, Apus sends an idempotent direct fallback with only the missing media. Media already delivered by the completion reply is not posted again.

If completion delivery cannot be confirmed and the original session still exists, Apus tries to retain the generated media references in that session's history. Completion stays bound to the original session and storage location; a replacement session never receives it. Transient queue admission failures use a bounded retry. If both queue admission and history storage remain unavailable, the completion ends with a diagnostic and process-local media references; durable recovery is not guaranteed. The local history notice does not resend media to the channel.

Speech-to-text and Voice Call

Deepgram, DeepInfra, ElevenLabs, Google, Groq, Mistral, OpenAI, OpenRouter, SenseAudio, and xAI can all transcribe inbound audio through the batch tools.media.audio path when configured. Channel plugins that preflight a voice note for mention gating or command parsing mark the transcribed attachment on the inbound context, so the shared media-understanding pass reuses that transcript instead of making a second STT call for the same audio.

Deepgram, ElevenLabs, Mistral, OpenAI, and xAI also register Voice Call streaming STT providers, so live phone audio can be forwarded to the selected vendor without waiting for a completed recording.

For live user conversations, prefer Talk mode. Batch audio attachments stay on the media path; browser realtime, native push-to-talk, telephony, and meeting audio should use Talk events and the session-scoped catalogs returned by the Gateway.

Provider mappings (how vendors split across surfaces)

Image, video, music, batch TTS, batch STT, backend realtime voice, and
media-understanding surfaces.

Image, video, batch TTS, batch STT, Voice Call streaming STT, backend
realtime voice, and memory-embedding surfaces.

Chat/model routing, image generation/editing, text-to-video, batch TTS,
batch STT, image media understanding, and memory-embedding surfaces.
DeepInfra also exposes reranking, classification, object-detection, and
other native model types; Apus has no provider contract for those
categories yet, so this plugin does not register them.

Image, video, search, code-execution, batch TTS, batch STT, and Voice
Call streaming STT. xAI Realtime voice is an upstream capability but is
not registered in Apus until the shared realtime-voice contract can
represent it.

Related

• Image generation
• Video generation
• Music generation
• Text-to-speech
• Media playback
• Media understanding
• Audio nodes
• Talk mode