Apus agents generate videos from text prompts, reference images, or existing videos through video_generate. Eighteen provider backends are supported; the agent picks the right one automatically based on config and available API keys. The 18 provider IDs span 17 plugins; MiniMax registers separate API-key (minimax) and OAuth (minimax-portal) backends.

video_generate has three runtime modes, resolved from the reference inputs in the call:

• generate - no reference media (text-to-video).
• imageToVideo - one or more reference images.
• videoToVideo - one or more reference videos.

Providers can support any subset of those modes. The tool validates the active mode before submission and reports supported modes in action=list.

Quick start

Set an API key for any supported provider:

```bash
export GEMINI_API_KEY="your-key"
```

```bash
apus config set agents.defaults.mediaModels.video.primary "google/veo-3.1-fast-generate-preview"
```

> Generate a 5-second cinematic video of a friendly lobster surfing at sunset.

The agent calls `video_generate` automatically. No tool allowlisting
is needed.

How async generation works

Video generation is asynchronous:

1. Apus submits the request to the provider and immediately returns a task id.
2. The provider processes the job in the background (typically 30 seconds to several minutes depending on the provider and resolution; slow queue-backed providers can run up to the configured timeout).
3. When the video is ready, Apus wakes the same session with an internal completion event.
4. The agent reports it through the session's normal visible-reply mode: automatic final reply, or message(action="send") when the session requires the message tool. If the requester session is inactive, or its wake fails and generated media is still missing from the completion reply, Apus sends an idempotent direct fallback with the media.

While a job is in flight, duplicate video_generate calls in the same chat return the current task status instead of starting another generation. Use action: "status" to check without triggering a new generation. Direct chats keep separate tasks even when they share the main session transcript; completion returns to the requesting peer.

Outside of session-backed agent runs (for example, direct tool invocations), the tool falls back to inline generation and returns the final media path in the same turn.

Generated video files save under Apus-managed media storage when the provider returns bytes. The default cap is 16MB (the shared video media limit); agents.defaults.mediaMaxMb raises it for larger renders. When a provider also returns a hosted output URL, Apus delivers that URL instead of failing the task if local persistence rejects an oversized file.

Task lifecycle

Supported providers

Some providers accept additional or alternate API key env vars. See individual provider pages for details.

Run video_generate action=list to inspect available providers, models, and runtime modes at runtime.

Capability matrix

The explicit mode contract used by video_generate, contract tests, and the shared live sweep:

Tool parameters

Required

Content inputs

Style controls

adaptive is a provider-specific sentinel: it is forwarded as-is to providers that declare adaptive in their capabilities (e.g. BytePlus Seedance uses it to auto-detect the ratio from the input image dimensions). Providers that do not declare it surface the value via details.ignoredOverrides in the tool result so the drop is visible.

Advanced

For Runway and Together, an explicit timeout covers submission, polling, and downloading together. Individual HTTP requests retain a 120-second limit. Without an explicit timeout, their 120-second polling budget starts after submission; request and download limits remain separate.

Reference inputs select the runtime mode:

• No reference media -> generate
• Any image reference -> imageToVideo
• Any video reference -> videoToVideo
• Reference audio inputs do not change the resolved mode; they apply on top of whatever mode the image/video references select, and only work with providers that declare maxInputAudios.

Mixed image and video references are not a stable shared capability surface. Prefer one reference type per request.

Fallback and typed options

Some capability checks apply at the fallback layer rather than the tool boundary, so a request that exceeds the primary provider's limits can still run on a capable fallback:

• Active candidate declaring no maxInputAudios (or 0) is skipped when the request contains audio references; next candidate is tried. The same guard applies to image and video reference counts against maxInputImages/maxInputVideos.
• Active candidate's maxDurationSeconds below the requested durationSeconds with no declared supportedDurationSeconds list -> skipped.
• Request contains providerOptions and the active candidate explicitly declares a typed providerOptions schema -> skipped if supplied keys are not in the schema or value types do not match. Providers without a declared schema receive options as-is (backward-compatible pass-through). A provider can opt out of all provider options by declaring an empty schema (capabilities.providerOptions: {}), which causes the same skip as a type mismatch.

The first skip reason in a request logs at warn so operators see when their primary provider was passed over; subsequent skips log at debug to keep long fallback chains quiet. If every candidate is skipped, the aggregated error includes the skip reason for each. If a candidate fails during generation, its provider, model, and error log at warn before the next candidate is tried.

Actions

Model selection

For video_generate, Apus resolves the model in this order:

1. model tool parameter - when set, only this model is tried.
2. agents.defaults.mediaModels.video.primary from config.
3. agents.defaults.mediaModels.video.fallbacks in order.
4. Auto-detection - only when neither a primary nor fallback model is configured, using configured provider defaults. The current default provider comes first, then remaining providers in alphabetical order.

If a provider fails, the next candidate is tried automatically. If all candidates fail, the error includes details from each attempt.

Explicit video model configuration limits fallback to the configured list; Apus does not append auto-detected providers.

{
  agents: {
    defaults: {
      mediaModels: {
        video: {
          primary: "google/veo-3.1-fast-generate-preview",
          fallbacks: ["runway/gen4.5", "qwen/wan2.6-t2v"],
          timeoutMs: 180000, // optional per-tool provider request timeout override
        },
      },
    },
  },
}

Provider notes

Uses the DashScope / Model Studio async endpoint. Image-to-video and
Wan 2.7 reference images accept local files or remote URLs; local images
are sent as data URIs, up to 20 MB per image before encoding.
A text-to-video model with exactly one image and no video uses its
same-generation image-to-video sibling when that model is in the known
catalog, such as `wan2.6-t2v` to `wan2.6-i2v`. The result reports the
resolved model. Reference videos and Wan 2.6 reference-to-video images
still require remote `http(s)` URLs.

Requires the official `@openclaw/byteplus-provider` plugin.
Provider id: `byteplus`.

Models: `seedance-1-0-pro-250528` (default),
`seedance-1-5-pro-251215`.

Uses the unified `content[]` API. Supports up to 2 input images
(`first_frame` + `last_frame`). Pass images positionally or set each
image's `role` explicitly.

Supported `providerOptions` keys: `seed` (number), `draft` (boolean -
forces 480p), `camera_fixed` (boolean).

Requires the [`@openclaw/byteplus-modelark`](https://www.npmjs.com/package/@openclaw/byteplus-modelark)
plugin (external, not bundled). Provider id: `byteplus-seedance15`. Model:
`seedance-1-5-pro-251215`.

Uses the unified `content[]` API. Supports at most 2 input images
(`first_frame` + `last_frame`). All inputs must be remote `https://`
URLs. Set `role: "first_frame"` / `"last_frame"` on each image, or
pass images positionally.

`aspectRatio: "adaptive"` auto-detects ratio from the input image.
`audio: true` maps to `generate_audio`. `providerOptions.seed`
(number) is forwarded.

Requires the [`@openclaw/byteplus-modelark`](https://www.npmjs.com/package/@openclaw/byteplus-modelark)
plugin (external, not bundled). Provider id: `byteplus-seedance2`. Models:
`dreamina-seedance-2-0-260128`,
`dreamina-seedance-2-0-fast-260128`.

Uses the unified `content[]` API. Supports up to 9 reference images,
3 reference videos, and 3 reference audios. All inputs must be remote
`https://` URLs. Set `role` on each asset - supported values:
`"first_frame"`, `"last_frame"`, `"reference_image"`,
`"reference_video"`, `"reference_audio"`.

`aspectRatio: "adaptive"` auto-detects ratio from the input image.
`audio: true` maps to `generate_audio`. `providerOptions.seed`
(number) is forwarded.

Workflow-driven local or cloud execution. Supports text-to-video and
image-to-video through the configured graph.

Uses a queue-backed flow for long-running jobs. Apus waits up to 20
minutes by default before treating an in-progress fal queue job as timed
out. Most fal video models
accept a single image reference. Seedance 2.0 reference-to-video
models accept up to 9 images, 3 videos, and 3 audio references, with
at most 12 total reference files.

Supports one image or one video reference. Generated-audio requests are
ignored with a warning on the Gemini API path because that API rejects
the `generateAudio` parameter for current Veo video generation.

Uses Kie's market task API for Kling, Grok Imagine, Wan, Hailuo, and
Seedance. A single reference image automatically selects the family's
image-to-video variant. Local images are uploaded through Kie's
documented base64 upload API. Generation can take several minutes;
the provider waits up to ten minutes by default. See [Kie AI](/providers/kie)
for model-specific limits.

Single image reference only. MiniMax accepts `768P` and `1080P`
resolutions; requests such as `720P` are normalized to the closest
supported value before submission.

Uses native Wan 2.6 and Hailuo 2.3 asynchronous routes. One image with a
`-t2v` model selects that family's `-i2v` route. Both families accept
local images as data URIs. Wan defaults to silent output; set
`audio: true` for generated audio. Hailuo supports 1080P only at 6 seconds.
See [NovitaAI](/providers/novita) for model IDs and provider options.

Uses OpenRouter's asynchronous `/videos` API. Apus submits the
job, polls `polling_url`, and downloads either `unsigned_urls` or the
documented job content endpoint. The bundled `google/veo-3.1-fast` default
advertises 4/6/8 second durations, `720P`/`1080P` resolutions, and
`16:9`/`9:16` aspect ratios.

Same DashScope backend as Alibaba. Image-to-video and Wan 2.7 image
references accept local files up to 20 MB before encoding or remote URLs.
With exactly one image and no video, `wan2.6-t2v` automatically uses
`wan2.6-i2v` and reports that model in the result. Reference videos and
Wan 2.6 reference-to-video images require remote `http(s)` URLs.

Supports local files via data URIs. Video-to-video requires
`runway/gen4_aleph`. Text-only runs expose `16:9` and `9:16` aspect
ratios.

Single image reference only.

Uses `https://www.vydra.ai/api/v1` directly to avoid auth-dropping
redirects. `veo3` is text-to-video only; `kling` requires
a remote image URL.

The default `grok-imagine-video` model supports text-to-video, single
first-frame image-to-video, up to 7 `reference_image` inputs through xAI
`reference_images`, and remote video edit/extend flows. Generation defaults
to `480P`; single-image image-to-video inherits the source ratio when
`aspectRatio` is omitted. Video edit/extend inherit the input geometry and
do not accept aspect-ratio or resolution overrides. Extension accepts 2-10
seconds.

`grok-imagine-video-1.5` is image-to-video only: provide exactly one image.
It supports 1-15 seconds and `480P`, `720P`, or `1080P`, defaulting to
`480P`; omit `aspectRatio` to inherit the source image ratio. The preview
and dated 1.5 identifiers receive the same validation and are forwarded
unchanged.

CogVideoX-3 supports text or one PNG/JPEG image, including local files up
to 5 MB sent as data URIs. Durations normalize to 5 or 10 seconds;
`audio: true` enables sound. Video uses the general API endpoint in the
configured global or China region, including when chat uses a Coding
Plan endpoint. See [Z.AI](/providers/zai) for sizes and provider options.

Provider capability modes

The shared video-generation contract supports mode-specific capabilities instead of only flat aggregate limits. New provider implementations should prefer explicit mode blocks:

capabilities: {
  generate: {
    maxVideos: 1,
    maxDurationSeconds: 10,
    supportsResolution: true,
  },
  imageToVideo: {
    enabled: true,
    maxVideos: 1,
    maxInputImages: 1,
    maxInputImagesByModel: { "provider/reference-to-video": 9 },
    maxDurationSeconds: 5,
  },
  videoToVideo: {
    enabled: true,
    maxVideos: 1,
    maxInputVideos: 1,
    maxDurationSeconds: 5,
  },
}

Flat aggregate fields such as maxInputImages and maxInputVideos are not enough to advertise transform-mode support. Providers should declare generate, imageToVideo, and videoToVideo explicitly so live tests, contract tests, and the shared video_generate tool can validate mode support deterministically.

When one model in a provider has wider reference-input support than the rest, use maxInputImagesByModel, maxInputVideosByModel, or maxInputAudiosByModel instead of raising the mode-wide limit.

Live tests

Opt-in live coverage for the shared bundled providers:

APUS_LIVE_TEST=1 pnpm test:live -- extensions/video-generation-providers.live.test.ts

Repo wrapper:

pnpm test:live:media video

This live file uses already-exported provider env vars ahead of stored auth profiles by default, and runs a release-safe smoke by default:

• generate for every non-FAL provider in the sweep.
• One-second lobster prompt.
• Per-provider operation cap from APUS_LIVE_VIDEO_GENERATION_TIMEOUT_MS (180000 by default).

FAL is opt-in because provider-side queue latency can dominate release time:

pnpm test:live:media video --video-providers fal

Set APUS_LIVE_VIDEO_GENERATION_FULL_MODES=1 to also run declared transform modes the shared sweep can exercise safely with local media:

• imageToVideo when capabilities.imageToVideo.enabled.
• videoToVideo when capabilities.videoToVideo.enabled and the provider/model accepts buffer-backed local video input in the shared sweep.

In the shared sweep, buffer-backed videoToVideo runs for runway only with runway/gen4_aleph, and for fal only with a reference-to-video model.

Configuration

Set the default video-generation model in your Apus config:

{
  agents: {
    defaults: {
      mediaModels: {
        video: {
          primary: "qwen/wan2.6-t2v",
          fallbacks: ["qwen/wan2.6-r2v-flash"],
        },
      },
    },
  },
}

Or via the CLI:

apus config set agents.defaults.mediaModels.video.primary "qwen/wan2.6-t2v"

Related

• Alibaba Model Studio
• BytePlus
• ComfyUI
• Configuration reference
• fal
• Google (Gemini)
• Kie AI
• MiniMax
• Models
• NovitaAI
• OpenRouter
• PixVerse
• Qwen
• Runway
• Together AI
• Tools overview
• Vydra
• xAI
• Z.AI
• Media overview - how the media tools fit together