Agent run failed with a storage error

An error naming the Gateway state database identifies a storage failure observed during the run. The chat banner, recorded assistant error, and embedded_run_agent_end log show the same diagnosis. Provider response bodies remain redacted.

A transcript writer ownership error means the run lost its session write claim. Retry in the current session and inspect Gateway logs if it recurs. Storage failures do not trigger provider credential rotation or automatic replay of the run.

Use apus logs --follow to correlate the run with storage activity. SQLite can contend between connections or worker threads in one Gateway process; seeing only one process with the database open does not rule out contention. See database concurrency notes. Avoid full database compaction while runs are active.

Provider rejected the request

A reply beginning with LLM request rejected: includes the provider's request-validation message. Apus redacts credentials, bounds the diagnostic, and displays it as literal text rather than exposing the full response body. Known context, token-limit, and storage failures retain their specific recovery guidance.

For Invalid service_tier argument, check the selected model and speed setting. A provider or account supporting Fast or Ultrafast does not mean every model supports that tier. Retry with Standard (/fast off) or select a model that supports the requested tier. This error alone does not mean the conversation is corrupt. See OpenAI Fast mode.

Unreadable conversation history

If a reply says Apus could not read the conversation's history, ask the Gateway operator to try apus doctor --fix on the host and profile that own the session. This notice also appears for unmentioned group turns when silent replies are allowed.

Doctor can restore a missing header when the stored transcript entries are already canonical. It does not repair every malformed or unsupported history. If the error persists, preserve the state and contact support with the Gateway logs; the history may need migration or recovery from a backup. Repeated /new or /compact commands do not repair a transcript that cannot be loaded.

See Doctor's session repairs and running Doctor.

No replies

If channels are up but nothing answers, check routing and policy before reconnecting anything.

apus status
apus channels status --probe
apus pairing list --channel <channel> [--account <id>]
apus config get channels
apus logs --follow

Look for:

• Pairing pending for DM senders.
• Group mention gating (requireMention, mentionPatterns).
• Channel/group allowlist mismatches.

Common signatures:

• drop guild message (mention required → group message ignored until mention.
• pairing request → sender needs approval.
• blocked / allowlist → sender/channel was filtered by policy.

A Session delivery: entry <id> was dead-lettered and moved to failed warning means a queued session delivery finished terminal cleanup and will not be replayed. Use the queue entry ID to correlate earlier logs, then check the conversation before retrying: an interrupted delivery may already have produced side effects. Batch recovery includes the finalized entry in its failure count.

Related:

• Channel troubleshooting
• Groups
• Pairing

If a shared model catalog worker exits and its automatic runtime replacement fails, opening the model picker, preparing a chat, or sending a new message can check the failed runtime again without restarting the Gateway. The picker shows Loading models… while checking, then returns to Models unavailable if preparation still fails. Repeated foreground checks have a short cooldown. Cron and Heartbeat get one check per failure episode; later scheduled runs do not keep rebuilding a runtime that remains unavailable. Existing messages and tasks are not replayed.

This recovery still waits for earlier preparation work to settle. A preparation that never finishes, unrelated configuration errors, and Gateway shutdown require their own diagnosis; a model check does not bypass those lifecycle boundaries.

Dashboard control UI connectivity

When the dashboard/control UI will not connect, validate its URL, authentication, and device identity.

apus gateway status
apus status
apus logs --follow
apus doctor
apus gateway status --json

Look for:

• Correct check URL and dashboard URL.
• Auth mode/token mismatch between client and gateway.
• Clients that connect without the required device identity. The current Control UI can create and sign identity over plain HTTP; see Insecure HTTP.

If a local browser cannot connect to 127.0.0.1:18789 after an update, first recover the local Gateway service and confirm it is serving the dashboard:

apus gateway restart
lsof -i :18789
curl http://127.0.0.1:18789

If curl returns Apus HTML, the Gateway is working and the remaining issue is likely browser cache, an old deep link, or stale tab state. Open http://127.0.0.1:18789 directly and navigate from the dashboard. If restart does not leave the service running, run apus gateway start and recheck apus gateway status.

- `device identity required` → the client did not provide the identity required by its role and auth policy. Plain HTTP alone is not the cause. In token/password mode, the Control UI still requires browser device identity; the shared secret does not replace it.
- `origin not allowed` → the browser `Origin` is not allowed and is not a private same-origin load. Private same-origin loads, including private LAN/Tailscale addresses and `.local` or `.ts.net` hosts, do not need an allowlist entry. Public or cross-origin browser deployments need an entry in `gateway.controlUi.allowedOrigins`.
- `device nonce required` / `device nonce mismatch` → client is not completing the challenge-based device auth flow (`connect.challenge` + `device.nonce`).
- `device signature invalid` / `device signature expired` → client signed the wrong payload (or stale timestamp) for the current handshake.
- `AUTH_TOKEN_MISMATCH` with `canRetryWithDeviceToken=true` → client can do one trusted retry with cached device token.
- That cached-token retry reuses the cached scope set stored with the paired device token. Explicit `deviceToken` / explicit `scopes` callers keep their requested scope set instead.
- `AUTH_SCOPE_MISMATCH` → the device token was recognized, but its approved scopes do not cover this connect request; re-pair or approve the requested scope contract instead of rotating a shared gateway token.
- Outside that retry path, connect auth precedence is explicit shared token/password first, then explicit `deviceToken`, then stored device token, then bootstrap token.
- On the async Tailscale Serve Control UI path, failed attempts for the same `{scope, ip}` are serialized before the limiter records the failure. Two bad concurrent retries from the same client can therefore surface `retry later` on the second attempt instead of two plain mismatches.
- `too many failed authentication attempts (retry later)` from a browser-origin loopback client → repeated failures from that same normalized `Origin` are locked out temporarily; another localhost origin uses a separate bucket.
- Repeated `unauthorized` after that retry → shared token/device token drift; refresh token config and re-approve/rotate device token if needed.
- `gateway connect failed:` → wrong host/port/url target.

Auth detail codes quick map

Use error.details.code from the failed connect response to pick the next action:

Device auth v2 migration check:

apus --version
apus doctor
apus gateway status

If logs show nonce/signature errors, update the connecting client and verify it:

Client waits for the gateway-issued `connect.challenge`.

Client signs the challenge-bound payload.

Client sends `connect.params.device.nonce` with the same challenge nonce.

If apus devices rotate / revoke / remove is denied unexpectedly:

• Paired-device token sessions can manage only their own device unless the caller also has operator.admin.
• apus devices rotate --scope ... can only request operator scopes that the caller session already holds.

Related:

• Configuration (gateway auth modes)
• Control UI
• Gateway protocol auth — the wire contract behind AUTH_TOKEN_MISMATCH and AUTH_SCOPE_MISMATCH
• Devices
• Remote access
• Trusted proxy auth