Failed updates enter built-in triage after update recovery settles. In an interactive terminal, Apus shows the selected agent, saved prompt path when available, and use of your own account/tokens, then asks before launching triage. Only an affirmative Yes proceeds. Enter, n, cancellation, or no answer within 30 seconds skips the launch and preserves diagnostics and a manual recovery command. Use apus triage --agent codex to choose another agent. With --yes, --json, or no interactive terminal, eligible failures can start one owned automatic repair; other failures retain diagnostics and handoff commands. See automatic recovery. The original update failure and exit status remain authoritative; diagnostics do not turn a failed update into a successful one.

If the update health check emits a complete lint report but does not exit before its deadline, the failure identifies that completion separately from process termination. An exited child whose output pipes remain open is reported separately. Without a complete lint report, a deadline does not establish that the checks finished. When automatic repair cannot find a usable inference route before starting a repair turn, it is recorded as skipped; the original update check remains the reported failure.

In the Control UI, a failed attempt opens Ask Apus with its recorded details and asks it to investigate before retrying. A lost connection or verification timeout is presented as an unknown outcome. The tab remembers the latest 32 investigated attempt identities, scoped to their Gateway and profile. Status checks, switching between those scopes, and reloading the same tab do not automatically send those investigations again. If the browser cannot read or save that history, the failure details remain visible without an automatic diagnostic request. Ask Apus manually or run apus triage on the host. If the Gateway or agent is unavailable, use apus triage on the Gateway host. Automatic diagnosis keeps your unsent composer draft, including when its conversation session must restart.

Control UI → Settings → Updates keeps the latest recorded attempt visible, including its time, before/after identities, reason code, failing step, and bounded diagnostic detail. A version or revision verification failure stays visible when a status check rereads the same attempt. An unknown verification outcome resolves when the expected version or revision arrives, or when the same attempt reports its final failure or cancellation. A newer recorded attempt can also replace it. Intentional cancellations, already-current installs, and updates still in progress do not start triage.

When a managed-service update cannot hand off to its helper, the saved attempt includes a bounded failure kind and a next step: runtime or spawn failure, permission or service refusal, ownership refusal, payload/control-channel failure, timeout, early helper exit, or preparation failure. If the failure happens before ownership transfer, the recovery outcome records that the Gateway kept serving. A transfer error alone does not establish whether the helper accepted ownership; check apus update status and apus gateway status --deep before retrying.

For a final failed attempt, Report update failure is separate from Retry and Ask Apus. It previews a bounded report containing the Apus version, platform, update target, failed phase, sanitized diagnostics, and verified rollback outcome. The report excludes secrets, tokens, chat content, raw logs, private absolute paths, and recovery commands. Nothing is submitted until an identified administrator confirms that preview. Named administrators receive a prefilled issue to review and submit using their own GitHub account in their browser. This path never invokes the host's GitHub CLI, including for authentication or reconciliation. Connecting My GitHub does not grant host-account publication authority. Only the Gateway owner or an internal system administrator can authorize the existing host GitHub CLI issue flow. Fallback and pending outcomes retain the sanitized report locally; a confirmed issue keeps only its durable issue URL. Apus first makes a silent, read-only request with the active github.com account. A missing CLI or a failed, unavailable, or timed-out authentication check returns a prefilled issue link without starting issue creation. In the Control UI, an interrupted preparation for the recorded attempt can be retried after its local reservation expires. Once issue creation starts, a timeout, signal, nonzero exit, or malformed response without a verified issue URL leaves the attempt pending without a replay link, because the issue-creation outcome may be unknown. The action is tied to one update-attempt identity and cannot submit that attempt twice; reconnecting or refreshing status never reports it automatically. A CLI reporting error returns to the explicit action menu and never starts diagnosis on the user's behalf.

Control UI remediation uses typed product actions only. It leads with an authenticated Gateway or native action when the connected UI has the required capability and scope, preserves confirmations for disruptive operations, and keeps terminal commands as secondary host-side fallbacks. It never parses localized guidance or executes an arbitrary command string.

Recover in the Control UI

  1. Select Check status when the Gateway restarted, disconnected, or did not report a final result. This reads update.status; it does not start another update. Recovery controls stay disabled while the check is pending, and a rejected request appears as an error on the page or in the update dialog. The dialog shows Checking status… while waiting and Status refreshed. after a successful read, even when the recorded failure has not changed. If the Gateway is disconnected, reconnect before checking or retrying.
  2. Open View details and address the recorded failing step. Diagnostic text is bounded and redacted for display; use Gateway logs when more context is required.
  3. Select Retry update only after the cause is resolved. The Control UI uses the normal confirmed update flow and states that running sessions are interrupted while the Gateway restarts.

The controls require a connected Gateway, support for the corresponding typed Gateway method, and administrator scope. When those conditions are not met, use the CLI fallback on the Gateway host.

Doctor cannot enter maintenance during finalization

finalize:doctor can report Doctor could not enter maintenance when a Gateway still owns the selected state directory. A starting Gateway and a healthy serving Gateway retain that ownership for their entire process lifetime; waiting for readiness does not release the lock.

Maintenance admission refusals finish the update with a recorded warning when no data is at risk, including contention from an unknown or non-serving holder. Repair restores a managed service it stopped before reporting that warning. Doctor and plugin maintenance remain pending. Resolve the reported ownership or availability problem, then run apus update repair. Check apus update status --json and apus gateway status --deep for pending migrations, the recorded warning, and current health.

Do not delete lock files to force entry. A dead process releases the physical lock, and lease owners reclaim provably dead identities. A maintenance warning never authorizes concurrent repair or discards recovery backups. Active migration writes, unreadable state, incomplete migrations, and unconfirmed subprocess cleanup retain their failure and recovery guidance.

On Windows, windows-task-inspection-failed means Apus could not query Task Scheduler to verify service absence. Check Task Scheduler availability and the service account's query permissions, then run apus gateway status --deep before retrying. Install failures and update reports include the safe failure category and, when available, a numeric errno, hexadecimal HRESULT, exit code, or timeout budget. These facts appear before the recovery guidance so bounded reports retain them. Preserve those facts when reporting the problem; task definitions and raw native output are excluded.

Node and global install permissions

If package recovery reports nlink=2 (or another value above 1), let updater workers finish, then run apus doctor --fix from the affected installation. Doctor can reclaim an abandoned updater runtime, including one retained in an npm prefix's lib/ directory, and release its extra journal link. Then retry the update. Do not weaken recovery permissions or delete the activation journal; if Doctor reports a live owner or cannot identify the runtime, follow that diagnostic before retrying.

For node-runtime-preflight, upgrade the runtime named in the message to a version satisfying both the candidate's full engine range and the updater's supported Node range. The suggested version is the lowest supported release in that intersection. Follow the recovery steps for the detected runtime manager (nvm, fnm, Volta, or system Node). The next step runs update through the original installation's absolute apus.mjs launcher using the selected Node. It does not rely on apus remaining on PATH after a version-manager switch, or recommend a global install into an uninspected prefix. Extended-stable recovery uses --channel extended-stable so the resolver selects the supported monthly release; other package channels retain the inspected version with --tag. An explicit channel switch is included in recovery because the runtime refusal happens before that preference is saved. If an already-current service is stopped or its definition cannot be refreshed, have its deployment owner select the supported Node in that definition before retrying. Switching the shell runtime does not change a service's pinned Node path.

Keep the same service account, profile, and state/config overrides. Recovery restores the recorded service selectors, including overrides absent from your shell; credentials are not included. The ordinary update invocation rechecks the selected npm prefix and managed service before installation, and retains the original package owner. Its normal runtime selection, service refresh, restart, and verification checks apply. Containers redeploy the target image with the same state/config mounts.

global-install-foreign-destination means the package transaction's destination is foreign or its ownership could not be established. The check uses the resolved installation target: an existing npm-global installation keeps its own prefix when nvm, fnm, Homebrew Node, or an operator's npm prefix change selects a different prefix in the shell. An unrelated installation at that shell prefix does not block an update to the original installation. pnpm global installations retain their pnpm owner and do not use this npm destination check.

The actual destination must be empty, or its canonical package path must match the running installation or selected managed service, with any existing launcher pointing inside that package. An inaccessible or unreadable destination stops the update before staging; an unknown destination is never treated as empty. Restore inspection access or the selected package layout. Ask the deployment owner to verify unreadable layouts and explicitly select the intended installation. The saved outcome and public failure report name the destination prefix, package, launcher, running installation, and classified ownership cause. Public paths replace your home with ~ and redact other home-directory usernames. apus update status and Doctor retain the warning and recovery step. A symlinked prefix that resolves to the same installation is admitted; spelling alone does not make a destination foreign. Switch the runtime back and retry through the retained absolute launcher. Alternatively, with the destination owner's agreement, explicitly select that installation for the intended service using a printed gateway install --force command when available, then update. This changes the service binding; it is not permission to overwrite another deployment's package. A protected service definition uses deployment-owner instructions instead; --force cannot replace a sealed mount. Dry-run returns the same refusal. Recorded attempts remain in update history and are shown by Doctor. If the active CLI and service point at different installations, follow Gateway service recovery to select the intended installation while preserving its state and service account. An older updater that refuses before staging cannot load a candidate's improved diagnostics; resolve its prefix mismatch before retrying the update.

If the ranges do not overlap, install a supported Node and select a compatible Apus target; that candidate cannot run through this updater on a supported Node release. See Node.js.

For global-install-permission-denied, check the named directory and owner. If you own the directory, the message gives a scoped chmod u+rwx command. For an administrator-owned npm prefix, have that account perform the package update or grant the intended updater write access. Keep the Gateway's existing state and configuration; invoking the whole updater with sudo can select a different home and service account. Do not recursively change ownership of a shared system prefix. A personal install can instead use a user-writable npm prefix.

Permission errors discovered after admission carry the same reason. The report's rollback and service-recovery constraints still apply if activation had begun. Inside a container, the same next action also directs you to pull or build the target Apus image and redeploy with the same state/config mounts. Package changes inside a running container are not durable.

System-scope systemd services

A system-scope Gateway service does not prevent a package update when the invoking account can write the installation. Both apus update --yes and the Gateway update action update the package and record a warning with the exact operator restart command, such as sudo systemctl restart apus-gateway.service. The updater does not stop or restart the system service and never invokes sudo. The running Gateway can exit when it detects that its installation has been replaced; restart the unit after the update. Use the unit name printed in your result, including any instance name, then check apus gateway status --deep.

If the same Gateway unit exists in both user and system scopes, updates retain these system-scope restrictions. A differently named Gateway does not create this conflict. Installation-replacement restarts wait for the helper to confirm updater and cleanup settlement; an interrupted helper alone does not permit a restart. Inspect any surviving updater before manually restarting after an interruption.

The restart remains operator-managed even when the updater runs as root: managed update handoffs own user-scope service supervision and recovery, not the system service's lifecycle. Pending Doctor or plugin maintenance is recorded as a warning when it cannot safely run alongside the current Gateway. Run apus update repair after resolving the reported maintenance condition.

If the installation is not writable, the update stops before package mutation with managed-service-handoff-failed and prints the exact package-update and restart commands. Have the installation's owning account run those commands while preserving the Gateway's service account, state, and configuration. Do not run the whole updater under a different home or recursively change ownership of a shared system prefix.

An installed updater that reports Managed update handoff requires a user-scope systemd unit refuses before loading the candidate. A candidate cannot repair that admission decision. Use the manual package-manager procedure once, then restart the named system unit; subsequent updates can use the fixed updater.

Candidate migration rehearsal timeouts

Current updaters give snapshot preparation and each candidate check separate budgets derived from the copied database and plugin sizes. SQLite integrity checks report their database size, elapsed time, and active phase while running. A check that exhausts its budget reports candidate-migration-rehearsal: <step> exceeded budget after <n> s (<last output>). The last output is an observation, not proof that the check completed. Preserve that detail when reporting a slow integrity check or migration. An explicit --timeout still controls the candidate check deadline.

The published 2026.9.3 updater shares a five-minute deadline across snapshot preparation and candidate checks. Increasing --timeout cannot extend that cap. Stopping the Gateway can remove writer contention, but it cannot extend this deadline or eliminate migration work on the copied databases. Candidate-side Doctor improvements can reduce that work; they cannot change the installed updater's deadline.

That release can also report a skipped repair with “could not provide a usable inference route” after a rehearsal timeout. This comes from a broad error handler while preparing automatic repair, not from an inference check performed by the rehearsal. Inspect the original failed step and elapsed time. Current updaters retain the candidate-checks-timeout reason and do not run inference repair for that failure. If the installed updater cannot finish, preserve a verified backup and use the installation owner's manual package-manager procedure with an exact compatible target, then run the target's apus doctor --fix before starting its Gateway.

Published 2026.9.4 on large agent fleets

The published 2026.9.4 updater shares a five-minute deadline across snapshot preparation and candidate checks. Its failure log tail combines output from those checks: Doctor complete. can belong to the preceding repair pass, even when lint is the failed step. The elapsed time in the final log line measures the whole rehearsal; the step duration measures the individual check. A complete lint JSON report is needed to establish that lint finished before termination.

Published Apus 2026.9.4 can spend many minutes preparing model catalogs and chat metadata after its HTTP listener binds. In an instrumented 480-agent control with no update, HTTP checks remained unanswered during 944 seconds of observation; the Gateway then logged ready at 947.5 seconds. Stopping that instance eventually required systemd's existing 5-minute-30-second stop limit. These are measurements of one synthetic fixture, not expected startup budgets.

A second, uninstrumented 480-agent control first passed signed Gateway handshake, serving-build, and health-RPC checks, then lost HTTP responsiveness without any update. The 25-minute post-readiness control completed; sampled failures spanned 24 minutes before a final three-minute serving check also failed. The original PID and installation remained. Shared schema 17, all 481 physical agent databases at schema 19, and config bytes stayed unchanged; captured state-maintenance leases were empty.

The main thread consumed nearly one CPU core. Logs showed existing scheduled review attempts, fleet-wide integrity checks, and memory-plugin startup cleanup errors. This reproduces a published Gateway fleet preparation/background maintenance availability problem independently of updating. Its exact JavaScript hot loop remains unprofiled, and the original failed-update run lacked the live-state evidence needed to exclude an additional state or recovery defect. A retained package, PID, or serviceRestartSafe: true does not establish that the previous Gateway is serving. See the investigation.

Before recovery, preserve the update report and a verified backup. Keep the same service account, profile, package manager, and installation prefix. Have that installation's owner stop the Gateway and other writers before manual replacement. When the installed updater cannot complete, use the manual package-manager procedure with an exact target compatible with the retained state, then run the target's apus doctor --fix before starting its Gateway. If the retained binary cannot read the current state, follow backup recovery; changing schema markers or deleting lease rows does not reverse migrations.

Verify the actual serving version/build through an authenticated Gateway RPC and check /readyz before declaring recovery or removing backups. The plain-start control did not verify these recovery steps or establish that restarting the same 2026.9.4 fleet resolves the failed-update condition.

Headless nodes waiting on 2026.9.6

A headless node running published 2026.9.6 with the default plugins prepares an automatic update but never activates it. Its log shows node auto-update <version> is ready; waiting for active work to finish even when no commands run. That release's bundled File Transfer plugin does not report its idle state, and the running node makes the idle decision before any newer code loads, so a later release cannot repair this automatically.

Update the node once through the normal workflow, then restart it:

apus update
apus node restart

For a foreground node, stop apus node run and start it again instead. Later releases report File Transfer commands as idle between invocations, so subsequent automatic node updates activate normally.

Plugin repair warnings

post-update-plugins / plugin-convergence with post-plugin-doctor-execution-failed can describe a Doctor child failure after the package was already installed. Updated convergence records that execution failure as a warning, retains its exit reason and available plugin diagnostics, and continues to config validation, readiness checks, and Gateway activation. apus update status shows the warning even when the update succeeds. A later failure report keeps it in a separate Warnings section.

A throwing plugin config-repair hook leaves that plugin's input unchanged and names the plugin in its warning. Repair the plugin, then run apus doctor --fix or apus update repair. A live or unverified Gateway and explicit state-migration or config-write refusals remain blocking. So does a Doctor child whose shutdown could not be confirmed: it may still write state. Preserve the backup and resolve that specific refusal before retrying.

Doctor's configured-plugin repair and payload-verification warnings do not block Gateway readiness. A tracked plugin whose payload is unavailable is marked unavailable, and its configuration and pending migration inputs stay preserved. This includes host-link repair failures during updates. apus update status --json lists pending plugin migration warnings, and Doctor reports the affected plugin and repair command. Run apus update repair, then apus doctor --fix to retry after restoring access to the plugin source.

Missing configured plugins.load.paths are availability warnings. The update continues and the Gateway can become ready with the available plugins. The update report and Doctor lint identify the unavailable path with configured-plugin-path-unavailable. Permission, I/O, and other filesystem inspection failures use the distinct configured-plugin-path-inspection-failed warning with the original error code and message. For permission errors, fix permissions on the reported path, then run apus doctor --fix; for other failures, resolve the reported filesystem problem first. Both warnings preserve uninspected configuration and let the update continue.

A load path can contain several plugins or override a bundled plugin, so discovery cannot infer which settings belong to its missing payload. Doctor preserves uninspected plugin settings, channel settings, model selections, and load-path entries instead of treating them as stale or applying another plugin's repair. Restore the path or correct its plugins.load.paths entry, then run apus doctor --fix to resume inspection and repair. Other discovery errors retain their existing diagnostics.

Official version-bound runtime plugins installed through ClawHub use their declared ClawHub source for the new core release cohort. The released 2026.9.4 catalog omitted that source for Codex; the correction is on main in #148518.

Missing temporary plugin captures

An ENOENT path containing apus-plugin-build- can identify a missing runtime source capture even when the installed plugin files still exist. Reloading or replacing that plugin reports the unavailable recovery snapshot as a warning and loads the installed replacement after normal cleanup. Run apus plugins reload <id>, or reinstall the plugin if its installed payload also needs repair. If replacement fails, the missing previous code cannot be restored; healthy plugins retain their available recovery snapshots.

Older releases can reject enable, uninstall, and reinstall while trying to copy that same missing capture. Restart the Gateway through its service owner before retrying, or upgrade the host. See plugin source lifetime.

Database snapshots under continuous writes

Older updaters can report that a database "did not stabilize after 10" attempts while the Gateway keeps writing. Current update schema inspection and rehearsal use a consistent SQLite online backup. Rehearsal progress records copied pages, bytes, and elapsed time in the update ledger.

This cannot retrofit the installed 2026.9.5 driver. For that hop, stop the service through its service owner, run apus update from a separate terminal, then start the service. On a Linux user service, stop it with systemctl --user stop apus-gateway.service. If service shutdown itself hangs, treat that as a separate shutdown problem; do not start a second updater.

Snapshot parse errors from 2026.9.5 and 2026.9.6

An update started from 2026.9.5 or 2026.9.6 can stop with a message such as Update state snapshot failed (exit): Assigning to rvalue (308:4). The installed updater could not parse valid JavaScript that assigns to import.meta.url in a plugin's dependency, for example @jsquash/png or @jsquash/avif. The fix is in the target release, but the installed updater runs this check before the target starts. Disable the plugin for this one update:

apus plugins disable <id>
apus update
apus plugins enable <id>

Updates from the fixed release onward inspect these plugins normally.

The 2026.9.5 updater can also report Cannot use 'import.meta' outside a module for ESM plugins, including bundles using import.meta.dir. Use the same temporary disable/update/enable sequence: a new candidate cannot replace the parser already running in the installed updater. Version 2026.9.6 admits retained import.meta syntax. Current snapshot inventory also records unparseable entries as named plugin warnings instead of aborting the snapshot.

Large model-catalog temporary directories

Older releases can retain several complete plugin copies inside apus-model-catalog-* directories. A scan of only top-level apus-plugin-build-* paths misses those nested copies. Current catalog workers reuse the selected runtime capture for provider discovery and remove their scratch tree when its owner retires.

Upgrade the host, then run apus doctor to inspect legacy captures. On Linux and macOS, apus doctor --fix removes whole legacy catalog trees only during maintenance when no other Apus process is running. Do not delete captures based on their age or absence from open-file or memory-map lists: an idle owner can still need them. Modern captures use SQLite custody to prove retirement. See plugin source lifetime.

Unrelated Node services running node dist/index.js do not count as Apus owners. Doctor resolves generic entrypoints against their installation's package identity and honors Apus service markers. If a live PID cannot be classified, Doctor preserves the captures and reports that PID and the inspection failure (including a missing or unreadable package manifest); this remains a maintenance warning and does not fail the update. Retry apus doctor --fix after resolving the reported inspection problem. On macOS, unreadable arguments from a process owned by another UID do not block cleanup; Doctor records that exclusion once at debug level. Unreadable arguments from the same UID, or an unknown UID, still preserve legacy captures. Doctor also preserves legacy capture roots owned by another UID, including in privileged runs. Managed native captures use their recorded custody and installed-index references rather than the host process census, as they do during startup cleanup. Windows host-wide legacy capture cleanup remains report-only.

Reason codes

Unknown reason codes remain visible. Check the Gateway logs before retrying.

Retained legacy session history

Invalid entries in a legacy sessions.json and malformed JSONL transcripts do not fail an update when Doctor can verify the imported SQLite state and retain the originals. Doctor skips entries without a valid session ID and imports the readable transcript prefix. It reports the file and reason as warnings in its migration report and in apus update status --json, including updates started by older releases that cannot record Doctor warnings themselves.

While a plugin migration is pending, the original files stay in place with a verified import receipt. Repeated Doctor repairs preserve current SQLite edits and deletions. After plugin migration finishes, damaged originals remain in the protected migration archive for manual recovery; update cleanup cannot discard them as fully imported history.

Preserve the named files and your pre-update backup. Inspect them with apus doctor --session-sqlite dry-run --session-sqlite-all-agents --json. Do not overwrite receipt-bound originals to repair them. Changed or newly appeared transcripts can contain history absent from SQLite and still block readiness. Check apus update status --json, stop the Gateway, and run apus doctor --session-sqlite recover --session-sqlite-all-agents before retrying. See session recovery.

CLI fallback

Run these commands on the Gateway host, not on the computer that merely has the Control UI open:

apus update status --json
apus triage

Use apus update --dry-run to preview a new attempt. If a package update failed after installation began, follow the installer recovery steps in Updating.

If the installed CLI is damaged or the filesystem cannot write diagnostics, automatic triage reports that failure and preserves the original update error. Repair the installed command, then run apus triage. Managed updates retain their detached helper log even when the Gateway cannot start; the recorded outcome points to the available diagnostics or the failed collection attempt. Restart notices summarize the diagnostic outcome. Saved artifact paths and exact, installation-specific recovery commands remain in the host command output or managed update helper log rather than the notice sent to an agent or channel.

If the updater crashes or is killed after the Gateway stops, the Gateway stays stopped unless the updater completed and verified recovery. Inspect apus gateway status --deep, repair the reported dependency or installation failure, and rerun apus update. A failed Git dependency install restores and rebuilds the previous runtime before allowing an automatic restart. Restarts after verified recovery still check the installed configuration, service ownership, and Gateway health.

Rollback boundary

Do not restore state as the first response to an update failure. First reinstall known-good code while preserving current state. Restore a verified pre-update state snapshot only when older code cannot read the current config or database. See Rollback.

Support diagnostics

Collect the following without posting credentials, raw config, or unredacted process output: