Short guide to verify Gateway and channel health without guessing. It covers the CLI health checks, the HTTP check endpoints, the dedicated health command, and uptime monitoring.

Quick checks

For Discord and other chat providers, session rows are not socket liveness. apus sessions, Gateway sessions.list, and the agent sessions_list tool read stored conversation state. A provider can reconnect and show healthy channel status before any new session row is materialized. Use the channel status and health commands above for live connectivity checks.

Per-agent session counts and recent activity include only that agent's sessions, even when agents share a SQLite session store. Status counts each physical store once in its aggregate. The top-level health session summary represents the default agent, or the first configured agent when there is no default; it is not a fleet total. A running Gateway serves clean health and status session summaries from its resident session-row projection. Store hydration and exact dirty-row refreshes retain the existing read-only SQLite fallback.

Deep diagnostics

apus health --json reports modelRuntime.degraded: true when a large fleet's model preparation exceeds the startup budget. pendingAgents names the agents still preparing and stage identifies the current acquisition phase. The Gateway remains running and completed agents remain usable. Background preparation clears the degraded status when all runtimes are ready.

If a newer model-runtime refresh replaces preparation in progress, startup follows the admitted replacement through preparation and publication. Cancelling the retired acquisition does not stop the Gateway; genuine replacement failures still propagate.

Health and status collection groups fast session-store reads into short work slices, keeping busy background preparation from delaying every individual read. Slow reads finish their transaction before yielding to other Gateway work.

Health monitor config

Inbound ingress health

Channel connectivity and inbound admission are separate failure domains. A channel can hold a healthy transport connection — sending replies normally — while its durable ingress queue is unavailable, so not a single inbound message is admitted.

HTTP checks

The Gateway exposes three unauthenticated GET/HEAD check pairs:

Endpoints Meaning Use
/health, /healthz The HTTP server is live. Process liveness and restart decisions.
/startup, /startupz Startup sidecars and agent database inspection/preparation have settled, and the Gateway is not draining. Channel health is not consulted. Startup-phase monitoring and update verification.
/ready, /readyz Startup sidecars have settled, the Gateway is not draining, required agent databases have no confirmed failures, and configured channel accounts pass deep readiness checks. Traffic admission and operator monitoring.

/startupz returns 503 with status: "starting" while startup sidecars or agent database inspection/preparation are pending, 503 with status: "draining" during drain, and 200 with status: "started" otherwise. After sidecars settle, pending inspection for any agent, including an optional agent, reports pendingReason: "agent-database-inspection". This lets apus update candidate verification wait within its startup budget before checking readiness. A settled inspection failure does not keep startup pending; readiness and per-agent admission report the failure.

A default or system agent database with a confirmed admission failure keeps readiness false, with failing: ["agent-database:<id>"] and the exact admission reason and repair hint in agentDatabases. Pending startup inspection is reported there with code agent-database-inspection-pending while the Gateway can report ready and serve the Control UI and healthy agents. Agent-scoped sessions.list, plus sessions.resolve, sessions.describe, and models.list, wait up to 20 seconds for their selected agents to finish startup preparation, then recheck request authorization before reading. Requests targeting healthy agents proceed independently. Unscoped session lists continue returning admitted agents while other stores prepare. Writes still require completed admission. RPC requests refused for pending inspection return UNAVAILABLE with retryable: true and retryAfterMs: 250; callers can retry the same request after that delay. Inspection failures and ownership mismatches remain non-retryable. A refused optional agent can remain isolated while healthy agents serve requests. Gateway ready announcements use the same readiness decision.

A broken Telegram or other channel account can also make /readyz return 503 while /startupz remains started. Neither check replaces the other: startup completion alone does not certify agent or channel availability.

Disabled, unconfigured, and ordinarily unlinked accounts do not fail channel readiness. An enabled, configured account with a recorded terminal disconnect, blocked lifecycle, or unavailable ingress still fails its health check even if it becomes unlinked. For example, a terminal WhatsApp logout makes /readyz return 503 with failing: ["whatsapp"] in detailed responses. Terminal disconnects and blocked accounts require operator action; the health monitor does not restart them.

Remote unauthenticated startup responses contain only ok and status. Local-direct and authenticated callers also receive version, uptimeMs, and pendingReason while startup is pending. Readiness details follow the same local-or-authenticated gate because they can name failing subsystems.

Shared-state integrity failure

A terminal shared-state admission failure immediately makes /ready and /readyz return 503, including failures discovered by a SQLite worker after startup. Detailed responses include failing: ["state-database"] and stateDatabase.reason with the recorded refusal. This bypasses cached channel health; checks read the admission owner's recorded result without querying SQLite.

/healthz still reports HTTP liveness. Supervisors that need to detect a Gateway that is running but cannot admit work must monitor /readyz.

Plugin replacement recovery

During plugin replacement or recovery, /readyz returns 503. Detailed responses include failing: ["plugin-reload"] and a pluginReload object with the affected pluginIds, the current phase (reloading, recovering, or failed), and any recovery deadlineAtMs and actionable reason. These owner-reported facts bypass the channel readiness cache, so a failed replacement cannot appear as only a generic channel outage or stale healthy result.

The health monitor does not spend channel restart attempts while replacement holds channel admission paused. After successful rollback, the previous plugin configuration restarts its channels and ordinary readiness checks resume. If automatic recovery reaches its deadline, phase: "failed" retains the failure reason and next action. Admission pauses are released, allowing the monitor to restart callable channels; a plugin whose admitted work or cleanup still owns resources requires the reported repair or retry before it can restart. See Config hot reload for the recovery contract.

CPU pressure and event-loop delay

Detailed readiness can include the latest completed eventLoop diagnostic snapshot. The sampler owns observation windows; health reads do not reset a pending measurement. No snapshot is available until the first window completes. Its cpuCoreRatio measures user and system CPU time across the whole Gateway process, including worker and native threads, divided by elapsed wall time. The unit is core equivalents: 1 means one CPU core fully occupied over the interval, and parallel work can produce values above 1. It is not a percentage of the host's total CPU capacity.

The health RPC also reads the latest completed sample when returning a cached summary or publishing a newly collected one. Slow channel checks do not freeze its CPU and delay readings. If the sampler resets, health responses omit eventLoop until a new window completes instead of reviving a cached sample.

The optional cpuBreakdown separates independent native counters:

Thread values use the same core-equivalent unit as cpuCoreRatio. Worker reads are asynchronous and must complete within 100 ms of each sampling boundary; these are not atomic cross-thread measurements. The residual can vary with measurement skew. A timed-out request never delays the event-loop sample, and at most one native request is outstanding per tracked worker, even across monitor resets. Startup, worker creation/exit, host CPU topology changes, counter resets, and collection failures require fresh baselines before publishing the affected rates. Missing fields mean unavailable, not zero.

The main-thread and host counters are collected independently on Node and Bun. Tracked worker CPU and the residual are omitted on Bun: its worker API can report zero when native counter collection fails. Unsupported or failed native APIs on any platform leave the corresponding field absent.

The Control UI's CPU box reads the same sampler through system.info.eventLoop. Its detail overlay shows host usage separately from the process and thread breakdown. Process and thread percentages use 100% for one fully occupied core; host usage uses 100% for all reported logical CPUs. Values show a dash until their first complete measurement, or while unavailable.

Event-loop delay and utilization describe the main thread separately. A cpu degradation reason reports process CPU pressure with delay co-evidence; it does not identify the thread consuming CPU or prove a main-thread hang. Inspect the delay measurements alongside CPU pressure. The eventLoop diagnostic does not change the readiness result by itself.

Uptime monitoring

External uptime monitoring services should use the dedicated /health endpoint, not /v1/chat/completions.

When no x-apus-session-key header or user field is provided, /v1/chat/completions generates a new random session for each request. Monitoring services that ping every 15 minutes create ~96 sessions/day, each consuming 4-22KB. Over time this causes session store bloat and can lead to context window overflow.

Monitoring service setup examples

When something fails

Dedicated "health" command

apus health asks the running gateway for its health snapshot (no direct channel sockets from the CLI). By default it returns a fresh cached gateway snapshot and the gateway refreshes that cache in the background; --verbose forces a live check instead. Connections and cached health reads share a one-minute background refresh cadence, so repeated diagnostic connections do not each rebuild the health snapshot. Explicit live checks and refreshes for missing or stale health still run immediately. Snapshots describe loaded and configured channels. Stored credentials alone do not activate a channel or add it to Gateway health; use channel setup to enable it. The command reports linked creds/auth age when available, per-channel check summaries, session-store summary, and check duration. Live checks use bounded account concurrency and a Gateway-owned deadline, so one slow account returns a structured timeout while completed sibling results remain available. The command exits non-zero if the gateway is unreachable or the Gateway call itself times out.

Queue warnings

A successful health RPC reports top-level ok: true. That value means the Gateway produced the snapshot; it does not mean every delivery queue is clear. Check deliveryQueues.ingressPressure for durable inbound lanes that may be blocking later events. The field is omitted when no pressured lanes are found.

Ingress pressure uses conservative built-in diagnostic thresholds, not authoritative retry or claim policy for any plugin. A durable lane appears only when an active pending or claimed row has either reached at least eight attempts and has a recorded delivery error, or a claimed row has not refreshed its claim for 30 minutes. Ordinary retries 1-7 are absent. Claim-recovery increments without a recorded error are also absent, while live claims stay absent because their claim timestamp is refreshed. Rows without a durable lane key are omitted because they cannot prove that later events are blocked; runtime persists a derived lane after a real derived-lane retry.

Each result is grouped by channel account and reports pressured lane, pending, claimed, and blocked counts plus the oldest affected receive time. All active rows in a pressured lane contribute to those counts. The snapshot never includes lane IDs, event IDs, payloads, claim owners or tokens, recorded errors, or session and target identifiers.

Options:

The health snapshot includes: ok (boolean), ts (timestamp), durationMs (check time), per-channel status, agent availability, session-store summary, and optional delivery-queue warnings.

Related