Model-call observation units

Every apus.model.call span identifies what its lifecycle measures through apus.model_call.observation_unit:

Both units remain model-call spans so trace backends can render model input, output, usage, and hierarchy. Request spans use the API-derived GenAI operation (chat, generate_content, or text_completion), while turn spans use gen_ai.operation.name = invoke_agent. Both contribute to gen_ai.client.operation.duration, where the operation name keeps direct request latency separate from full-turn latency. Apus's OTEL model-call metrics also include apus.model_call.observation_unit; the Prometheus model-call metrics expose the equivalent observation_unit label.

Claude Code CLI model-call fidelity

Claude Code CLI turns emit one synthetic, turn-level apus.model.call span. These are not Anthropic HTTP request spans. They use apus.api = claude-code, apus.model_call.observation_unit = turn, and identify the operation as gen_ai.operation.name = invoke_agent. They identify Apus's CLI boundary through apus.transport:

Claude CLI diagnostics are instantiated only while the process diagnostic dispatcher is enabled and an internal or trusted event listener is attached. With no observability plugin or other listener active, Claude CLI turns skip the synthetic trace hierarchy, content buffers, and diagnostic stream-byte accounting. When content capture is enabled, prompt and system-prompt fields are capped at 128 KiB each; assistant output is capped at 128 KiB across at most 200 envelopes, with 16 KiB and one item reserved for a final visible fallback response. A marker records truncation when the limit is reached.

Apus gives Claude CLI turns the same ownership hierarchy used by other agent runtimes: apus.harness.run (apus.harness.id = claude-cli) contains apus.run, which contains the Claude apus.model.call span. The harness and run spans are synthetic Apus turn boundaries, not Claude Code internal phases. One-shot and managed stdio turns use the same hierarchy; a real fresh-session retry creates another model-call child inside the same Apus run.

The span starts when Apus admits the prepared CLI turn and ends only after that turn succeeds or fails. For managed sessions, an interim success result does not end the span while Claude reports result-holding background agents or workflows; the final post-drain result does. Abort, timeout, process failure, output/parse failure, and other turn failures end the same span with an error.

Claude Code reports per-assistant-message usage and may also report cumulative usage on its terminal result. Apus reply accounting continues to use the last assistant message so existing cost semantics do not change; the turn-level model-call span uses terminal cumulative usage when available, including cache-read and cache-creation tokens.

For these CLI spans, byte and timing fields describe the observable Apus CLI boundary:

With captureContent enabled, the span exports the effective prompt Apus sends to Claude Code and visible assistant text/tool-call identity through gen_ai.input.messages and gen_ai.output.messages. Tool arguments, internal thinking, opaque thinking signatures, tool results, and system prompts are omitted from the Claude assistant envelope. Apus does not claim access to Claude Code's private system prompt, hidden resumed or compacted request payload, native internal tool schemas, raw Anthropic HTTP request, internal retries, upstream request id, or true network TTFB. Because Claude Code does not expose its effective native tool definitions accurately, these spans do not populate gen_ai.tool.definitions.

External Claude harness tool spans remain metadata-only even when tool content capture is enabled. As with every model span, captured Claude CLI content uses the trusted listener-only path and the exporter's existing redaction and size bounds; content remains off by default.

Exported metrics

Gateway RPC

Authenticated Gateway WebSocket requests, including dedicated worker RPCs, emit these metrics while diagnostics and an interested exporter are enabled. They exclude the connection handshake, malformed request frames, and HTTP routes.

Metric Type Measurement
apus.gateway.rpc.requests counter Valid requests received, including requests subsequently rejected
apus.gateway.rpc.first_response_ms histogram Receipt through the first successfully sent response
apus.gateway.rpc.handler_ms histogram Actual handler invocation through return or throw
apus.gateway.rpc.admission_ms histogram Receipt through actual handler invocation
apus.gateway.rpc.queue_wait_ms histogram Operator start-queue or worker frame-queue wait, when applicable
apus.gateway.rpc.outcomes counter Observations by phase and outcome

Request and timing metrics have only apus.gateway.rpc.method: an exact core, worker, or registered plugin method name, other for unregistered generic requests, or unknown for unrecognized dedicated worker RPCs. Outcome metrics have only apus.gateway.rpc.phase and apus.gateway.rpc.outcome, so errors do not multiply every method's series. No request, connection, session, or trace IDs appear in metric attributes.

Operator admission includes authorization, lazy router and handler loading, and start-queue wait. Worker admission includes the socket FIFO wait and outer frame validation. Worker handler duration includes method validation and service invocation. Worker timing starts at the JavaScript socket callback, excluding network delay and event-loop delay before that callback. Queue wait is a subset of admission for handlers that start, separate from command/session lane apus.queue.wait_ms. Handler and admission samples exist only for invoked handlers. Queue wait is recorded when dispatch settles.

Worker inference-start timing measures acceptance and setup, not provider completion. Long worker computer and session operations retain their handler timing after releasing the socket FIFO. Worker dispatch outcomes describe return or throw. Connection closure can suppress a response after successful execution and does not by itself prove execution cancellation.

A sent response means the WebSocket sender accepted the frame, not that the client received it. Early acknowledgments count as the first response; later responses do not add another sample. Unavailable or suppressed sends contribute outcomes but no first-response sample. A handler may return before a retained callback sends its response, and detached agent work can continue afterward. These durations measure elapsed time, including asynchronous waits, rather than CPU time or event-loop blocking time.

Observations use the bounded diagnostic queue. Check apus.diagnostic.async_queue.dropped before treating counts or latency distributions as complete during saturation.

Model usage

Message flow

Talk

Queues and sessions

Session liveness telemetry

A processing session does not age toward the built-in liveness threshold while Apus observes reply, tool, status, block, or ACP runtime progress. Typing keepalives do not count as progress, so a silent model or harness can still be detected.

Apus classifies sessions by the work it can still observe:

Recovery emits structured session.recovery.requested and session.recovery.completed events. Diagnostic session state is marked idle only after a mutating recovery outcome (aborted or released) and only if the same processing generation is still current.

Only session.stuck emits the apus.session.stuck counter, the apus.session.stuck_age_ms histogram, and the apus.session.stuck span. Repeated session.stuck diagnostics back off while the session remains unchanged, so dashboards should alert on sustained increases rather than every heartbeat tick. For the config knob and defaults, see Configuration reference.

Liveness warnings also emit:

The CPU ratio measures whole-process CPU usage in core equivalents, including worker and native threads, and can exceed 1. Event-loop delay and utilization measure the main thread separately. See CPU pressure and event-loop delay.

Gateway event-loop observation windows

These metrics use the existing diagnostics plugin setup and require metrics to be active. Each accepted health-monitor window is recorded once, so a later healthy readiness result does not erase an earlier high-delay observation. Health and scrape reads do not commit or reset samples. The process-wide observations carry no request trace context and create no spans or logs, including with a preloaded SDK.

The monitor samples elapsed event-loop intervals every 20 milliseconds and completes windows after at least one second or sooner for a delay warning. Ordinary window resets preserve the pending interval even when health is read before an overdue sample. Counts and quantiles describe completed windows and their maxima, not individual stalls or the sampled delay distribution's overall p99. Intentional monitor resets discard unfinished windows; collection does not backfill periods without an interested exporter. Diagnostic queue drops, SDK/export failures, and restarts limit coverage. Use the represented-duration counter and exporter/drop telemetry to assess it. Readiness decisions and persistent liveness-warning thresholds are unchanged. For pull metrics and example queries, see Prometheus event-loop windows.

Harness lifecycle

Tool execution and loop detection

Exec

Diagnostics internals (memory, payloads, exporter health)

GC duration uses Node.js performance entries and is exported only when metrics are enabled. It is not CPU time or a guaranteed stop-the-world pause. Observation starts when the existing diagnostics heartbeat sees an interested consumer; registration after startup can wait until its next 30-second tick, with no backfill. Diagnostics disable/shutdown disconnects immediately. See GC duration coverage and correlation limits.