Set up OpenTelemetry export

Quick start

apus plugins install clawhub:@openclaw/diagnostics-otel

{
  plugins: {
    allow: ["diagnostics-otel"],
    entries: {
      "diagnostics-otel": { enabled: true },
    },
  },
  diagnostics: {
    enabled: true,
    otel: {
      enabled: true,
      endpoint: "http://otel-collector:4318",
      protocol: "http/protobuf",
      serviceName: "apus-gateway",
      traces: true,
      metrics: true,
      logs: true,
      sampleRate: 0.2,
      flushIntervalMs: 60000,
    },
  },
}

Or enable the plugin from the CLI: apus plugins enable diagnostics-otel.

With the plugin loaded, changes to diagnostics.otel hot-reload only its exporter service. The previous generation unsubscribes and flushes before the replacement starts with the new endpoint, headers, sampling, and signal settings. Other plugin services, channels, and Gateway connections stay running. A cleanup or startup failure requests Gateway recovery instead of leaving a partially replaced exporter.

diagnostics.enabled also hot-applies to the shared dispatcher and its heartbeat. The Gateway owns that process-wide heartbeat; stopping a channel leaves it running. Standalone hosts using the plugin SDK own startDiagnosticHeartbeat and stopDiagnosticHeartbeat for their process, rather than each channel owning them. Disabling it stops diagnostic sampling and recovery listeners; enabling it starts them again. Preloaded OpenTelemetry SDKs keep ownership of their providers and transport: these changes do not shut down or reconfigure the host SDK.

diagnostics.otel.protocol accepts only http/protobuf. If a persisted config, including a value supplied through ${VAR} interpolation, still resolves this field to the retired grpc value, run apus doctor --fix. Doctor repairs directly authored values and the deepest internal single-file include that solely owns the changed diagnostics.otel keys, including an unambiguous nested include chain. For root includes, actual array-entry includes, include arrays, sibling overrides, same-path or ancestor merges, changes spanning ownership boundaries, external include targets, an owning file that still authors a nested $include directive, or another ambiguous source, Doctor leaves the files unchanged and lists the candidate source file or files to edit manually. When the same run also needs a root-owned repair, such as a legacy agent roster, Doctor refuses that write; the refused write leaves every file unchanged (earlier writes in the same run stay saved), and Doctor names the boundary to repair by hand before rerunning, plus the included file or files when the root file authors that boundary's $include (an agent-roster boundary is named without its file).

When diagnostics.otel.protocol is unset, each plugin-owned OTLP signal first checks its nonblank OTEL_EXPORTER_OTLP_*_PROTOCOL value, then OTEL_EXPORTER_OTLP_PROTOCOL, then defaults to http/protobuf. Doctor does not rewrite process environment variables. An unsupported value disables only that plugin-owned OTLP signal; supported sibling signals continue, as does the stdout branch of logsExporter: "both". Preloaded trace and metric SDKs own their own transport selection and are not rejected by this plugin.

Which processes export

Gateway starts the exporter at startup and exports from the Gateway process for every run it executes, including apus agent turns dispatched to it.
One-shot local runs (apus agent --local) execute in the CLI process. When OTel export is configured and the plugin is enabled, that same CLI process starts one exporter instance for the run and flushes buffered spans, metrics, and logs before the process exits. The CLI waits at most 5 seconds for the diagnostic-event queue to drain and 10 more for the flush, so an unreachable collector cannot hold the command open. A collector that accepts the connection but never answers can still delay exit until the exporter's own request timeout (OTEL_EXPORTER_OTLP_TIMEOUT). Plugin registration resources remain open until exporter cleanup finishes, even when either wait times out. In JSON output mode, these one-shot runs suppress only the stdout JSONL log sink so command stdout stays reserved for the JSON response; OTLP traces, metrics, and logs continue when configured.
apus agent exec also runs the agent embedded in the CLI process, but does not start this exporter, so its runs export no telemetry. Dispatch through the Gateway, or use apus agent --local, when you need traces from a headless run.

Exporter health

apus doctor and apus status --all show a bounded, redacted snapshot of the running Gateway's latest trusted exporter state for each signal and transport. For diagnostics-otel, the snapshot distinguishes:

OTLP/HTTP protobuf with an endpoint supplied by config or an OTEL_* environment fallback.
OTLP/HTTP protobuf using the exporter dependency's default endpoint because no endpoint was supplied.
Stdout log export.
Trace or metric export owned by an externally preloaded OpenTelemetry SDK.

OTLP export failure and recovery transitions are recorded from the exporter's final result callback, after dependency-owned retries finish. A retryable response that later succeeds is therefore not reported as a failure. Startup, log preparation or emit, export, and shutdown failures use fixed reason categories rather than raw errors.

The snapshot never includes endpoint values, headers, certificates, payloads, or raw error messages. Transport is retained only in this local health projection. It is not added to the existing apus.telemetry.exporter.events metric attributes, and existing Prometheus label sets are unchanged.

Without an exporter

Keep diagnostics events available to plugins or custom sinks without running diagnostics-otel:

{
  diagnostics: { enabled: true },
}

For targeted debug output without raising logging.level, use diagnostics flags. Flags are case-insensitive and support wildcards (telegram.* or *):

{
  diagnostics: { flags: ["telegram.http"] },
}

Or as a one-off env override:

APUS_DIAGNOSTICS=telegram.http,telegram.payload apus gateway

Flag output goes to the standard log file (logging.file) and is still redacted by the always-on log redaction policy. Full guide: Diagnostics flags.

Disable

{
  diagnostics: { otel: { enabled: false } },
}

Or leave diagnostics-otel out of plugins.allow, or run apus plugins disable diagnostics-otel.

When the plugin would otherwise own NodeSDK, keep propagation available while disabling every plugin-owned exporter, listener, health route, and stdout sink:

OTEL_SDK_DISABLED=true apus gateway