Use this page for day-1 startup and day-2 operations of the Gateway service.
Deep troubleshooting
Symptom-first diagnostics with exact command ladders and log signatures.
Configuration
Task-oriented setup guide + full configuration reference.
Secrets management
SecretRef contract, runtime snapshot behavior, and migrate/reload operations.
Secrets plan contract
Exact `secrets apply` target/path rules and ref-only auth-profile behavior.
5-minute local startup
Start the Gateway
apus gateway --port 18789
# debug/trace mirrored to stdio
apus gateway --port 18789 --verbose
# force-kill listener on selected port, then start
apus gateway --force
Verify service health
apus gateway status
apus status
apus logs --follow
Healthy baseline: Runtime: running, a successful connectivity check, and a Capability line that matches what you expect. Use apus gateway status --require-rpc for read-scope RPC proof, not just reachability.
Validate channel readiness
apus channels status --probe
With a reachable gateway this runs live per-account channel checks and optional audits. If the gateway is unreachable, the CLI falls back to config-only channel summaries.
Gateway config reload watches the active config file path (resolved from profile/state defaults, or APUS_CONFIG_PATH when set). Default mode is gateway.reload.mode="hybrid". After the first successful load, the running process serves the active in-memory config snapshot; a successful reload swaps that snapshot atomically.
Runtime model
- One always-on process for routing, control plane, and channel connections.
- Single multiplexed port for:
- WebSocket control/RPC
- HTTP APIs (
/v1/models,/v1/embeddings,/v1/chat/completions,/v1/responses,/tools/invoke) - Plugin HTTP routes, such as optional
/api/v1/admin/rpc - Control UI and hooks
- Default bind mode:
loopback. Inside a detected container environment the effective default isauto(resolves to0.0.0.0for port-forwarding), unless Tailscale serve/funnel is active, which always forcesloopback. - Auth is required by default. Shared-secret setups use
gateway.auth.token/gateway.auth.password(orAPUS_GATEWAY_TOKEN/APUS_GATEWAY_PASSWORD), and non-loopback reverse-proxy setups can usegateway.auth.mode: "trusted-proxy".
OpenAI-compatible endpoints
Apus's highest-leverage compatibility surface:
GET /v1/modelsGET /v1/models/{id}POST /v1/embeddingsPOST /v1/chat/completionsPOST /v1/responses
Why this set matters:
- Most Open WebUI, LobeChat, and LibreChat integrations check
/v1/modelsfirst. - Many RAG and memory pipelines expect
/v1/embeddings. - Agent-native clients increasingly prefer
/v1/responses.
/v1/models is agent-first: it returns apus, apus/default, and apus/<agentId> for every configured agent. apus/default is the stable alias that always maps to the configured default agent. Send x-apus-model when you want a backend provider/model override; otherwise the selected agent's normal model and embedding setup stays in control.
All of these run on the main Gateway port and use the same trusted operator auth boundary as the rest of the Gateway HTTP API.
Admin HTTP RPC (POST /api/v1/admin/rpc) is a separate, default-off plugin route for host tooling that cannot use WebSocket RPC. See Admin HTTP RPC.
Port and bind precedence
| Setting | Resolution order |
|---|---|
| Gateway port | --port → APUS_GATEWAY_PORT → gateway.port → 18789 |
| Bind mode | CLI/override → gateway.bind → loopback (or auto in containers) |
Installed gateway services record the resolved --port in supervisor metadata. After changing gateway.port, run apus doctor --fix or apus gateway install --force so launchd/systemd/schtasks starts the process on the new port.
Gateway startup uses the same effective port and bind when it seeds local Control UI origins for non-loopback binds. For example, --bind lan --port 3000 seeds http://localhost:3000 and http://127.0.0.1:3000 before runtime validation runs. Add any remote browser origins, such as HTTPS proxy URLs, to gateway.controlUi.allowedOrigins explicitly.
Hot reload modes
gateway.reload.mode |
Behavior |
|---|---|
off |
No config reload |
hybrid (default) |
Hot-apply when safe, restart when required |
The earlier hot and restart modes were retired in v2026.7.2-beta.4, stable from v2026.8.1. apus doctor --fix maps both to hybrid.
Operator command set
apus gateway status
apus gateway status --deep # adds a system-level service scan
apus gateway status --json
apus gateway install
apus gateway restart
apus gateway stop
apus secrets reload
apus logs --follow
apus doctor
gateway status --deep is for extra service discovery (LaunchDaemons/systemd system units/schtasks), not a deeper RPC health check.
Multiple gateways (same host)
Most installs should run one gateway per machine. A single gateway can host multiple agents and channels. You only need multiple gateways when you intentionally want isolation or a rescue bot.
Useful checks:
apus gateway status --deep
apus gateway probe
What to expect:
gateway status --deepcan reportOther gateway-like services detected (best effort)and print cleanup hints when stale launchd/systemd/schtasks installs are still around.gateway probecan warn aboutmultiple reachable gateway identitieswhen distinct gateways answer, or when Apus cannot prove reachable targets are the same gateway. An SSH tunnel, proxy URL, or configured remote URL to the same gateway is one gateway with multiple transports, even when transport ports differ.- If that is intentional, isolate ports, config/state, and workspace roots per gateway.
Checklist per instance:
- Unique
gateway.port - Unique
APUS_CONFIG_PATH - Unique
APUS_STATE_DIR - Unique
agents.defaults.workspace
Example:
APUS_CONFIG_PATH=~/.apus/a.json APUS_STATE_DIR=~/.apus-a apus gateway --port 19001
APUS_CONFIG_PATH=~/.apus/b.json APUS_STATE_DIR=~/.apus-b apus gateway --port 19002
Detailed setup: /gateway/multiple-gateways.
Remote access
Preferred: Tailscale/VPN. Fallback: SSH tunnel.
ssh -N -L 18789:127.0.0.1:18789 user@gateway-host
Then connect clients locally to ws://127.0.0.1:18789.
SSH tunnels do not bypass gateway auth. For shared-secret auth, clients still
must send token/password even over the tunnel. For identity-bearing modes,
the request still has to satisfy that auth path.
See: Remote Gateway, Authentication, Tailscale.
Supervision and service lifecycle
Native service-control commands receive only the operating-system environment needed for executable lookup, account identity, locale, and service-manager routing. They do not inherit application credentials or arbitrary shell variables. The Gateway payload and its installed service definition retain their separately configured environments.
Use supervised runs for production-like reliability.
macOS (launchd)
apus gateway install
apus gateway status
apus gateway restart
apus gateway stop
Use apus gateway restart for restarts. Do not chain apus gateway stop and apus gateway start as a restart substitute.
On macOS, gateway stop uses launchctl bootout and verifies that the LaunchAgent is unloaded and its process has exited before reporting success. This removes the LaunchAgent from the current boot session without persisting a disable, so KeepAlive auto-recovery still works after unexpected crashes and gateway start re-enables cleanly. To also persistently suppress auto-respawn across reboots, pass --disable: apus gateway stop --disable.
If shutdown cannot be verified, the command fails with the exact launchctl bootout gui/<uid>/<label> command to run from an external terminal in the service owner's logged-in macOS session. A free Gateway port alone does not prove that the service is stopped.
LaunchAgent labels are ai.apus.gateway (default) or ai.apus.<profile> (named profile). apus doctor audits and repairs service config drift.
On macOS, service restarts preserve multiline environment values, including PEM keys, embedded quotes, and trailing newlines. If an older restart already truncated a value in the generated service environment, another restart cannot reconstruct it. Restore the complete value in its authored configuration or environment and reinstall with apus gateway install --force.
Existing system LaunchDaemons
Apus installs and manages a per-user LaunchAgent. It does not install or manage system LaunchDaemons. If a custom LaunchDaemon already uses the same gateway label, Apus refuses to write, start, restart, or repair a user LaunchAgent because two KeepAlive managers can repeatedly restart the same gateway.
The ownership check reads launchctl print system/<label> and also checks installed plists under /Library/LaunchDaemons. It fails closed when system ownership cannot be verified, and --force does not bypass it. apus gateway status reports a loaded same-label system job; add --deep to scan installed system service files.
The runtime and standalone updater parse captured plist bytes with the native parser. If endpoint protection denies pathname parsing during a detached restart, its ownership scan tries a bounded read and parses the captured bytes instead. Actual permission-denied reads are skipped, while malformed data and other read failures still block activation. Loaded same-label jobs remain blocked; an unloaded same-label plist hidden by read denial cannot be detected. The detached restart fallback uses macOS's /usr/bin/perl; if that reader is unavailable, the scan still refuses unverifiable activation. This does not change endpoint-protection policy or suppress its alerts.
Choose one lifecycle owner before retrying:
- To keep the custom system LaunchDaemon, remove any competing user LaunchAgent and set
APUS_SERVICE_REPAIR_POLICY=externalwhen running Doctor so it remains diagnostic-only for service lifecycle. - To return to the supported user LaunchAgent, unload the system job with
sudo launchctl bootout system/<label>, remove or relocate its actual plist, sign in to the macOS desktop as the target user, then runapus gateway install.
For the default profile, <label> is ai.apus.gateway. Named profiles use ai.apus.<profile>.
Linux (systemd user)
apus gateway install
systemctl --user enable --now apus-gateway[-<profile>].service
apus gateway status
For persistence after logout, enable lingering:
sudo loginctl enable-linger $(whoami)
On a headless server without a desktop session, also make sure XDG_RUNTIME_DIR is set (export XDG_RUNTIME_DIR=/run/user/$(id -u)) before retrying systemctl --user commands.
Service inspection preserves an explicit DBUS_SESSION_BUS_ADDRESS that reaches
the user manager. Otherwise it tries $XDG_RUNTIME_DIR/bus, then the private
manager socket for inspection. Install, status, and update admission reuse the
selected route; gateway status --deep shows it. Update admission rechecks routes
that timed out during earlier discovery. A socket's existence alone does
not replace a working custom bus. If no route reaches the manager, check
XDG_RUNTIME_DIR, log in once
or enable lingering, and verify systemctl --user status. On Debian/Ubuntu,
dbus-user-session provides the user bus; start it with
systemctl --user start dbus.socket if needed. An absent unit is safe to install;
an unreadable existing definition must be repaired by its owner first.
Manual user-unit example when you need a custom install path:
[Unit]
Description=Apus Gateway
After=network-online.target
Wants=network-online.target
StartLimitBurst=10
StartLimitIntervalSec=300
[Service]
ExecStart=/usr/local/bin/apus gateway --port 18789
Restart=always
RestartSec=5
RestartPreventExitStatus=78
TimeoutStopSec=330
TimeoutStartSec=30
SuccessExitStatus=0 143
OOMPolicy=continue
KillMode=mixed
[Install]
WantedBy=default.target
TimeoutStopSec=330 covers the Gateway's maximum 315-second stop drain plus a 15-second cleanup and exit margin. The Gateway clamps its drain to the installed unit's effective stop timeout; see Systemd stop deadlines. To inspect the current managed unit body, run systemctl --user cat apus-gateway.service (or systemctl --user cat apus-gateway-<profile>.service for a named profile).
Windows (native)
apus gateway install
apus gateway status --json
apus gateway restart
apus gateway stop
Native Windows managed startup uses a Scheduled Task named Apus Gateway
(or Apus Gateway (<profile>) for named profiles). If Scheduled Task
creation is denied, Apus falls back to a per-user Startup-folder launcher
that points at gateway.cmd inside the state directory.
Linux (system service)
Use a system unit for multi-user/always-on hosts.
Start with the user-unit example, install it under
/etc/systemd/system/apus-gateway[-<profile>].service, adjust
ExecStart= if your apus binary lives elsewhere, and add User= to
its [Service] section:
[Service]
User=<user>
Replace <user> with the non-root account that owns the Apus state and
configuration. A system unit without User= runs as root. Running the Gateway
and its agent commands as root is unsafe and unsupported for this setup.
When Group= is omitted, systemd uses the selected account's primary group.
By default, User= also supplies that account's HOME, which Apus uses
for normal state and configuration lookup. For intentional custom locations,
set APUS_STATE_DIR and APUS_CONFIG_PATH in the unit environment.
Do not copy configuration into root's home as a workaround. On a single-user
host, the user unit above with loginctl enable-linger is the supported way
to keep the Gateway running without a login session.
Do not also let apus doctor --fix install a user-level gateway service for the same profile/port. Doctor refuses that automatic install when it finds a system-level Apus gateway service; use APUS_SERVICE_REPAIR_POLICY=external when the system unit owns the lifecycle.
apus gateway status --deep inspects the installed system unit and reports
systemd system. Run Doctor from the non-root User= account with the same state
and config paths. For offline repair, stop the unit through its system service
owner first, run apus doctor --fix, then start the unit through that owner.
Doctor can verify a stopped system unit without rewriting its definition or
creating a competing user service. An unavailable manager or an unverified
service account still blocks maintenance.
After writing the unit, reload systemd and enable it:
sudo systemctl daemon-reload
sudo systemctl enable --now apus-gateway[-<profile>].service
Invalid configuration errors exit with code 78. Linux systemd units use RestartPreventExitStatus=78 to stop relaunching until the config is fixed. launchd and Windows Task Scheduler do not have an equivalent per-exit-code stop rule, so the Gateway also persists rapid unclean boot history and suppresses channel/provider account auto-start after repeated startup failures. In that safe mode the control plane still starts for inspection and repair, main-session restart recovery pauses until the full breaker window drains and then resumes automatically in the same process, config hot reloads and secrets.reload refuse automatic channel restarts, and an explicit operator channels.start request can override the suppression. Step-by-step recovery lives in Restart recovery.
Dev profile quick path
apus --dev setup
apus --dev gateway --allow-unconfigured
apus --dev status
Defaults include isolated state/config and base gateway port 19001.
Protocol quick reference (operator view)
- First client frame must be
connect. - Gateway returns a
hello-okframe with asnapshot(presence,health,stateVersion,uptimeMs) pluspolicylimits (maxPayload,maxBufferedBytes,tickIntervalMs). hello-ok.features.methods/eventsare a conservative discovery list, not a generated dump of every callable helper route.- Requests:
req(method, params)→res(ok/payload|error). - Common events include
connect.challenge,agent,chat,session.message,session.operation,session.tool, opt-insession.approval,sessions.changed,presence,tick,health,heartbeat, pairing/approval lifecycle events, andshutdown.
Agent runs are two-stage:
- Immediate accepted ack (
status:"accepted") - Final completion response (
status:"ok"|"error"), with streamedagentevents in between.
See full protocol docs: Gateway Protocol.
Operational checks
Liveness
- Open WS and send
connect. - Expect
hello-okresponse with snapshot.
Readiness
apus gateway status
apus channels status --probe
apus health
Gap recovery
Events are not replayed. On sequence gaps, refresh state (health, system-presence) before continuing.
Common failure signatures
| Signature | Likely issue |
|---|---|
refusing to bind gateway ... without auth |
Non-loopback bind without a valid gateway auth path |
another gateway instance is already listening / EADDRINUSE |
Port conflict |
Gateway start blocked: set gateway.mode=local |
Config set to remote mode, or gateway.mode is missing from a damaged config |
unauthorized during connect |
Auth mismatch between client and gateway |
For full diagnosis ladders, use Gateway Troubleshooting.
Safety guarantees
- Gateway protocol clients fail fast when Gateway is unavailable (no implicit direct-channel fallback).
- Invalid/non-connect first frames are rejected and closed.
- Graceful shutdown emits
shutdownevent before socket close.
Related
- Configuration
- Gateway troubleshooting
- Background exec and process tool — the agent-facing exec and process tool, not a Gateway service control
- Health
- Doctor
- Authentication
- Remote access
- Secrets management
- CLI backends — running an external CLI agent as a Gateway backend