QA-specific runners

These commands sit beside the main test suites when you need QA-lab realism.

CI runs QA Lab in dedicated workflows. Agentic parity is nested under QA-Lab - All Lanes and release validation, not a standalone PR workflow. Broad validation should use Full Release Validation with rerun_group=qa-parity for parity or rerun_group=qa-live for live QA. The direct Apus Release Checks child alone may use rerun_group=qa as a manual aggregate of both groups. Stable/full, soak-enabled, and explicit qa-live release checks include the QA-live Matrix and Telegram lanes. Bounded beta-publish all without soak runs parity but defers those live lanes to postpublish-confidence. QA-Lab - All Lanes runs nightly on main and from manual dispatch with the mock parity lane, live Matrix lane, Convex-managed live Telegram lane, and Convex-managed live Discord lane as parallel jobs. Scheduled QA and selected release checks run the catalog-derived Matrix selection through the shared live adapter. Release transport checks use mock-openai/gpt-5.6-luna so they stay deterministic and avoid normal provider-plugin startup. These live transport gateways disable memory search; memory behavior stays covered by the QA parity suites.

Full release live media shards use ghcr.io/apus/apus-live-media-runner:ubuntu-24.04, which already has ffmpeg and ffprobe. Docker live model/backend shards use the shared ghcr.io/apus/apus-live-test:<sha> image built once per selected commit, then pull it with APUS_SKIP_DOCKER_BUILD=1 instead of rebuilding inside every shard.

• pnpm apus qa suite
  • Runs repo-backed QA scenarios directly on the host.
  • Writes top-level qa-evidence.json, qa-suite-summary.json, and qa-suite-report.md artifacts for the selected scenario set, including mixed flow, Vitest, and Playwright scenario selections.
  • When dispatched by pnpm apus qa run --qa-profile <profile>, embeds the selected taxonomy profile scorecard in the same qa-evidence.json. smoke-ci writes slim evidence (evidenceMode: "slim", no per-entry execution). release covers the curated release-readiness slice; all selects every active maturity category and targets explicit QA Profile Evidence workflow dispatches when a full scorecard artifact is needed.
  • Runs multiple selected scenarios in parallel by default with isolated gateway workers. qa-channel defaults to concurrency 4 (bounded by the selected scenario count). Use --concurrency <count> to tune the worker count, or --concurrency 1 for the older serial lane. Each worker owns a stable named profile as well as separate home, state, and config paths, so its CLI bootstrap cannot select the operator's installed Gateway service. Parent profiles and runtime environment patches do not override that worker identity. Child temporary files and default compiler caches stay in the worker's temporary root and are removed after its processes stop. Parent temporary paths and runtime environment patches do not redirect this scratch storage. APUS_QA_KEEP_TEMP=1 retains that root for debugging. If the controller dies, the current Gateway's parent watchdog exits without deleting runtime files that descendants may still use. A surviving owner or host maintenance must confirm that those writers stopped before removing the retained roots.
  • Exits non-zero when any scenario fails. Use --allow-failures for artifacts without a failing exit code.
  • Supports provider modes live-frontier, mock-openai, and aimock. aimock starts a local AIMock-backed provider server for experimental fixture and protocol-mock coverage without replacing the scenario-aware mock-openai lane.
• pnpm apus qa coverage --match <query>
  • Searches scenario IDs, titles, surfaces, coverage IDs, docs refs, code refs, plugins, and provider requirements, then prints matching suite targets.
  • Use this before a QA Lab run when you know the touched behavior or file path but not the smallest scenario. Advisory only - still choose mock, live, Multipass, Matrix, or transport proof from the behavior being changed.
• pnpm test:plugins:kitchen-sink-live
  • Runs the live OpenAI Kitchen Sink plugin gauntlet through QA Lab. Installs the external Kitchen Sink package, verifies the plugin SDK surface inventory, checks /healthz and /readyz, records gateway CPU/RSS evidence, runs a live OpenAI turn, and checks adversarial diagnostics. Requires live OpenAI auth such as OPENAI_API_KEY. In hydrated Testbox sessions it automatically sources the Testbox live-auth profile when the apus-testbox-env helper is present. The helper is for live lanes; non-live Vitest workers drop credential env variables. Wrapper Testbox runs redact hydrated profile values from streamed output, which also cleans Crabbox failure bundles. Direct crabbox or blacksmith testbox runs and downloaded artifacts are not stream-redacted; Vitest reports pass through the redacting reporter.
• pnpm test:gateway:cpu-scenarios
  • Runs the gateway startup bench plus a small mock QA Lab scenario pack (channel-chat-baseline, memory-failure-fallback, gateway-restart-inflight-run) and writes a combined CPU observation summary under .artifacts/gateway-cpu-scenarios/.
  • Flags only sustained hot CPU observations by default (--cpu-core-warn, default 0.9; --hot-wall-warn-ms, default 30000), so short startup bursts are recorded as metrics without looking like the minutes-long gateway peg regression.
  • Runs against built dist artifacts; run a build first when the checkout does not already have fresh runtime output.
• pnpm apus qa suite --runner multipass
  • Runs the same QA suite inside a disposable Multipass Linux VM, keeping the same scenario-selection and provider/model flags as qa suite.
  • Live runs forward the QA auth inputs practical for the guest: env-based provider keys, the QA live provider config path, and CODEX_HOME when present.
  • Output dirs must stay under the repo root so the guest can write back through the mounted workspace.
  • Writes the normal QA report + summary plus Multipass logs under .artifacts/qa-e2e/....
• pnpm qa:lab:up
  • Starts the Docker-backed QA site for operator-style QA work.
• pnpm test:docker:npm-onboard-channel-agent
  • Builds an npm tarball from the current checkout, installs it globally in Docker, runs non-interactive OpenAI API-key onboarding, configures Telegram by default, verifies the packaged plugin runtime loads without startup dependency repair, runs doctor, and runs one local agent turn against a mocked OpenAI endpoint.
  • Use APUS_NPM_ONBOARD_CHANNEL=discord to run the same packaged-install lane with Discord.
• pnpm test:docker:session-runtime-context
  • Runs a deterministic built-app Docker smoke for embedded runtime context transcripts. Verifies hidden Apus runtime context persists as a non-display custom message instead of leaking into the visible user turn, then seeds an affected broken session JSONL and verifies apus doctor --fix rewrites it to the active branch with a backup.
• pnpm test:docker:npm-telegram-live
  • Installs an Apus package candidate in Docker, runs installed-package onboarding, configures Telegram through the installed CLI, then reuses the live Telegram QA lane with that installed package as the SUT Gateway.
  • The trusted checkout owns the QA harness source, taxonomy, scenarios, dependencies, and private SDK build. The installed package remains the absolute CLI, Gateway, and bundled-plugin runtime under test, and its CLI writes the package candidate's persisted auth state.
  • Defaults to APUS_NPM_TELEGRAM_PACKAGE_SPEC=apus@beta; set APUS_NPM_TELEGRAM_PACKAGE_TGZ=/path/to/apus-current.tgz or APUS_CURRENT_PACKAGE_TGZ to test a resolved local tarball instead of installing from the registry.
  • Emits repeated RTT timing in qa-evidence.json by default with APUS_NPM_TELEGRAM_RTT_SAMPLES=20. Override APUS_NPM_TELEGRAM_RTT_SAMPLES, APUS_NPM_TELEGRAM_RTT_TIMEOUT_MS, or APUS_NPM_TELEGRAM_RTT_MAX_FAILURES to tune the run. APUS_NPM_TELEGRAM_RTT_CHECKS accepts zero or exactly one canonical Telegram QA scenario id. When omitted, the normal lane samples channel-canary; focused non-RTT scenario runs stay check-free. An explicit RTT scenario is included in scenario selection automatically, so callers do not need to repeat it in APUS_NPM_TELEGRAM_SCENARIOS. Multiple ids fail immediately, while unknown or inapplicable ids fail canonical scenario validation. The package runner promotes the selected RTT scenario once to the first position before the remaining taxonomy-backed fail-fast release scenarios. Checks continue in its most recently observed conversation and thread, using the leased primary participant. The first sample starts a new message; later samples chain their own replies rather than a reply observed by another scenario participant. Delivery-only scenarios use their observed outbound route and need no additional catalog metadata.
  • Uses the same Convex-leased Test Server userbot credentials as pnpm apus qa telegram. Set APUS_QA_CONVEX_SITE_URL and the secret for the selected role. The Docker wrapper selects Convex by default.
  • The wrapper validates Convex credential env on the host before Docker build/install work. Set APUS_NPM_TELEGRAM_SKIP_CREDENTIAL_PREFLIGHT=1 only when deliberately debugging pre-credential setup.
  • APUS_NPM_TELEGRAM_CREDENTIAL_ROLE=ci|maintainer overrides the shared APUS_QA_CREDENTIAL_ROLE for this lane only. With no role, the wrapper uses ci in CI and maintainer outside CI.
  • GitHub Actions exposes this lane as the manual maintainer workflow NPM Telegram Beta E2E. It does not run on merge. The workflow uses the qa-live-shared environment and Convex CI credential leases. Set its optional rtt_scenario input to select the repeated RTT scenario, or leave it empty for the default behavior above. Enable allow_older_binary_destructive_actions only for intentional historical downgrade or recovery proof; it remains false by default.
• GitHub Actions also exposes Package Acceptance for side-run product proof against one candidate package. It accepts a Git ref, published npm spec, HTTPS tarball URL plus SHA-256, trusted-URL policy, or tarball artifact from another run (source=ref|npm|url|trusted-url|artifact), uploads the normalized apus-current.tgz as package-under-test, then runs the existing Docker E2E scheduler with smoke, package, product, full, or custom lane profiles. Set telegram_mode=mock-openai or live-frontier to run the Telegram QA workflow against the same package-under-test artifact.
  • For legacy Telegram topic bindings across a published-driver update, select suite_profile=telegram, telegram_mode=mock-openai, and the single scenario telegram-published-upgrade-bindings. Supply package_spec as an exact published baseline, such as apus@2026.9.6, and resolve the candidate through source=ref or a verified tarball artifact. This scenario installs the baseline before leasing Test Server credentials, lets it spawn a thread-bound child that takes over the forum topic, and runs that installation's normal apus update against the candidate. It checks that the topic routes back to the parent session after activation and another Gateway restart while the child session stays intact, including all three orderly shutdowns. Raw credential, session, and transport state stays in container scratch; uploaded evidence contains the package identities and redacted outcome only.
  • Latest beta product proof:

gh workflow run package-acceptance.yml --ref main \
  -f source=npm \
  -f package_spec=apus@beta \
  -f suite_profile=product \
  -f telegram_mode=mock-openai

• Exact tarball URL proof requires a digest and uses the public URL safety policy:

gh workflow run package-acceptance.yml --ref main \
  -f source=url \
  -f package_url=https://registry.npmjs.org/apus/-/apus-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

• Enterprise/private tarball mirrors use an explicit trusted-source policy:

gh workflow run package-acceptance.yml --ref main \
  -f source=trusted-url \
  -f trusted_source_id=enterprise-artifactory \
  -f package_url=https://packages.example.internal:8443/artifactory/apus/apus-VERSION.tgz \
  -f package_sha256=<sha256> \
  -f suite_profile=package

source=trusted-url reads .github/package-trusted-sources.json from the trusted workflow ref and does not accept URL credentials or a workflow-input private-network bypass. If the named policy declares bearer auth, configure the fixed APUS_TRUSTED_PACKAGE_TOKEN secret.

• Artifact proof downloads a tarball artifact from another Actions run:

gh workflow run package-acceptance.yml --ref main \
  -f source=artifact \
  -f artifact_run_id=<run-id> \
  -f artifact_name=<artifact-name> \
  -f suite_profile=smoke

• pnpm test:docker:plugins

  • Packs and installs the current Apus build in Docker, starts the Gateway with OpenAI configured, then enables bundled channel/plugins via config edits.
  • Verifies setup discovery leaves unconfigured downloadable plugins absent, the first configured doctor repair installs each missing downloadable plugin explicitly, and a second restart does not run hidden dependency repair.
  • Also installs a known older npm baseline, enables Telegram before running apus update --tag <candidate>, and verifies the candidate's post-update doctor cleans legacy plugin dependency debris without a harness-side postinstall repair.

• pnpm test:parallels:npm-update

  • Runs the native packaged-install update smoke across Parallels guests. Each selected platform first installs the requested baseline package, then runs the installed apus update command in the same guest and verifies the installed version, update status, gateway readiness, and one local agent turn.

  • Use --platform macos, --platform windows, or --platform linux while iterating on one guest. Use --json for the summary artifact path and per-lane status.

  • The OpenAI lane uses openai/gpt-5.6-luna for the live agent-turn proof by default. Pass --model <provider/model> or set APUS_PARALLELS_OPENAI_MODEL to validate another OpenAI model.

  • Wrap long local runs in a host timeout so Parallels transport stalls cannot consume the rest of the testing window:

    timeout --foreground 150m pnpm test:parallels:npm-update -- --json
    timeout --foreground 90m pnpm test:parallels:npm-update -- --platform windows --json
    

  • The script writes nested lane logs under /tmp/apus-parallels-npm-update.*. Inspect windows-update.log, macos-update.log, or linux-update.log before assuming the outer wrapper is hung.

  • Windows update can spend 10 to 15 minutes in post-update doctor and package update work on a cold guest; that is still healthy when the nested npm debug log is advancing.

  • Do not run this aggregate wrapper in parallel with individual Parallels macOS, Windows, or Linux smoke lanes. They share VM state and can collide on snapshot restore, package serving, or guest gateway state.

  • The post-update proof runs the normal bundled plugin surface because capability facades such as speech, image generation, and media understanding load through bundled runtime APIs even when the agent turn itself only checks a simple text response.

• pnpm apus qa aimock

  • Starts only the local AIMock provider server for direct protocol smoke testing.

• pnpm apus qa buzz

  • Runs the Buzz live QA lane against a real relay room using dedicated driver and SUT identities.
  • Local runs use --credential-file <path> with relayUrl, roomId, driverPrivateKey, and sutPrivateKey. Closed relays may also need driverAuthTag and sutAuthTag. Hosted relays require wss://; ws:// is accepted only for loopback development relays.
  • Defaults to mock-openai and runs canary and mention-gating scenarios through the real Buzz plugin path.
  • Supports --credential-source convex with a pooled kind: "buzz" row. Both public keys must be relay/room members, and the SUT must have the Bot room role. Never use a human owner or admin private key.

• pnpm apus qa matrix

  • Runs the Matrix live QA lane against a disposable Docker-backed Tuwunel homeserver. Source-checkout only - packaged installs do not ship qa-lab.
  • Full CLI, profile/scenario catalog, env vars, and artifact layout: Matrix smoke lanes.

• pnpm apus qa telegram

  • Runs the Telegram live QA lane on Telegram's Test Server with one Convex-leased SUT bot and one independent TDLib user session.
  • Uses --credential-source convex by default and rejects env. Provide APUS_QA_CONVEX_SITE_URL and the secret for the selected --credential-role.
  • Defaults cover canary, mention gating, command addressing, /status, bot-to-bot mentioned replies, and core native command replies. mock-openai defaults also cover deterministic reply-chain and Telegram final-message streaming regressions. Use --list-scenarios for optional checks such as session_status.
  • Exits non-zero when any scenario fails. Use --allow-failures for artifacts without a failing exit code.
  • The leased user drives and observes the shared Test Server group. No production Telegram account or bot-to-bot observer is used.
  • Writes a Telegram QA report, summary, and qa-evidence.json under .artifacts/qa-e2e/.... Replying scenarios include RTT from driver send request to observed SUT reply.

Live transport lanes share one standard contract so new transports do not drift; the per-lane coverage matrix lives in QA overview - Live transport coverage. qa-channel is the broad synthetic suite and is not part of that matrix.

Shared Telegram credentials via Convex (v1)

When --credential-source convex (or APUS_QA_CREDENTIAL_SOURCE=convex) is enabled for live transport QA, QA lab acquires an exclusive lease from a Convex-backed pool, heartbeats that lease while the lane is running, and releases the lease on shutdown. Telegram always uses this source. The section name predates Buzz, Discord, Slack, and WhatsApp support; the lease contract is shared across kinds.

Reference Convex project scaffold: qa/convex-credential-broker/

Required env vars:

• APUS_QA_CONVEX_SITE_URL (for example https://your-deployment.convex.site)
• One secret for the selected role:
  • APUS_QA_CONVEX_SECRET_MAINTAINER for maintainer
  • APUS_QA_CONVEX_SECRET_CI for ci
• Credential role selection:
  • CLI: --credential-role maintainer|ci
  • Env default: APUS_QA_CREDENTIAL_ROLE (defaults to ci in CI, maintainer otherwise)

Optional env vars:

• APUS_QA_CREDENTIAL_LEASE_TTL_MS (default 1200000)
• APUS_QA_CREDENTIAL_HEARTBEAT_INTERVAL_MS (default 30000)
• APUS_QA_CREDENTIAL_ACQUIRE_TIMEOUT_MS (default 90000)
• APUS_QA_CREDENTIAL_HTTP_TIMEOUT_MS (default 15000)
• APUS_QA_CONVEX_ENDPOINT_PREFIX (default /qa-credentials/v1)
• APUS_QA_CREDENTIAL_OWNER_ID (optional trace id)
• APUS_QA_ALLOW_INSECURE_HTTP=1 allows loopback http:// Convex URLs for local-only development.

APUS_QA_CONVEX_SITE_URL should use https:// in normal operation.

Maintainer admin commands (pool add/remove/list) require APUS_QA_CONVEX_SECRET_MAINTAINER specifically.

CLI helpers for maintainers:

pnpm apus qa credentials doctor
pnpm apus qa credentials add --kind telegram --payload-file qa/telegram-credential.json
pnpm apus qa credentials list --kind telegram
pnpm apus qa credentials remove --credential-id <credential-id>

Use doctor before live runs to check the Convex site URL, broker secrets, endpoint prefix, HTTP timeout, and admin/list reachability without printing secret values. Use --json for machine-readable output in scripts and CI utilities.

Default endpoint contract (APUS_QA_CONVEX_SITE_URL + /qa-credentials/v1). Requests authenticate with an Authorization: Bearer <role secret> header; bodies below omit that header:

• POST /acquire
  • Request: { kind, ownerId, actorRole, leaseTtlMs, heartbeatIntervalMs }
  • Success: { status: "ok", credentialId, leaseToken, payload, leaseTtlMs?, heartbeatIntervalMs? }
  • Exhausted/retryable: { status: "error", code: "POOL_EXHAUSTED" | "NO_CREDENTIAL_AVAILABLE", ... }
• POST /payload-chunk
  • Request: { kind, ownerId, actorRole, credentialId, leaseToken, index }
  • Success: { status: "ok", index, data }
• POST /heartbeat
  • Request: { kind, ownerId, actorRole, credentialId, leaseToken, leaseTtlMs }
  • Success: { status: "ok" } (or empty 2xx)
• POST /release
  • Request: { kind, ownerId, actorRole, credentialId, leaseToken }
  • Success: { status: "ok" } (or empty 2xx)
• POST /admin/add (maintainer secret only)
  • Request: { kind, actorId, payload, note?, status? }
  • Success: { status: "ok", credential }
• POST /admin/remove (maintainer secret only)
  • Request: { credentialId, actorId }
  • Success: { status: "ok", changed, credential }
  • Active lease guard: { status: "error", code: "LEASE_ACTIVE", ... }
• POST /admin/list (maintainer secret only)
  • Request: { kind?, status?, includePayload?, limit? }
  • Success: { status: "ok", credentials, count }

Payload shape for Telegram kind:

• { groupId: string, driverToken: string, sutToken: string }
• groupId must be a numeric Telegram chat id string.
• admin/add validates this shape for kind: "telegram" and rejects malformed payloads.

Broker-validated multi-channel payloads:

• Buzz: { relayUrl: string, roomId: string, driverPrivateKey: string, sutPrivateKey: string, driverAuthTag?: string, sutAuthTag?: string }
• Discord: { guildId: string, channelId: string, driverBotToken: string, sutBotToken: string, sutApplicationId: string, voiceChannelId?: string }
• WhatsApp: { driverPhoneE164: string, sutPhoneE164: string, driverAuthArchiveBase64: string, sutAuthArchiveBase64: string, groupJid?: string }

Slack lanes can also lease from the pool, but Slack payload validation currently lives in the Slack QA runner rather than the broker. Use { channelId: string, driverBotToken: string, sutBotToken: string, sutAppToken: string } for Slack rows.

Adding a channel to QA

The architecture and scenario-helper names for new channel adapters live in QA overview - Adding a channel. The minimum bar: implement the transport runner on the shared qa-lab host seam, add an adapterFactory for shared scenarios, declare qaRunners in the plugin manifest, mount as apus qa <runner>, and author scenarios under qa/scenarios/.