Prove behavior at the owning boundary

Prefer focused integration tests through real production owners, with deterministic external I/O fixtures, not mocked internal decisions or whole-product boots. Keep independent unit contracts and the cost budget.

Use synthetic content and keep credentials out of captures. Select failure cases from the changed contract, not a universal matrix. If proof needs many private controls, inspect fragmented ownership rather than adding test-only exports.

Test Temp Directories

Use the shared helpers in test/helpers/temp-dir.ts for test-owned temporary directories so ownership is explicit and cleanup stays in the test lifecycle:

import { afterEach } from "vitest";
import { useAutoCleanupTempDirTracker } from "../helpers/temp-dir.js";

const tempDirs = useAutoCleanupTempDirTracker(afterEach);

it("uses a temp workspace", () => {
  const workspace = tempDirs.make("apus-example-");
  // use workspace
});

useAutoCleanupTempDirTracker(afterEach) intentionally exposes no manual cleanup method - Vitest owns cleanup after each test. Older lower-level helpers (makeTempDir, cleanupTempDirs, createTempDirTracker) still exist for tests that have not migrated; avoid new usage of them and avoid new bare fs.mkdtemp* calls unless a test is explicitly verifying raw temp-dir behavior. When a bare temp dir is genuinely needed, add an auditable allow comment with a reason:

// apus-temp-dir: allow verifies raw fs cleanup behavior
const workspace = fs.mkdtempSync(prefix);

node scripts/report-test-temp-creations.mjs reports new bare temp-dir creation and new manual shared-helper usage in added diff lines, without blocking existing cleanup styles. It follows the same test-path classification as scripts/changed-lanes.mjs and skips the shared helper implementation itself. check:changed runs this report for changed test paths as a warning-only CI signal (GitHub warning annotations, not failures).

Copy fixture trees whose files a test later executes directly (stubs on PATH, shebang wrappers, native binaries) with copyTreeCloseOnExec from test/helpers/close-on-exec-copy.ts, not a recursive fs.cpSync without a filter. On Node 24 that copy path opens files without close-on-exec, so a child forked by another Vitest thread mid-copy keeps the file writable and a later execve fails with ETXTBSY.

Agent reliability evals (skills)

We already have a few CI-safe tests that behave like "agent reliability evals":

What's still missing for skills (see Skills):

Future evals should stay deterministic first:

Cost budget

CI selects tests by their owning area, so per-PR cost is paid repeatedly. Budgets, measured with pnpm test <file> --maxWorkers=1 on one worker:

withTestTimeout and raceWithTimeoutResult are grandfathered wall-clock races; check:test-timeout-race-ratchet keeps their per-file counts in config/test-timeout-race-baseline.txt shrink-only. Wait for the owned completion signal with awaitGateBeforeSettlement(gate, operation, message) or withinTest(work, signal) from test/helpers/promise.ts, or use vi.useFakeTimers() through the owner's injected clock seam. After removing sites, run pnpm check:test-timeout-race-ratchet --prune to shrink the baseline.

Module mocks and export completeness

New first-party vi.mock and vi.doMock factories should preserve the real module's exports when the fixture only needs to override a few functions:

vi.mock("./runtime.js", async (importOriginal) => ({
  ...(await importOriginal<typeof import("./runtime.js")>()),
  start: vi.fn(),
}));

Use pass-through for export completeness only. importOriginal and vi.importActual can return a separate module instance for stateful singletons; they do not guarantee shared state or lifecycle identity. Keep a closed mock when isolating real state or initialization is the fixture's purpose, and explain that contract on the line immediately above the mock call:

// mock-isolation: Keep the database and process-wide cache outside this fixture.
vi.mock("./runtime.js", () => ({ start: vi.fn() }));

pnpm check:test-mock-exports checks literal first-party module registrations, including relative, workspace-package, and TypeScript-path aliases. Factories without a recognizable real-module return/spread need the annotation, including indirect factories the syntax check cannot prove. Existing unannotated factories have an exact source-target and token-fingerprint baseline; new or changed factories cannot borrow another site's allowance. After removing or annotating existing factories, run pnpm check:test-mock-exports --prune to shrink that baseline. This guard runs with the existing CI ratchets and check:changed; it does not rewrite tests.

Raw SQLite state access

closeApusStateDatabaseForTest() closes native handles synchronously, but worker-backed state writes (plugin state, deferred plugin migrations, and other worker stores) keep a worker connection whose retirement only starts at that call. Its final close checkpoints and deletes the WAL under an exclusive lock at an arbitrary later time. Before opening the database with a raw DatabaseSync, or copying, hashing, or snapshotting its files, await closeApusStateDatabaseAsync() (or closeStateDatabaseForTest() from src/test-utils/database-cleanup.ts, which also clears failure latches). Otherwise the raw connection can fail with SQLITE_BUSY, or the snapshot can change underneath the test. PRAGMA locking_mode=EXCLUSIVE; BEGIN EXCLUSIVE on a raw connection proves that no other connection remains.

Skills watchers

skills.status and skill snapshot preparation start real @openclaw/fs-safe watchers. In shared-worker lanes, the non-isolated runner closes any watchers a file leaves open and fails that file with skills watchers failed; otherwise their re-armed timers land on a later file's fake clock and abort its vi.runAllTimersAsync(). Close them in afterEach with closeSkillsWatchers(true), or set skills.load.watch: false when the test does not exercise watching.

Flake triage

A failure without a related change is a defect. Never re-run, re-push, or refresh to get green.

  1. Reproduce in the failing shard's file order first (the plan's file list is in the job log), then alone. Order-only failures are shared-state leaks from an earlier file.
  2. Classify: fixture (temp state, ports, cwd, env, module singletons), ordering (assertion before the owned completion signal), or product (a real race).
  3. Fix at the owner. Product races get a regression that fails on the original defect; fixture leaks are fixed in the fixture owner, not in the failing test.
  4. Proof bar: 20 clean standalone runs of the file, 3 clean runs of the original shard, and the owner's sibling tests. Record the root cause in the PR.
  5. If the owner is another lane or PR, cite that fix; it is the only reason to proceed on a red job.

Adding regressions (guidance)

For inventory-growth and capacity regressions, pass bounded synthetic inventories and fixed timing data through the real planner. Put the fixture at the capacity boundary and preserve coverage, ownership, and execution-budget assertions. Keep real-checkout inventory coverage in its integration tests instead of rebuilding the growing repository plan for every synthetic variation.

When you fix a provider/model issue discovered in live: