The Test Harness

Smoke

The fast per-push sanity layer. Industry term, originally from electronics — "plug it in and see if smoke comes out." Catches the most common silent failure mode: a write API that quietly stopped persisting.

• 4 write-flow tests via the real API
  Marketplace purchase · DM boon grant · DM XP apply · session journal patch. Each test truncates the test DB, seeds known fixtures, POSTs to the real route, and reads the DB back to confirm persistence.
• Coverage diff at the end
  Greps app/api for POST/PUT/PATCH/DELETE handlers and diffs against tests/coverage-snapshot.txt. New write routes that don’t yet have a flow test get printed by name so they don’t slip in silently.
• Wall time
  ~3s after the build, ~2 min total including the build.
• Failure surface
  Red X on the commit, plus GitHub’s default email.

Canary + UI flows

The broader nightly sweep. Named after coal-mine canaries — a cheap, fast signal that catches obvious breakage before deeper inspection. Two specs share one job.

• Canary: page render sweep
  Globs every app/**/page.tsx, substitutes fixture IDs into dynamic segments, navigates each route in a real Chromium browser via Playwright, asserts status < 500. 4xx is fine — it just means auth/not-found, the page didn’t crash. 40 routes currently covered.
• UI flows: 4 browser-driven flows
  Same persistence paths smoke hits via API, but driven through clicks and typing in a real browser. Catches "API saves but the button is broken / handler missing / element no longer in the DOM" — a regression class smoke can’t see.
• Wall time
  ~10s for both specs combined.
• Failure surface
  Resend email to kevintraywick@gmail.com with subject [nightly-qc] Failed on <branch>. Manual force-fail via gh workflow run nightly-qc.yml -f force_fail=true.

Sheet sync

The deepest, slowest layer. Confirms the character sheet import end-to-end still works through the real Claude Haiku Vision pipeline.

• What it does
  POSTs the April Lane fixture PDF (fictional 5e character, no real-world identifying info) to /api/player-sheets/import, then /confirm. Asserts sheet_imported_at is stamped and ≥1 of {species, class, level} is populated.
• Why weekly
  Hits Anthropic’s API and costs real money per run. Daily would be wasteful and the failure modes here change slowly (provider response shape, our extraction prompt, the spend cap).
• Wall time
  ~4.5s including the Vision call.
• Failure surface
  Resend email with subject [sheet-sync] Failed on <branch>.

The spec lives at docs/plans/2026-05-15-001-test-persistence-harness-plan.md and includes a retrospective listing the 6 schema-drift and ordering bugs the harness caught during its own bring-up.