AI eval regression suite (golden set, deterministic graders over the turn trace, floor ratchet)

Generated from pattern-library/registry.json — do not edit by hand; change the registry and run bun run patterns:build.

Category: AI · Tier: deep · Maturity: proven (verified 2026-10-23 @ cf85a2c-dirty) · Risk: low — the suite is advisory until the floor is raised; a flaky phrase check shows as a named failing fact, not a vague score

A committed golden set of questions, each with the facts its recorded turn must show — citations surfaced not recalled, the right tool called, no forbidden tool, an approval stop, no invented command — graded by pure functions over the turn trace, with a pass-rate floor that only rises; live runs go through the app's own routes against a local dev server.

When to use: Use when prompts, retrieval, tools or models change often and unit tests mock the model, so quality regressions would ship unnoticed.

Requires: ai-provider, postgres

Avoid when

• Turns are not recorded — without a trace of tools and citations the graders have nothing deterministic to read
• The quality question is open-ended style or tone that no fact in the trace can decide

Alternatives

• LLM-as-judge suites (promptfoo, Braintrust, LangSmith) — prefer when the facts to check are judgments no trace field records, and the judge is calibrated against human labels
• Offline retrieval benchmarks (Acc@1, Recall@k) — prefer when only the retrieval layer changes and no model call is needed to score it

Docs

• docs/blueprint/ai/evals.md#ai-eval-regression-suite — Golden set, graders, floor ratchet, where it runs (GitHub · GitLab)

Code

• src/lib/server/ai/eval/eval-case.ts — The case and expectation schema (GitHub · GitLab)
• src/lib/server/ai/eval/graders.ts — One pure grader per expectation kind (GitHub · GitLab)
• src/lib/server/ai/eval/scorecard.ts — Case pass, run pass rate, floor check (GitHub · GitLab)
• src/lib/server/ai/eval/run-eval.ts — HTTP client over the app's own routes; runEval (GitHub · GitLab)
• src/lib/server/ai/eval/golden-cases.json — The golden set (GitHub · GitLab)
• scripts/ai/eval.ts — bun run ai
  [--check] (GitHub · GitLab)

Tests

• src/lib/server/ai/eval/graders.test.ts — Graders over the committed turn fixtures and synthetic failures; scorecard and floor (GitHub · GitLab)
• src/lib/server/ai/eval/golden-cases.test.ts — Schema, unique ids, tools exist on the case's surface (GitHub · GitLab)
• src/lib/server/ai/eval/run-eval.test.ts — The runner against a stand-in HTTP server: cookie, CSRF header, envelopes, failed cases (GitHub · GitLab)

Invariants

• Every expectation is a deterministic check over the recorded turn; no model grades a model.
• A case names only tools that exist on its surface.
• A run passes only at or above the committed floor, and the floor never moves down.
• A case whose turn cannot be obtained fails with its reason and the run continues.

Emulation notes

• Grade the trace (tools, citation matches, outcome), not the prose — those facts are stable across runs where wording is not.
• Run evals through the app's real endpoints as a signed-in user, so the graded turn is exactly what a user's turn records.
• Start the floor at the first honest run and log every raise; a floor set by hope fails the first time and gets ignored.

Depends on

• Turn trace (recorded account of every AI turn) + turn graph
• Performance budget and ratchet (targets aspire, ceilings hold)

Machine-readable record: ai-eval-regression-suite in pattern-library/registry.json.