Eval harness
Regression-grade evals that run in CI, not in prod. A model swap fails the build first, not a customer's session.
What it is
@caisson/ai-evals is a regression gate for prompt and model changes: defineEval() scores a version-bound dataset through a grader taxonomy, then compareToBaseline() fails the build if the mean score, any individual scorer, or a Wilson confidence floor drops below the committed baseline, offline and deterministic, no live provider call inside CI.
What ships in the module
Version-bound eval runs
defineEval({ name, promptVersionId, cases, scorers, threshold }) grades every case with every scorer and returns a deterministic EvalRun. Each dataset carries a promptVersionId FK, so a score is always attributable to one immutable prompt version, never a floating "current prompt."
Six graders, two classes
Deterministic exactGrader/regexGrader/jsonShapeGrader/schemaGrader run pure, offline, no model. judgeGrader routes through the Judge port for model-graded scoring. injectionGrader is its own fail-closed substring-denial class that a graded input can never talk its way past, an empty rubric throws instead of silently passing.
Committed-baseline regression gate
gateAgainstBaseline() compares each run to a committed JSON baseline and fails closed: a missing baseline, a score below threshold, or any scorer regression blocks the gate. BLESS=1 bun run eval is the one sanctioned path to rewrite it, mirroring the golden-fixture discipline in @caisson/testing.
Offline judge via cassette replay
cassetteJudge() replays recorded verdicts from a committed cassette, zero network, zero provider secret, in CI. An unrecorded case id is a hard cassette-miss error, not a silent pass. recordingJudge() wraps a real local judge to mint a fresh cassette for review before it's committed.
Wilson confidence floor and exit classifier
wilsonLowerBound() threads an opt-in confidence floor into the baseline gate so a small lucky-draw sample can't pass as reliable. classifyExit() tags WHY a run exited (error, timeout, budget-exhausted, refusal, empty-output) as a signal orthogonal to pass/fail.
Reflexivity queue for judge/human disagreement
captureDisagreement() enqueues a case only when the model verdict and a human verdict disagree; consolidateReflexivityQueue() dedupes and caps the list for operator review. Nothing here auto-writes a committed dataset, merging a candidate back in stays a human act.
Browser-safe entry point
Import @caisson/ai-evals/browser inside a client bundle for the gate's rules with no file I/O: the baseline boundary schema, compareToBaseline, the pre-bless eligibility check, the bless merge, and wilsonLowerBound. gateAgainstBaseline stays on the main entry because it reads and writes the committed baseline file, and every browser-entry export is also on the main entry.
export function compareToBaseline(
run: EvalRun,
baseline: BaselineFile,
): BaselineComparison {
const findings: RegressionFinding[] = [];
if (run.score + EPS < run.threshold) {
findings.push({
kind: "below-threshold",
actual: run.score,
baseline: run.threshold,
detail: `score ${run.score} < threshold ${run.threshold}`,
});
}
const prior = baseline.evals[run.name];
if (prior === undefined) {
findings.push({
kind: "missing-baseline",
actual: run.score,
detail: `no committed baseline for eval "${run.name}" — bless to record it`,
});
return { eval: run.name, passed: false, findings, blessed: false };
}- The EPS tolerance on the threshold compare (run.score + EPS < run.threshold) avoids a false regression from float rounding noise, not just a strict less-than.
- A missing baseline returns its own missing-baseline finding immediately, it's never silently treated as a pass.