Evals that actually catch regressions

Most eval suites are built the same way: collect a few hundred representative examples, score them, watch the number. Six weeks later the number is 94% and has been 94% for a month, while users keep reporting problems the suite never sees.

The suite isn't broken. It's answering a different question than the one you have. "How good is this system in general" is a research question. The engineering question is "did this change break something that used to work", and it needs a different instrument.

Build the set from failures, not from samples

A representative sample is mostly cases the system already handles. It has to be — that's what representative means. So most of your eval budget gets spent re-confirming the easy path, and the score saturates near the top where it can no longer move.

Build the set out of things that have actually gone wrong instead. Every bug report, every escalation, every time someone says "it did something weird" — that becomes a case. The suite grows from real failures, so it stays hard, and every entry has a known cause. When you're starting with no failure history, spend an afternoon trying to break the thing on purpose and use what you find.

Forty cases assembled this way are more useful than four hundred sampled ones. They also run in a couple of minutes, which is what makes people actually run them.

Grade the cheapest way that works

There's a strong pull toward LLM-as-judge for everything. Resist it where a simpler check exists — deterministic graders are faster, free, and don't drift.

Check	Grader	Notes
Output shape / schema	Parser	Never use a judge for this
Contains a required fact	String or regex	Brittle to phrasing; still often correct
Code works	Run the tests	The only grader that matters for code
Right tool, right arguments	Assert on the trace	Catches most agent regressions
Tone, helpfulness, faithfulness	LLM judge	Needs its own validation

That fourth row is the one people skip. For an agent, asserting on the trajectory — which tools were called, in what order, with what arguments — catches regressions that final-output scoring sails past. An agent that reaches the right answer after nine unnecessary calls is a regression even though the output is fine.

If you do use a judge, validate the judge

An LLM judge is a model with its own error rate, and an unvalidated one gives you false confidence at scale. Label fifty cases by hand, run the judge against them, and measure agreement. Below about 80% you're measuring the judge, not the system.

Judges also do better with a rubric than a scale. "Rate helpfulness 1–10" produces 7s. "Does the response answer the question asked — yes or no? Does it include the account ID — yes or no?" produces signal you can act on. Binary questions with explicit criteria, then aggregate.

Handle variance honestly

Run the same case twice at temperature 0 and you can still get two answers. Non-determinism is a property of the stack, not a bug you can configure away, and it means a single run is not a measurement.

Run each case at least three times and report the pass rate, not the pass.
Track flaky cases separately — a case that passes 2-of-3 consistently is telling you something real about reliability.
Don't chase a two-point movement on forty cases. That's noise. Know roughly how big a change has to be before it means anything, and write it down so nobody relitigates it every week.

Load-bearing

The goal is not a high score. It's a suite that goes down when you break something. If your number has never dropped, you have a dashboard, not a test.

Wire it into the work you already do

An eval suite that runs when someone remembers to run it will stop running. Put it in CI on every change to prompts, tools or model version. Keep it under five minutes so nobody routes around it. Fail the build on a regression against the recorded baseline, not against an absolute threshold — the question is always "worse than before", never "below 90%".

Then add the case that caused the last incident. Every time. That single habit is most of the value: the suite becomes a record of everything you've already learned the hard way, and it stops you learning any of it twice.

The version-bump test

The real proving ground is a model upgrade. When a new version lands, you want to know within an hour whether your system is better, worse, or differently-shaped on the cases you care about. Teams with a failure-derived suite answer that the same day. Teams without one ship the upgrade and find out from users a week later.

That's the whole argument for doing this work before you feel like you need it.