Levels of Evaluation

A cost-ordered hierarchy of LLM evaluation, from cheap-and-frequent to expensive-and-rare. The ordering is the point: cost L3 > L2 > L1, and that dictates cadence.

Level 1 — unit tests / assertions. Code-based assertions (like pytest): fast, cheap, deterministic. Run on every code change. Organize them for reuse beyond tests — the same assertion error can drive automatic retries or data cleaning during inference. A product can have hundreds of these, continuously updated as new failures surface. Pass rate need not be 100% — it's a product decision about tolerable failures.
Level 2 — human & model eval. For failures assertions can't catch (subjective quality). Prerequisite: logging traces. Involves Error Analysis by hand and, once calibrated, an LLM-as-Judge. Run on a set cadence, not every change.
Level 3 — A/B testing. Confirms the product actually drives desired user outcomes. Not very different from A/B testing any product. Reserved for mature products; okay to defer until you're convinced the product is ready for real users.

There's no strict formula for when to introduce each level — balance fast user feedback, user perception, and product goals, as with any product. But conquer a good portion of Level 1 before leaning on Level 2, since model-based tests cost more to build and run.

Claims

Order evaluation by cost — cheap assertions every change, human/model eval on cadence, A/B tests only when mature. best practice — context: allocating a finite eval budget across a product's lifecycle; the cost gradient (L3>L2>L1) is what sets the cadence, so this is contingent on those relative costs holding.
Level 1 assertions should be fast, cheap, and reusable beyond tests (retries, data cleaning). best practice — context: LLM apps with code-checkable failure modes (format, schema, forbidden content); reuse is what makes the up-front cost pay off.
You don't need a 100% pass rate — pass rate is a product decision. observation — unlike deterministic software tests, tolerable failure is set by product tolerance, not by correctness.
Conquer Level 1 before investing in Level 2. best practice — context: early-stage eval build-out; model-based tests are more work, so front-load the cheap coverage.

Eval-Driven Development — the flywheel these levels plug into; which evals to write is decided by error analysis, not by filling all three levels top-down.
Error Analysis — the core of Level 2; assertions (Level 1) are often written from the failure modes it surfaces.
LLM-as-Judge — the automated half of Level 2.
Synthetic Data Generation — how you get inputs to exercise Level 1 and Level 2 before you have production traffic.
Distillate: Your AI Product Needs Evals.

Levels of Evaluation

Claims

Related

Linked from