Your AI Product Needs Evals
TL;DR
Hamel Husain's foundational (March 2024) argument that unsuccessful LLM products share one root cause — no robust evaluation system — and that success, as in software engineering, is gated by iteration speed: evaluate → debug → change, run fast in a loop. Most teams do only the third (prompt/model changes) and never escape the demo. The concrete structure is a cost-ordered Levels of Evaluation: Level 1 cheap code assertions on every change, Level 2 human + calibrated LLM-as-Judge on a cadence, Level 3 A/B tests only when mature. The engine underneath is Error Analysis ("you can never stop looking at data; remove ALL friction") and Synthetic Data Generation (generate inputs, not outputs). The payoff compounds: the same eval infrastructure is your debugging and fine-tuning infrastructure — "superpowers for free." Keep it simple, build problem-specific evals, don't buy generic frameworks.
Concepts introduced
- Levels of Evaluation — new page. The three-tier, cost-ordered eval hierarchy (assertions / human+model / A/B) whose cost gradient sets each level's cadence.
- Synthetic Data Generation — promoted from held. Generate synthetic inputs to bootstrap evals with no traffic; the same generator feeds fine-tuning data.
- Error Analysis — updated: "remove ALL friction from looking at data" and "you can never stop looking at data — no free lunch" added as principles.
- LLM-as-Judge — updated: use the strongest model you can afford + run a mini-eval for the judge; and the imbalanced-class caveat (measure precision/recall, not raw agreement).
- Eval-Driven Development — updated: eval infra is debug + fine-tune infra; build problem-specific evals, not generic frameworks; fine-tune for syntax/style/rules vs RAG for facts.
Held, not dropped (themes this capture touches that don't warrant their own page yet — spin out on demand):
- Trace logging / observability — traces as the prerequisite substrate for Level 2 (LangSmith/LangChain, and the vendors in the footnotes). Currently folded into Error Analysis's "complete trace" definition; promote if a tracing-specific distillate lands.
- Fine-tuning — mechanics of assembling fine-tuning data ("99% of the labor is the data"); noted as a claim on Eval-Driven Development, not yet its own page.
- RAG evaluation — the capture explicitly scopes this out (defers to Jason Liu's "levels of complexity" post); held for a future RAG-eval capture.
- Cost-tiered eval cadence as a budgeting discipline — the L3>L2>L1 cost gradient as a general attention/spend allocation idea; lives inside Levels of Evaluation for now.
Key claims
- Iteration speed is the whole game; the eval→debug→change loop is what separates great from mediocre AI products. (principle — Eval-Driven Development) — most teams do only "change" and stall at a demo.
- Evaluation is the most critical component — spend most of your time making it robust and streamlined. (principle — Eval-Driven Development)
- Order evaluation by cost: assertions every change, human/model eval on cadence, A/B only when mature. (best practice — Levels of Evaluation) — contingent on the L3>L2>L1 cost gradient.
- Level 1 assertions must be fast, cheap, and reusable beyond tests (retries, data cleaning); pass rate is a product decision, not necessarily 100%. (best practice / observation — Levels of Evaluation)
- Remove ALL friction from looking at data; you can never stop looking. (principle — Error Analysis) — the highest-leverage discipline, restated from the source that originated it.
- Generate synthetic inputs, not outputs, across features × scenarios; the same generator curates fine-tuning data. (best practice / observation — Synthetic Data Generation)
- Calibrate the LLM judge against a human; use the strongest model you can afford and run a mini-eval for the judge itself. (principle + best practice — LLM-as-Judge)
- Don't gate on raw agreement when classes are imbalanced — measure precision/recall. (best practice — LLM-as-Judge) — a genuine refinement to the judge concept.
- Eval infrastructure is also your debugging and fine-tuning infrastructure — free. (principle — Eval-Driven Development)
- Build problem-specific evals; don't rely on generic frameworks; keep it simple. (principle — Eval-Driven Development)
- Fine-tuning is for syntax/style/rules; RAG is for context/facts. (best practice — Eval-Driven Development)
Why this builds on (and corroborates) the existing graph
This is the source article the graph was already circling. The prior distillate AI Evals — Self-Study Plan (Hamel & Shreya) cites Hamel's later "Field Guide to Rapidly Improving AI Products"; this is the earlier, originating statement of the same flywheel. So its core — look at your data, evals-first, calibrate the judge, failures become regression tests — corroborates Eval-Driven Development, Error Analysis, and LLM-as-Judge as an independent (indeed originating) witness. Following the constructive-dedup rule, none of those concepts are duplicated; they gain a second, earlier provenance rather than a copy.
Where it builds on rather than repeats — and why the
dominant stance is builds_on:
- New structure. The cost-ordered three levels (assertions / human+model / A/B) is a load-bearing organizing frame the graph lacked → new Levels of Evaluation page.
- A held theme promoted. The self-study distillate explicitly held "synthetic-data bootstrapping." This capture is a second concrete witness (test-case and fine-tuning generation are "almost identical"), so it's promoted to Synthetic Data Generation — held → concept, on demand.
- Sharpened claims. The imbalanced-class caveat on LLM-as-Judge (precision/recall over raw agreement) and the "eval infra = debug + fine-tune infra" principle on Eval-Driven Development are genuine additions, not restatements.
It also independently reinforces the "custom beats generic / don't
buy fancy tools" posture that surfaces in llmops-continual-improvement
(reference, not wikilink — that is engine-design research, not a vault
node).