article inbox/ai-evals-study-plan.md Read the original ↗

AI Evals — Self-Study Plan (Hamel & Shreya)

First vault distillate — proves the article → distillate → concept loop end-to-end. Distilled from a self-study plan for Hamel Husain & Shreya Shankar's AI Evals for Engineers & PMs. Stance toward the graph: corroborates llmops-continual-improvement — the same flywheel, arrived at independently — while building on it with concrete, hands-on operational detail (the data viewer, the synthetic-data grid, IR retrieval eval).

TL;DR

One loop, run forever: build an instrumented agent → error-analyze real traces by hand → write evals only where the analysis justifies them → improve, gate in CI, monitor drift. The single highest-ROI activity is error analysis (bottom-up open coding of real traces into a failure taxonomy), not tooling. Judge subjective failures with a calibrated LLM-as-judge (binary pass/fail + written critique, validated against human labels). Bootstrap with synthetic user inputs when you have no users. Count experiments, not features.

Concepts introduced

Error Analysis — the highest-ROI activity; the heart of the method.
LLM-as-Judge — the calibrated evaluator for subjective failures.
Eval-Driven Development — the reactive, error-analysis-first eval flywheel.

Further themes this capture touches but that don't yet have concept pages (held, not dropped — spin out on demand): synthetic-data bootstrapping (generate inputs not outputs, across features × scenarios × personas), retrieval eval (evaluate retrieval and generation separately; IR metrics — Recall@k, MRR), and cost/accuracy optimization (match model size to task, cascades, caching).

Key claims (see the concept pages for provenance + kind)

Look at your data. Read real traces by hand; error analysis is the highest-ROI activity and cannot be learned by reading. (principle — see Error Analysis)
Don't trust vibes; measure. Write evals so a "better" change can't silently regress. (principle — see Eval-Driven Development)
Calibrate the judge before you trust it. An LLM judge is itself a model; validate it against human labels and iterate to high agreement. (principle — see LLM-as-Judge)
Write evals reactively, only where error analysis reveals a real failure — not proactively for imagined ones. (best practice — see Eval-Driven Development)
Binary pass/fail + written critique beats 1–5 scales for LLM-judge evals. (best practice — see LLM-as-Judge)
Generate synthetic inputs, not outputs, grounded in real constraints, across a features × scenarios × personas grid. (best practice)
Count experiments run, not features shipped. (best practice — see Eval-Driven Development)

Why this corroborates the continual-improvement distillate

Every load-bearing point here re-appears, independently sourced, in llmops-continual-improvement: error analysis is highest-leverage (its P2 / this plan's Phase 2); evals-driven but error-analysis-first; calibrate the LLM judge (its P4); failures → permanent regression tests; crawl-walk-run. Two independent sources converging raises confidence — the corroborates stance in action (constructive dedup: not a near-duplicate to suppress but a second witness that bumps corroboration). Where it builds on rather than repeats: the hands-on mechanics — the custom trace data viewer, the synthetic-data grid, and separating retrieval from generation with IR metrics.

Its Phase 6 ("retrieve, don't RAG"; "you don't need a graph DB") independently reinforces firehose's own task-conditional graph-nav hedge (DECISIONS 2026-06-30) — reference, not wikilink: that survey is engine-design research, not domain knowledge.

The loop (the capture's spine)

Build an instrumented agent — every run leaves an inspectable trace.
Analyze — Error Analysis: 20–50 real traces, open-ended notes, cluster into a failure taxonomy, count frequencies. A small number of modes dominate → the roadmap.
Measure — write evals only where analysis justifies: code-based for deterministic failures, LLM-as-Judge for subjective ones (validated first).
Improve — prove which change moved the metric, wire evals into CI/CD, monitor drift.

Source

inbox/ai-evals-study-plan.md (parked capture; Hamel & Shreya self-study plan). Anchors: Field Guide to Rapidly Improving AI Products (hamel.dev), LLM-as-a-Judge guide, Who Validates the Validators? (arXiv 2404.12272), the O'Reilly Evals for AI Engineers book.