Error Analysis

The highest-ROI activity in improving an LLM app: read real production traces by hand, write open-ended notes on failures, cluster them bottom-up into a taxonomy, count frequencies. A small number of failure modes accounts for the majority of failures — that distribution is the improvement roadmap. It cannot be learned by reading; it must be done on a real trace corpus (a complete trace = the initial query plus every intermediate step, tool call, retrieval, and the final response).

Method (from the AI-evals plan, Phase 2):

Pull 20–50 real traces.
One row per trace; write open-ended notes on anything undesired — open coding. Bottom-up, not top-down: let failure modes emerge, don't start from generic "hallucination/toxicity" buckets.
Use an LLM to cluster the notes into a taxonomy.
Label each row; pivot-table the counts → prioritize by frequency.

Build a simple custom data viewer to make this fast: all context in one place, one-click correct/incorrect, an open-ended note field, filter/sort, keyboard nav. Custom beats generic and can be built in hours with AI-assisted coding.

Claims

Look at your data — human error analysis is the highest-leverage activity, above any tool. principle — durable: attention on real failures, not tooling, is the bottleneck. Corroborates P2 in llmops-continual-improvement.
Open-code bottom-up; let the taxonomy emerge — don't start from generic failure buckets. best practice — context: early failure discovery for a specific product; top-down taxonomies import biases and miss the failure modes that actually dominate.
Build a simple custom data viewer for traces. best practice — context: when a team is doing repeated trace review at any volume; a spreadsheet is a fine start, custom tooling pays off once review is habitual.
Remove ALL friction from looking at data; render traces in domain-specific ways. principle — durable: whatever raises the cost of looking at a trace (scattered context, generic UI) is the thing that stops you looking, so collapse it — one screen, one-click labels, links to every system involved.
You can never stop looking at data — there is no free lunch. principle — durable: read traces from all test cases and real users at minimum; you may sample more over time to lessen the burden, but never stop entirely.

Eval-Driven Development — error analysis is what tells you which evals to write.
LLM-as-Judge — the taxonomy's subjective failure modes become judge criteria.
Levels of Evaluation — hand trace review is the core of Level 2.
Synthetic Data Generation — labels from trace review curate which traces are kept.
Distillate: AI Evals — Self-Study Plan (Hamel & Shreya).
Distillate: Your AI Product Needs Evals.
Distillate: How Claude Is Creating a New Generation of Millionaires — failure-as-signal at the single-task altitude: a first version "60–70% right" is normal, "treat the misses like golden data, not failure," and let the model fix its own work.

Error Analysis

Claims

Related

Linked from