Synthetic Data Generation
Using an LLM to manufacture the inputs an eval system needs before real production traffic exists — and, later, to manufacture fine-tuning data. The recurring insight is that you generate inputs, not outputs: you prompt a model to write realistic user requests (across the product's features and scenarios), then run them through your system and let your Levels of Evaluation machinery judge the results.
Two uses, nearly the same exercise:
- Test cases. Prompt for N varied instructions that trigger each scenario you want to test (e.g. "write 50 instructions a real-estate agent would give to create contacts…", paired with a lookup instruction so you can assert exactly one result round-trips).
- Fine-tuning data. The same generate-inputs prompt, then filter with your Level 1/2 evals and human review to keep only high-quality traces. Curation and synthesis come nearly for free once an eval system exists.
Don't wait for production data. Make educated guesses about usage, generate synthetically, then let a small pilot of real users refine your generation strategy. Make the cases as challenging as possible while still representative — when the model struggles to pass them, those failure modes become the problems you later solve with fine-tuning.
Claims
- Generate synthetic inputs, not outputs, spanning features × scenarios. best practice — context: bootstrapping evals before you have users, or filling coverage gaps; grounding in real constraints is what keeps the data representative.
- The same generator feeds both test cases and fine-tuning data. observation — producing test inputs and producing fine-tuning inputs are almost the identical exercise, so an eval system doubles as a data-curation engine.
- Make synthetic cases as hard as possible while staying representative. best practice — context: input design for eval/curation; the cases the model fails are the ones worth having, but only if a real user could plausibly issue them.
- Let a small pilot of real users refine the synthetic-generation strategy. best practice — context: transitioning from zero-traffic guesses to production; real usage corrects the distribution your guesses got wrong.
Related
- Levels of Evaluation — synthetic inputs exercise both Level 1 assertions and Level 2 human/model eval.
- Eval-Driven Development — the eval system is also the data-generation and curation engine for fine-tuning.
- Error Analysis — labels from trace review curate which synthetic/real traces are worth keeping.
- Distillate: Your AI Product Needs Evals.