Repeated-Sampling Scaling
Trying more times reliably raises the chance a correct answer exists somewhere in the pile; finding it is a separate problem, and it is gated on having a mechanical checker. Where a checker exists, coverage converts straight into results. Where it doesn't, spend past a low ceiling buys answers nobody can identify.
The two halves, from the source's account of a Stanford 2024
repeated-sampling study (identified from a frame in the capture as the
"Large Language Monkeys" work at
scalingintelligence.stanford.edu/pubs/large_language_monkeys/;
all figures below are the source's claims and are check-worthy — see the
distillate):
- Coverage scales, smoothly. A cheap coding model given one attempt per bug fixed 15.9% of a standard benchmark; the same model, same harness, given 250 attempts reached 56% — beating the 43% achieved by the best single attempt from the best model available at the time. The improvement followed a predictable curve across four orders of magnitude of attempts. Pushed to 10,000 attempts, over 95% of runs contained a correct answer somewhere.
- Precision doesn't, without a verifier. Where something mechanical could grade each attempt — a test suite, a proof checker, unit tests — "coverage turned straight into results." Where there was none and the model had to pick the best answer out of its own pile, majority voting and reward models all stalled around 100 attempts. The source calls this "the half of the study nobody quotes," and it is the half that matters: the right answer is in there and nobody can tell which one it is. Every dollar past that line buys attempts that are generated and never found.
The distinction this page exists to protect: attempts are not agents. Agentic Simplicity and Bounded Fan-Out hold — correctly — that more agents is not monotonically better. Repeated sampling says more attempts very nearly is, inside a verifier. These are different axes and conflating them inverts both lessons. Reading "coverage scales log-linearly" as a licence to fan out a swarm is exactly the error Bounded Fan-Out warns against; reading "more agents ≠ better" as "more attempts ≠ better" throws away the one dial that does scale. What repeated sampling actually licenses is retries against a check, which is a property of the loop, not of the roster.
The corollary the source draws, and the reason this sits at the centre of Agent-Shape Triage: checkability is not hygiene, it is the switch that decides whether spend converts at all. It is why "is checking much cheaper than producing?" earns a place among only four questions.
Claims
- Coverage and precision are separate problems: raising the chance a correct answer exists in the pile does not raise your ability to pick it out. principle — durable, and the page's spine. The two are conflated by default because with a verifier they move together; the whole insight is visible only where the verifier is absent.
- Extra attempts only convert into results where something mechanical can grade each one; without a checker, spend stops buying outcomes and starts buying unfindable answers. principle — durable: this is a statement about the structure of search, not about a model generation. Generate-and-select needs a selector; where selection is as hard as generation, the extra generations are waste. This is the economic argument for Eval-Driven Development — the first in the vault that isn't "don't trust vibes."
- Attempts scale where agents don't — they are different dials, and the sampling law is not a licence to fan out. principle — durable, and the boundary that keeps this page from contradicting Bounded Fan-Out. Repeated sampling licenses retries against a check; it says nothing about roster size, and a swarm without a verifier is the worst of both (more spend, no selector).
- Where checking is expensive, expect multi-agent value to top out quickly — the source reads the study as ~100 attempts. best practice — context: designing a retry/fan-out budget for a task whose output can't be graded automatically. The specific number is study- and task-bound (it comes from 2024 bug-fixing benchmarks) and should be treated as an order-of-magnitude signal, not a constant; what generalises is the shape — a ceiling exists and it's low.
- Cheap model + many attempts can beat expensive model + one attempt. observation — the source's headline (56% vs 43%), and a striking one because it prices capability against persistence. It is the empirical basis for Model-Tier Routing's "cheapest tier that clears the bar," with a new wrinkle: the bar can sometimes be cleared by retrying rather than by upgrading. Check-worthy.
- The video states the improvement curve held across four orders of magnitude of attempts, and that at 10,000 attempts over 95% of runs contained a correct answer. observation — the source's claims; check-worthy. Partially corroborated within the capture by a frame at t=06:04 showing the arXiv paper with the 56%/43% figures marked on a SWE-bench Lite chart.
- The video states Anthropic found token spend — not prompt wording — explained ~80% of the difference between a good and a bad multi-agent research run, and that the agent team beat the then-frontier model alone by 90.2%. observation — the source's claim; check-worthy. Offered as the production-side echo of the academic result: a team of agents is how you spend more tokens than one agent can usefully hold. See Team-Forming Constraints.
Related
- Agent-Shape Triage — the decision procedure this law sits under: checkability is one of its four questions, and this page is why that question is load-bearing rather than hygienic.
- Eval-Driven Development — the same requirement reached from the other direction. That page argues for evals from non-determinism ("don't trust vibes; measure"); this one argues from economics (without a checker, marginal spend buys nothing). Two independent routes to "you need evals."
- Bounded Fan-Out — the page this one must not be read as contradicting. Agent count has costs and is not monotonically better; attempts against a verifier are the dial that does scale. Different axes.
- Team-Forming Constraints — the Anthropic half of the evidence: a team is how you spend more tokens than one context window can hold, which is exactly the capacity limit that page names.
- Evidence-Gated Completion — the per-task instance of "the checker decides, not the worker." This page says why a real checker is the precondition for scaling attempts at all; that page says what the check must produce.
- Levels of Evaluation — the menu the "mechanical checker" is drawn from; the study's cases (unit tests, proof checkers, exit codes) are the cheap deterministic end, which is precisely why they worked.
- LLM-as-Judge — the tempting substitute when no mechanical checker exists, and the source's warning aimed squarely at it: model-picks-best-from-pile is what stalled at ~100 attempts.
- Model-Tier Routing — "cheap model, many attempts, beats expensive model, one attempt" gives the cheapest-adequate-tier discipline a second lever: retry instead of upgrade, where a checker exists.
- Agentic Workflow Patterns — the evaluator-optimizer loop is repeated sampling with the verifier wired in; this page explains why that pattern works and its unverifiable cousin doesn't.
- Check Gaming — the failure mode that lurks under "just add a checker": a weak check is worse than none, because coverage converts into confidently-wrong results.
- Total Cost of Inference — the shaded area under the coverage/precision gap is money spent and not recovered; the source's "that's money you're not getting back if you don't design your systems correctly."
- Distillate: 1.6M agents registered for OpenClaw and did NOTHING.