Validation-Gated Update
A proposed change to a durable artifact is accepted only if it strictly improves a score on a held-out set the proposer never saw. Everything else is rejected — and kept, in a rejected-edit buffer, as negative evidence about what not to try again. The gate is what converts an open-loop generator of plausible edits into a process with a floor: the artifact after N updates is, by construction, no worse than the artifact after zero.
The mechanism is old (early stopping, model selection on a validation
split) and its transplant into text is the interesting move. In
SkillOpt, every candidate add/delete/replace edit to a skill document
faces a held-out validation score before it is allowed into
best_skill.md; in SkillOpt-Sleep the same gate stands
between a mined skill candidate and the operator's real skill directory.
Two properties travel with it. First, monotonicity: a
gated self-evolving system can run unattended for many rounds without
regressing, which is precisely the drift that makes unsupervised
improvement loops untrustworthy. Second, held-out-ness is
load-bearing: a gate scored on the same tasks the optimizer saw
measures memorization, not improvement, and a gated loop that overfits
its validation set has quietly deleted its own guarantee. The gate's
strength is exactly the strength of the split.
This is where a live tension sits. Self-Improving System holds — as a principle — that "self-improving" must not mean "fully autonomous," because an unsupervised loop drifts and removes the operator's judgment. A validation gate is a bid to mechanize that judgment: if the score is the operator's proxy, the gate can sign off without them. The bid succeeds only to the extent the score captures what the operator cares about. A gate is a proxy for judgment, not judgment; what it does not measure, it does not protect.
Claims
- A change to a durable artifact should be accepted only on strict improvement against a held-out score, and rejected changes should be retained as negative evidence. best practice — context: an automated proposer generating many candidate edits, where a trustworthy score exists and a held-out split is affordable. It is not available where quality is unmeasurable — the whole construction rests on the score, which is why Eval-Driven Development is a precondition rather than a companion.
- A gated update loop is monotone: it cannot end below where it started. principle — durable, and the reason gating is the load-bearing part of self-evolving-agent designs rather than a detail. The guarantee is what buys the right to run unattended.
- A gate is a proxy for judgment, not a replacement for it — it protects only what its score measures. principle — durable. The corollary is uncomfortable: an autonomous loop with an excellent gate on the wrong metric drifts confidently, which is worse than drifting visibly. This is Self-Improving System's "only trains chest" failure re-derived from the metric side.
- Score the gate on data the proposer never saw; a gate evaluated on the optimizer's own tasks measures memorization. best practice — context: any generate-and-select loop over text artifacts with a small task pool, which is the common case — held-out splits of a few hundred tasks are cheap to overfit across many epochs.
- SkillOpt places the gate in both of its loops: on each
candidate skill edit during training, and between a mined skill
candidate and the live skill directory in the nightly Sleep
cycle. observation —
the repo's
gate.pyandstaging.pysit inskillopt_sleep/, i.e. the gate is a first-class stage, not a check inside the optimizer.
Related
- Text-Space Optimization — the loop this gate governs; without it the optimizer is an unbounded rewriter of a working document.
- Offline Consolidation Cycle — the second place the gate appears: staging mined skills behind a held-out check before they touch the live directory.
- Eval-Driven Development — the precondition. "Don't trust vibes; measure" is what makes a gate possible at all; a gate is that principle made automatic and blocking.
- Pre-Deployment Validation — the sibling gate on premises rather than edits: verify the run can succeed before paying for it; this one verifies a change improved things before keeping it.
- Evidence-Gated Completion — the same accept-only-on-evidence shape applied to an agent's claim that it finished.
- Self-Improving System — the tension: that page holds autonomy must stay human-driven; this one is the argument that a held-out gate can stand in for the human's sign-off.
- Loop Training Mode — the human-supervised way to earn autonomy; a validation gate is the automated way to claim it.
- Agent Supervision — tiered approval routes attention to high-stakes changes; a gate is the degenerate tier where nothing reaches the operator because the score already refused it.
- Skill Pruning — the deletion test ("remove it; did behavior change?") is a validation-gated update with a sample size of one and a human as the score.
- Knowledge-Graph Lint — the sibling gate on structure rather than a score: it accepts a write only if the store's link graph stays connected (no new orphans), where this one accepts an edit only if a held-out score improves.
- Distillate: microsoft/SkillOpt — training skills like weights, without touching weights