repo github.com View on GitHub ↗

microsoft/SkillOpt — training skills like weights, without touching weights

TL;DR

SkillOpt is Microsoft Research's argument, shipped as code, that a skill document is the trainable state of a frozen agent — and that you can optimize it with the full apparatus of weight-space training (epochs, mini-batches, a "textual learning rate," a rejected-edit buffer, held-out validation) to produce a compact best_skill.md that adds zero inference-time model calls. The load-bearing part is not the optimizer; it is the gate: a candidate edit is accepted only if it strictly improves a held-out score, which is what lets the loop run unattended without regressing. That is precisely where it collides with this vault. Self-Improving System holds as a principle that "self-improving" must not mean "fully autonomous," because an unsupervised loop drifts and quietly removes the operator's judgment. SkillOpt's entire design is the counter-bid: a held-out validation gate is a mechanized substitute for that judgment. You should care because the tension is real, unresolved, and directly load-bearing for firehose's own compounding design — and because if the transfer claim holds (artifacts move across model scales and between Codex and Claude Code harnesses without re-optimization), then the optimized skill is not a fitted parameter but a portable, distributable asset, and the offline cost amortizes forever.

What it is and how it is organized

A Python package + paper artifact (MIT, ~11.7k★, arXiv 2605.23904, pip install skillopt), organized into two loops that share one gate:

The training loop (skillopt/) — envs/ (six benchmarks: alfworld, docvqa, livemath, officeqa, searchqa, spreadsheetbench, each a dataloader.py + rollout.py + initial.md seed skill), model/ (a backend router: OpenAI / Azure / Claude / Qwen / MiniMax chat, plus codex_exec and claude_code_exec harnesses), gradient/ and optimizer/ (the edit proposer), scheduler/, evaluation/. The loop is rollout → reflect → aggregate → select → update → evaluate. ckpt/ holds trained skills per benchmark; configs/ is an mmcv-style _base_ config tree.
SkillOpt-Sleep (skillopt_sleep/, the v0.2.0 headline) — the same idea pointed at your own agent's history instead of a benchmark: harvest.py (mine Claude Code / Codex / Copilot session transcripts) → mine.py / llm_miner.py → replay.py + dream.py (rerun recurring tasks, and perturbed variants) → judges.py → gate.py → staging.py → consolidate.py → memory.py. It is budgeted (budget.py) and scheduled (scheduler.py, plugins/run-sleep.sh). plugins/ ships shells for claude-code, codex, copilot, devin, openclaw.

The organizing tell is that gate.py and staging.py are top-level modules in the sleep engine, not helpers inside the optimizer. Promotion is a first-class stage, separate from generation. Read the two initial.md/best_skill.md pairs under ckpt/ and skillopt_sleep/gate.py and you have the thesis; the rest is benchmark plumbing and backend adapters. Everything else here — skillopt_webui/ (Gradio dashboard), the mkdocs site, data/*_split/ — is scaffolding.

Concepts introduced

Text-Space Optimization — the reframe: markdown is the trainable state; a separate optimizer model emits bounded add/delete/replace edits under a textual learning rate. Cost moves offline; the deployed artifact costs nothing extra at inference, unlike reflection or judge loops that pay per request forever.
Validation-Gated Update — accept an edit only on strict improvement against a held-out score; keep rejects as negative evidence. This is what makes the loop monotone, and it is the mechanism standing where the human's sign-off stands in Self-Improving System.
Offline Consolidation Cycle — SkillOpt-Sleep: nightly harvest → mine → replay → consolidate on the agent's own session history, staged behind the gate. Moves improvement work off both the request path and the operator's attention.
Skill Artifact Transfer — the artifacts reportedly move across model scales, between Codex and Claude Code, and to nearby benchmarks with no re-optimization. What transfers is a statement about the task, not a fit to the model — which is what makes the offline cost economic, and which quietly suggests the learned content is generic rather than model-specific.

Held, not dropped (present in the repo, no concept page yet — spin out on demand):

Backend/benchmark extension contracts — "a backend is a module + two registrations"; "a benchmark is a package with dataloader.py, rollout.py, initial.md." A clean plugin-seam design worth a page if the graph ever grows an extensibility contract concept.
Cross-tool plugin shells (claude-code / codex / copilot / devin / openclaw) — one engine, five agent harnesses; adjacent to Agentic Distribution but about engine portability, not skill portability.
Multi-objective reward + experience replay + "dream" rollouts — RL machinery imported wholesale into text-space; held pending evidence that dreamed variants beat plain replay.
The WebUI/Gradio observability dashboard — real, and uninteresting at triage altitude.
arXiv-paper-as-repo as a genre — trendshift badges, ckpt/, per-benchmark configs, a citation block. The shape tells you what the code is optimized for (reproducing a table), which is not the same as being optimized for your use.

Key claims

The skill document is the trainable state of a frozen agent; optimizing it is an optimization problem, not prompt-fiddling. principle → Text-Space Optimization
A gated update loop is monotone — it cannot end below where it started. principle → Validation-Gated Update
A gate is a proxy for judgment, not judgment; it protects only what its score measures. principle → Validation-Gated Update — the vault's reading of the source's mechanism, and the crux of the tension below.
Optimizing offline into a static artifact keeps inference-time cost at zero, unlike reflection or judge loops that pay per request. principle → Text-Space Optimization
What transfers across models was never fitted to a model — so a portable skill is a statement about the task. principle → Skill Artifact Transfer
Bound each edit's size ("textual learning rate") and retain rejected edits as negative memory. (best practice) — context: an automated optimizer editing a working document across many epochs, where one unbounded rewrite can destroy it. → Text-Space Optimization
Score the gate on data the proposer never saw. (best practice) — context: any generate-and- select loop over a small task pool, which overfits across epochs. → Validation-Gated Update
Stage promotion separately from generation — a self-evolving nightly loop otherwise means unreviewed writes to production skills. (best practice) — context: unattended cycles with write access to tomorrow's artifacts. → Offline Consolidation Cycle
Prefer optimizing the artifact over tuning the model when the model is frozen, changes often, or is not yours. (best practice) — context: hosted frontier models with short deprecation cycles; inverts if you own the weights and the task distribution is stable. → Skill Artifact Transfer
Ad-hoc skill evolution — hand-crafted, one-shot-generated, or loosely self-revised — does not reliably improve over its starting point under feedback. observation → the repo's stated motivation; the complaint is variance, not incapacity.
A separate optimizer model, not the target agent, proposes the edits. observation → Text-Space Optimization

Check-worthy source claims (the repo's assertions, tagged for a later grounding pass — not verified here; this call has no external access):

Best or tied-best on all 52 evaluated (model, benchmark, harness) cells, across six benchmarks, seven target models, and three harnesses (direct chat, Codex CLI, Claude Code CLI). observation
On GPT-5.5: +23.5 points average accuracy over no-skill in direct chat, +24.8 in the Codex loop, +19.1 inside Claude Code. observation — named model, named numbers.
The deployed best_skill.md is typically 300–2,000 tokens. observation
v0.2.0 shipped 2026-07-02 with the skillopt-sleep CLI; ~11,740★; arXiv 2605.23904; gbrain, gbrain-evals, and darwin-skill have integrated SkillOpt. observation

Why this contradicts Self-Improving System

This is the highest-value thing in the capture, and it is a real disagreement, not a difference of emphasis.

Self-Improving System holds, as its central principle: "self-improving" does not mean "fully autonomous" — full autonomy removes your judgment and drifts. Its illustration is the workout system that only ever trains chest: it believed it was improving you, and it was breaking you. The prescribed remedy is that the human stays the driver and signs off on direction.

SkillOpt asserts the opposite is achievable, and names the missing piece. Its diagnosis of "loosely controlled self-revision" is the same diagnosis — such loops don't reliably improve over their starting point. Its cure is not a human; it is a held-out validation gate. A candidate edit that does not strictly improve the score is rejected. Under that construction the loop is monotone by design, and SkillOpt-Sleep runs it nightly, unattended, over the operator's own sessions.

Stated plainly, the tension to resolve: Self-Improving System holds that judgment cannot be removed from a compounding loop without drift; SkillOpt asserts that a strict held-out gate is a sufficient substitute for that judgment. Neither side is adjudicated here. What can be said is where the disagreement actually lives — not in whether gates work, but in what the score measures. The chest-only workout system had a gate; it improved its metric every week. A gate protects only what it measures, and a benchmark suite is a narrower proxy for "did this make the agent better at my work" than an operator's judgment is. SkillOpt's own strongest evidence cuts both ways here: its gate is scored against six academic benchmarks, and its answer to "does that generalize" is the transfer result (Skill Artifact Transfer) — which is an empirical answer to a question Self-Improving System poses as structural. That is the shape of the resolution, if there is one: the gate substitutes for judgment exactly to the extent its score is the operator's proxy, and SkillOpt-Sleep's harvest-from-your-own-sessions design is a bid to make it one.

Secondarily, this builds on Minimal Skill Surface and Skill Pruning from an unexpected direction. Both hold that the deployed skill surface should be small, and both arrive there by authorial taste — the deletion test, branch analysis, hunting no-ops. SkillOpt arrives at a 300–2,000-token artifact by optimization: a delete edit that improves held-out score is a deletion test executed mechanically, with a benchmark rather than a human as the falsifier. Two independent routes converging on "small" is corroboration for the smallness thesis, and it sharpens Skill Pruning's deletion test by showing it is a special case of a gated update with n=1.