Agent Supervision

As coding agents become autonomous (Workflows vs Agents), the human bottleneck shifts from authoring the work to reviewing and steering it. Agent supervision is the discipline — and the tooling layer — that keeps a human in control of a fleet of semi-autonomous agents when human review bandwidth, not model throughput, is the scarce resource. It is the operator's workbench: the place where agent output is made legible, approved or rejected, and where attention is routed to the agents that actually need it.

Three levers make supervision tractable:

Legible output before approval. Render agent changes as reviewable diffs (red/green WYSIWYG across files, mockups, diagrams, data models, code) the human can accept, reject, edit, or annotate — rather than a wall of raw patch text. Review speed is gated on how cheaply a human can see what changed.
Fleet management. Run many sessions in parallel, track them on a kanban, search and resume, and link sessions↔︎files so the operator can hold a fleet in their head without losing the thread of any one task.
Attention routing. Surface which agents need you now versus which are still working or idle — via dashboards and push notifications, including async/mobile review. This is the firehose thesis applied to agent ops: the constraint is human attention, so the tool's job is pacing and triage, not maximizing agent count.

A useful framing: agent supervision is HCI for agents — the mirror image of the Agent-Computer Interface (ACI). ACI designs the agent's interface to its tools (agent → world); supervision designs the human's interface to the agents (human → fleet).

Claims

As agents gain autonomy, the human bottleneck moves from authoring to reviewing and steering — supervision bandwidth becomes the scarce resource. principle — durable: once the model can produce the work, the binding constraint is the human's capacity to review and redirect it, not the model's capacity to generate.
Make agent output legible before asking for approval — render changes as accept/reject/ annotate diffs, not raw patches. best practice — context: human-in-the-loop over file-mutating agents; legibility is what makes fast review possible, and the "best" diff representation depends on the artifact type (code vs. mockup vs. data model).
Route the operator's attention: make visible which agents need input vs. which are still working, so blocked and idle agents surface without polling. best practice — context: supervising many parallel sessions; attention routing is the mechanism that keeps fleet supervision from degrading into constant manual checking.
Store agent state and status in open, plain formats (markdown, slash commands, files on disk / in git) rather than an opaque application database. best practice — context: agent workbenches where portability, git-diffability, and transparency matter; open storage trades away the conveniences of a managed store for legibility and no lock-in.
Supervising agents is human-computer interaction for agents — the inverse of the ACI. observation — the two interface-design problems (human→agent, agent→tools) are distinct and both deserve deliberate attention.
Some review can be internalized: pair each builder agent with a validator agent (or a Stop-hook format check) that gates the work on an executable check before it reaches the human. best practice — context: multi-agent pipelines where a human can't review every sub-agent's output; automated agent-gates-agent validation reduces the review load but does not remove the human supervising the orchestrator, and it only catches what the check encodes. See Agent Task Graph.
Grant autonomy gradually — run a new autonomous loop under pause-for-approval at first, relax once its behaviour is verified, and keep human checkpoints at the few pivotal decisions where a wrong turn ruins the whole run. best practice — context: promoting a loop from supervised to unattended; the "AI is like an intern" framing — you gate the choices that shape everything downstream, not every step forever, and the more pivotal-but-unverified checkpoints you skip the further the agent drifts. See Loop Training Mode.
Make the review diff bidirectional — let the human annotate lines with instructions the agent reads back, not just accept/reject the agent's changes. best practice — context: a review tool where the agent has read/write access to the annotation store (via a loaded skill); the diff becomes a two-way channel where a note pinned to a line ("add a reset button") is itself a task, so the human never leaves the review view to re-describe the change. This extends legible-diff review from one-way (approve the output) to a closed loop. See Shared Review Surface.
Triage proposed changes into risk buckets so you review only the high-stakes ones — auto-apply low-risk changes to a change log, route high-stakes changes to a checkbox sign-off file, and hold "needs more context" items for you. best practice — context: a self-improving system where reviewing every change is abandoned in practice and auto-approving everything drifts; bucketing is the middle of that spectrum. A second, independent source (Austin Marchese) implements this as a /improve-system skill writing a dated review-<date>.md with approve / reject / approve-and-don't-ask-again checkboxes, plus a change-log.md for the auto-applied bucket. The system "learns what is high-stakes over time," so the buckets are not static. See Self-Improving System, Decision Log.

Workflows vs Agents — supervision is what you wrap around agents (autonomous, dynamically directing their own process); the more autonomy, the more supervision matters.
Agent-Computer Interface (ACI) — the inverse interface: ACI is agent→tools, supervision is human→agents (HCI for agents).
Agentic Simplicity — open/plain-file storage and "show the work" are the transparency principle applied to the operator's workbench.
Decision Log — presenting deduped agent decisions for approve/reject is supervision applied to intent: attention-routed, legible review of why, not just what, changed.
Spec-Driven Development — keeping the spec in sync is a human-approval loop over agent-proposed decisions; review bandwidth, not generation, is the bottleneck.
Agent Task Graph — the builder/validator pair is supervision internalized into the task graph: an agent gates another agent's work automatically.
Loop Training Mode — supervision applied to loops: graduated per-loop autonomy plus checkpoints at the high-leverage decisions.
Shared Review Surface — the legible-diff lever taken bidirectional: the human annotates the same surface the agent does, so review notes flow both ways.
Distillate: Nimbalyst — Visual Workbench for Supervising Coding Agents.
Distillate: Learnings from a No-Code Library: Keeping the Spec-Driven Development Triangle in Sync — a second source for the supervision thesis: Plumb routes decisions to a human, and "AI coding overwhelms human review" is the attention-bottleneck framing.
Distillate: Claude Code Task System: Orchestrating a Team of Agents Through a Task Graph.
Distillate: Loop Engineering, Illustrated: Triggers, Skills, Verification, Memory — loop "training mode" and checkpoint-the-pivotal-decisions as the loop-specific form of graduated oversight.
Distillate: The Library Meta-Skill: How I Distribute Private Skills, Agents and Prompts — "increase trust by knowing exactly what your agents run, down to the lines in your skills/prompts/agents"; references-not-copies over open plain files keeps the source legible.
Distillate: How to Build a Self-Improving System with Claude Code — the concrete three-bucket tiered-approval mechanism (auto-approve / needs-sign-off / needs-context) as the middle of the full-automation↔︎review-everything spectrum.
Self-Improving System — the personal-system frame where bucketed approval keeps a self-improving loop augmented rather than autonomous.
Distillate: Hunk: the review diff as a two-way human↔︎agent annotation channel — Hunk as a bidirectional review workbench: the human's line notes become the agent's tasks.
Distillate: Claude Fable 5 Bossed 20 Cheap AI Agents. The Whole Site Cost $8. — the builder/validator pair taken to its limit: "every task ships with a checking agent" that re-executes the work, so the human supervises the orchestrator rather than reading any individual sub-agent's output. See Authority-Independent Verification for the version where even the orchestrator's output is gated.

Agent Supervision

Claims

Related

Linked from