Harness / Model Fit
A harness is separable from its model (Harness / Model Separation), but it is not indifferent to it. Every harness encodes an implicit contract the model must satisfy, and the contract is almost never written down. The source names three clauses of Claude Code's: the model must have been trained on the harness's tools; its context window must be large enough to hold the harness's system prompt (and leave room to work); and it must emit the exact tool-call protocol the harness parses. A model that misses any clause does not fail cleanly — it "seems to misbehave." His analogy: a motorcycle engine in a truck, or the wrong grade of fuel.
The failure signature is what makes this worth its own page. In the
source's runs, a violated contract presented as: state
loss the model never reported (context overflow, mistaken for
forgetfulness); loss of tool-call visibility, so the
harness stopped streaming steps and "just spins forever until it
responds"; and — the cleanest instance — a harness tool firing into a
void. Asked to research a model, the substituted engine invoked
Web Search, received "Did 0 searches in 42ms"
twice, narrated its own confusion, fell back to raw fetches, and finally
answered from training data. The tool survived the swap; the
capability behind the tool did not. This is the sharp edge of
Agent-Computer Interface
(ACI): tool definitions are written for a model that was trained to
use them, and a tool the substituted model cannot actually execute still
appears in its toolbox.
The practical consequence is that a substituted model must be evaluated against the harness, not against a benchmark. A model's SWE-bench score says nothing about whether it emits the JSON the harness parses.
Claims
- A harness encodes an implicit contract on its model — trained on its tools, a context window that fits its system prompt, conformance to its tool-call protocol — and a model that misses it will misbehave rather than fail. principle — durable: the contract exists in any harness, and silent degradation is the default because harnesses parse for success and have no channel for "the engine doesn't fit."
- Tools in the harness are not capabilities in the model: a
tool definition can survive a model swap while the capability behind it
does not. principle —
durable and load-bearing. The source's
Web Search→ "Did 0 searches in 42ms" is the demonstration; the model kept trying because nothing told it the tool was inert. Sharpens Agent-Computer Interface (ACI), which assumes the model can execute what it is offered. - Raise a local model's context window before judging it — an undersized window presents as state loss and lost tool-call visibility, not as an error. best practice — context: local serving stacks whose defaults advertise a larger window than they serve (Ollama displaying 200k while serving far less). The same 9B model went from a four-minute failure to a 2m 5s success after a rebuild at 64k. The practice is "best" only where you control the serving config; behind a hosted endpoint you cannot apply it.
- Evaluate a substituted model against the harness, not against a benchmark. best practice — context: any BYO-model swap (BYO Inference Endpoint). Benchmark rank measures capability; the contract measures conformance, and they are close to uncorrelated. Cheap to check: give it one file-write task and watch whether the tool call renders. Consistent with Error Analysis — the answer comes from watching real usage, not from the model card.
- Tool-call visibility in the harness's transcript degrades with model size, and the operator loses supervision before they lose correctness. observation — the source's experience across three engines (9B local: no visible calls; 9B at 64k: calls visible; a large cloud model: full visibility, four subagents). Mechanism unexplained; held as an observation pending a second source. See Agent Supervision.
Related
- Harness / Model Separation — the enabling fact this page constrains. Separable, yes; interchangeable, no.
- BYO Inference Endpoint — where the swap is performed, and therefore where this contract gets tested.
- Agent-Computer Interface (ACI) — ACI assumes a model trained on the tools; this page is what happens when that assumption is dropped. A tool the model cannot execute is worse than no tool: it invites the attempt.
- Agent Supervision — degraded tool-call visibility is a supervision failure, not just a cosmetic one.
- Error Analysis — the discipline for finding contract violations: watch what the model actually does with the tools.
- Context Compression — an undersized context window is the contract clause most often violated, and compression is the harness-side mitigation.
- Open-Weight Capability Gap — benchmark rank is the axis this page argues is the wrong one for predicting fit.
- Distillate: Ollama + Claude Code = 99% CHEAPER