Total Cost of Inference
Eliminating the token bill does not eliminate the cost; it relocates it. This is the source's own correction to his own title — delivered, notably, right after his local model works: "there's really no such thing as free. You've got to find the balance between quality and price." Run a good model locally and you buy the hardware. Run it in someone's cloud and you buy a subscription or a VPS. Run a genuinely free hosted model and you pay in rate limits, in not knowing which model answered, or in capability. The bill is conserved; only its denomination changes.
The denominations the source demonstrates, in the order he pays them:
- Hardware. A 9B model fits a laptop. A model worth using may not. "You need more hardware to run larger models" is the constraint that makes capability-per-parameter the interesting axis (Open-Weight Capability Gap).
- Latency. Four minutes for the local 9B model to answer one question about a project. The cloud model answered "way, way quicker."
- Capability. The 9B model produced a joke and a file; it lost track of its own state and needed a context rebuild to call a tool visibly at all.
- Observability. The harness stopped streaming tool calls — "it basically just spins forever until it responds." The operator loses supervision before they lose correctness (Agent Supervision).
- Rate limits. OpenRouter's free tier is metered in requests per day and per minute, and money buys throughput: the source states 50 requests/day at zero balance, 1,000/day once $10 is loaded. For sustained agentic use — where one task is many requests — this binds long before cost does.
- Routing control. A free router that picks the most-available model per request evades rate limits by taking away the ability to know which engine ran (BYO Inference Endpoint).
- The silent bill. A partial endpoint override leaves the harness billing the vendor's paid models for tool calls. The cheapest configuration and the apparently cheapest configuration are not the same thing.
The operational conclusion follows, and it is not the video's title. Having demonstrated free twice, the source recommends paid-but-cheap: a small open model on a metered endpoint, which he prices at 14¢/40¢ per million tokens against Opus 4.6's $5/$25, "for like 50 to 100x cheaper rather than just being completely free — and I think that's still a huge win." Free is the demo; cheap is the deployment. The reason is that paid-but-cheap buys back the two denominations that actually hurt an agent: predictable routing and throughput.
There is a floor beneath which cheapness costs more than it saves. That is the correction this page makes to Model-Tier Routing's "route each sub-role to the cheapest tier that clears the bar" — the bar has to be priced in operator attention (a scarce, non-fungible currency), not just in dollars per token. A tier that saves $0.01 and costs four minutes of unsupervised spinning has not cleared it.
Claims
- Removing the token bill relocates the cost — into hardware, subscription, VPS, latency, capability, observability, rate limits, or routing control — it does not remove it. principle — durable: inference consumes compute, and compute is bought by someone. "Free" always names whose budget line it left, never that it vanished.
- The cheapest tier has a floor: below it, savings are paid for in operator attention, and attention is the scarce resource. principle — durable, and the necessary caveat to cheapest-tier-that-clears-the- bar. A model cheap enough to be worthless makes its supervisor do the work. See Model-Tier Routing.
- Price a free tier in requests, not tokens. best practice — context: hosted free endpoints for agentic (many-request-per-task) workloads, where per-day and per-minute request caps bind before any token budget does. The source states OpenRouter allows 50 free requests/day at zero balance and 1,000/day once $10 is loaded — money buying throughput, not tokens. Does not apply to local serving, where the constraint is hardware.
- Prefer paid-but-cheap over free for sustained agentic use: it buys predictable routing and throughput, which is what agents actually consume. best practice — context: the source's own landing place after demonstrating both free paths; the arithmetic depends on cheap metered models existing at a large multiple below the frontier, which is a property of the current market, not a law.
- Verify cost against the billing log, not the harness banner. best practice — context: any BYO-endpoint setup; the silent-bill failure mode is invisible from inside the harness. See BYO Inference Endpoint.
- Latency and lost observability are costs paid in operator attention, and they do not appear on any invoice. observation — the source pays both on the 9B local model without ever naming them as costs; he names only hardware and subscription. The unpriced denominations are the dangerous ones.
- The source states that Gemma 4 31B Instruct is $0.14 per million input tokens and $0.40 per million output on OpenRouter, against Opus 4.6 at $5 and $25, and that Ollama's cloud tiers are $0/$20/$100 per month, with the paid tiers gating concurrent cloud models and usage. observation — the source's on-screen claims; groundable, not verified here, and prices move.
Related
- Model-Tier Routing — this page supplies the floor that "cheapest tier that clears the bar" needs. The bar is priced in attention, not only dollars.
- BYO Inference Endpoint — the silent bill, and the routing control a free router takes away.
- Open-Weight Capability Gap — capability-per-parameter is the axis on which the hardware bill is set.
- Harness / Model Fit — the capability and observability denominations are contract violations wearing a cost label.
- Agent Supervision — lost tool-call visibility is the cost that lands on the human.
- Execution Commoditization — argues the savings from cheap execution are real but defend no moat; this page argues the savings are also smaller than they look.
- Agentic Simplicity — paying only for capability where it pays off, with the symmetric warning about paying too little.
- Distillate: Ollama + Claude Code = 99% CHEAPER