Context Compression
The mechanism that keeps a long-running conversation inside a finite context window: when the accumulated message history crosses a threshold of the window, the agent summarizes the previous messages, replaces them with the summary, and appends it to the context — so the conversation continues with its load-bearing state intact but its token footprint collapsed. Three design decisions define an implementation: when to trigger (the threshold, and at which moments it is checked), how usage is measured (estimation vs provider-reported token counts), and what the summary preserves (the structure of the compression prompt — the quality lever that decides whether the agent "forgets" mid-task). The Hermes implementation is the worked example: default trigger at 50% of the window, checks before every turn plus on context-window errors, chars÷4 estimation until real usage numbers exist, and a rich structured summary prompt.
Claims
- Past a threshold, summarize previous messages and substitute the summary for them in context. principle — durable: any bounded window with an unbounded conversation needs a compression step; the alternative is a hard failure at the window edge.
- Default the trigger to ~50% of the context window; raise it (70–80%) for models with smaller windows. best practice — context: Hermes' setup default; the threshold trades how often you pay the summarization cost against how much headroom the model keeps for the current task.
- Check compression at two moments: before each turn, and on a context-window error from the provider. best practice — context: the pre-turn check is the planned path; the on-error check is the safety net for when estimation undershoots the real count.
- Estimate context as characters÷4 before the first response;
after that, accumulate the provider's returned
usagefields. best practice — context: before anything is sent there is no ground-truth token count and a local tokenizer pass is "too expensive" for a good-enough check; the provider's post-response numbers come from the model's own tokenizer for free. (Attributed to the Hermes walkthrough; the exact usage fields vary by provider.) - The compression prompt's structure is the quality lever — a rich template preserves goal, constraints, completed actions, active state, blockers, key decisions, resolved questions, relevant files, and previous summaries. best practice — context: agents that must resume mid-task after compaction; the source contrasts Hermes' rich template with Pi's much more minimalist one — richer summaries carry more working state at more summary cost.
- The source singles out one field as "the single most
important": the user's most recent unfulfilled request, captured
verbatim — a just-asked question is an active task, and a reverse signal
(stop/undo/never mind) must be recorded verbatim so the cancelled task
is not carried forward.
observation — from Hermes'
ContextCompressorcode on screen; the detail that keeps a compacted agent from dropping or resurrecting the user's actual ask.
Related
- Agent Loop — compression runs inside the loop's build-context step; it is what lets the loop run indefinitely.
- Agent Gateway — long-lived multi-channel sessions are where unbounded history meets the finite window; the gateway's reconstructed context is what gets compressed.
- Layered Agent Memory — compression is lossy working-memory management; durable facts should already have been captured by the memory tiers before the summary discards the raw exchange.
- Evergreen vs Volatile Context — the same keep-what-pays-rent discipline applied to in-window history rather than the knowledge store.
- Prompt Caching — the complementary economics: caching cheapens the stable prefix, compression shrinks the growing suffix.
- Context Smart Zone — the sharper limit compaction is really racing: attention degrades before the window fills, so the threshold that matters is the usable band, not the nominal one.
- Spec-Driven Development —
the deliberate form of this mechanism:
/to-speccompresses a 46.1k grilling session into a durable document at a moment the operator chooses. Compaction is the harness salvaging an overrun window; a spec is the operator refusing to have one — and unlike a summary, the spec is an artifact the next session (and its reviewer) can be held to. - Distillate: Hermes Architecture EXPLAINED: Memory, Context & Gateways
- Distillate: The whole flow, end-to-end: the smart zone is the unit of work — compression as a named pipeline stage rather than a harness reflex.