Your 1M-Token Context Window Can Still Forget: A Production Postmortem

A million-token context window sounds like the end of agent memory problems.

It is not.

On September 14, 2026, one of our production engineering sessions reached 437,606 active tokens across 1,690 messages. The runtime reported 119.2% occupancy against its operational compression boundary. The model still had a large nominal context window, yet the agent was already showing the failure pattern every autonomous-systems operator fears:

• completed subtasks were reported as complete products,
• explicit requirements disappeared behind recent tool output,
• the agent switched to adjacent work instead of finishing the active flow,
• repeated compression attempts failed without reclaiming context.

The model was not too small. The context runtime was misconfigured.

This is a postmortem of the failure, the numbers behind it, and the architecture change it forced in our JIT Context OS.

The incident in five numbers

Signal	Measured value
Base session before meaningful work	approximately 30,000–32,000 tokens
Incorrect full-compression trigger	35,000 tokens
Fixed micro-summary output ceiling	1,500 tokens
Largest observed active session	437,606 tokens
Messages in that session	1,690

A separate active session later reached approximately 279,778 tokens. Its rolling summary was already 1,192 tokens, while the summarizer was still limited to a maximum output of 1,500 tokens. Merging the next exchange required reproducing the existing summary and adding new facts. The output hit finish_reason=length, so the runtime correctly rejected the partial summary to avoid silent data loss.

Correct safety behavior. Catastrophic liveness behavior.

The exchange remained uncompressed, the session continued to grow, and the same failure repeated.

A large context window is capacity, not governance

The first diagnosis was tempting but wrong:

Gemini cannot handle more than 35,000 tokens reliably.

That was not what happened. Gemini 3.8 Flash exposed a much larger context window. Hermes itself was configured to trigger compression at 35,000 tokens.

Those are completely different statements.

Model context size answers:

How much can the provider technically accept?

Context governance answers:

What enters the prompt, what remains verbatim, what is externalized, what is summarized, and when each transition happens?

A model can accept one million tokens and still perform badly if the first 300,000 are dominated by stale terminal output, duplicated schemas, historical assistant speculation, and irrelevant branches of work.

The context window is storage capacity. It is not an information architecture.

Root cause 1: compression started almost immediately

The session’s static starting footprint was already around 30,000–32,000 tokens:

• system and engineering instructions,
• user and persistent memory,
• available skill metadata,
• tool schemas,
• project context,
• the current JIT capsule.

With threshold_tokens: 35000, a single substantial tool call could cross the boundary. Full compression then ran every one or two turns.

That behavior created two problems:

1. Cache churn: frequent rewrites destroyed a stable prompt prefix.
2. Semantic churn: repeated summaries compressed material before the task had reached a stable checkpoint.

The fix was not to disable compression. Disabling it would only postpone failure until the provider limit.

We removed the fixed token threshold and restored a dynamic threshold equal to 35% of the active model’s context window. A large-context model now gets a proportionally larger runway; a smaller local model still receives an earlier safety boundary.

Root cause 2: spillover existed, but did not fire

Hermes already had a good mechanism for large tool results:

1. save the complete result to disk,
2. retain a short preview in the model context,
3. include a file path for targeted retrieval.

But the defaults were too permissive for tool-heavy coding sessions:

Budget	Previous value
Single tool result	100,000 characters
Aggregate tool output in one turn	200,000 characters
Preview after externalization	1,500 characters

Most real results were between 10,000 and 70,000 characters. They never crossed the per-result limit. Several could accumulate in one turn without crossing the aggregate limit.

The mechanism existed in code but was practically dormant.

There was hard evidence: the spillover directory contained no automatic dumps from the affected sessions, and the logs contained no Persisted large tool result events.

We changed the operating budget to:

Budget	New value
Single tool result	12,000 characters
Aggregate tool output in one turn	24,000 characters
Inline preview	800 characters
MCP result	10,000 characters
read_file response cap	20,000 characters
Terminal response cap	12,000 characters

A runtime check passed: a 13,000-character tool result was written in full to spillover storage, while only a 1,304-character preview and recovery path remained inline.

Nothing was lost. The model simply stopped carrying the whole file on every subsequent turn.

Root cause 3: a fixed summary budget created a deadlock

Micro-compaction folds older exchanges into a rolling summary. Before the fix, every merge used this ceiling:

max output = min(1,500 tokens, global summary limit)

That works while the rolling summary is small. Once the summary approaches the ceiling, every future merge must fit:

existing summary + new facts + structural overhead

into nearly the same space as the existing summary alone.

The system enters a compression deadlock:

1. the summary is too large to reproduce and extend,
2. the model returns a truncated merge,
3. the runtime rejects the partial result for safety,
4. the original exchange remains active,
5. the next attempt starts with even more context.

We replaced the fixed ceiling with an elastic output budget:

existing summary tokens
+ up to 1,000 tokens from the incoming exchange
+ 512 tokens of merge headroom

The value remains bounded by the compressor’s global summary cap, currently 4,000 tokens. Small merges still use 1,500. Mature summaries receive enough room to complete safely.

The focused compression, spillover, and budget suite completed with 98 tests passing.

Why JIT did not prevent the failure

The JIT capsule was not the source of the bloat. It remained a comparatively small, relevant bundle containing current intent, project state, active invariants, and evidence references.

But it entered a prompt already carrying a large static prefix and an unbounded stream of inline tool results.

A precise 400-token capsule cannot dominate 250,000 tokens of stale evidence merely because it appears near the end.

This exposed an important design boundary:

Retrieval quality and context lifecycle management are separate systems. A production JIT architecture needs both.

JIT chooses the right evidence. The host runtime must also control how long every other artifact remains active.

The corrected architecture: hybrid compression

The resulting architecture is deliberately hybrid.

Stage 1: deterministic spillover

Oversized tool results are written unchanged to local storage. The active prompt keeps only:

• the tool name,
• a bounded preview,
• the original size,
• the recovery path.

This is lossless and does not require an LLM.

Stage 2: deterministic pruning

Before semantic compression, the runtime:

• deduplicates byte-identical tool results,
• replaces old results with compact references,
• truncates oversized historical tool-call arguments,
• retires stale image payloads.

Again, no LLM is involved.

Stage 3: LLM micro-compaction

An auxiliary model merges completed older exchanges into a rolling semantic summary. In our runtime this role is assigned to Gemini 3.8 Flash.

The output budget is elastic, and a truncated response is never committed.

Stage 4: LLM full compression

When the dynamic context threshold is reached, or when the operator runs /compress, the auxiliary model summarizes the old middle of the conversation.

The system prompt and newest conversation tail remain verbatim. Historical rows are archived in the session database rather than destroyed.

This separation matters:

• deterministic stages handle bulk data and exact references,
• the LLM handles meaning,
• the database preserves auditability.

What this changes about context engineering

Many teams still treat context engineering as prompt composition:

Retrieve the best documents, then concatenate them.

That is only the input side.

A reliable agent runtime needs five policies:

1. Admission: what is allowed into active context?
2. Externalization: what should be stored outside the prompt immediately?
3. Retention: what must remain verbatim, and for how long?
4. Compression: what can be summarized, by which model, under which safety checks?
5. Verification: how do we prove that compressed memory still supports the real task?

Without these policies, a larger context window simply permits a larger haystack.

The operational invariant we added

The incident also exposed a product-completion failure. The agent had backend adapters, unit tests, and routes in the repository, but some user-visible flows were not complete in the live interface.

We formalized the rule as Bit-fals:

A feature is not DONE until its backend, frontend routing, UI interaction, and runtime effect have all been verified.

A unit test is not a browser render. A route returning HTTP 200 is not a successful image generation. A synchronization button is not a bidirectional calendar until a real read-write-read round trip succeeds.

Context governance and verification governance are connected. Once the agent loses the active acceptance criteria in a haystack, it starts optimizing for locally convenient proxies of completion.

The practical checklist

If you operate coding agents with long context windows, check these before blaming the model:

• Is the compression threshold proportional to the model window?
• Is the static first-turn prompt already near that threshold?
• Are 10–50k tool results externalized immediately?
• Can the summary budget grow with the existing summary?
• Does finish_reason=length prevent committing partial summaries?
• Are the newest user instructions and runtime evidence preserved verbatim?
• Can archived evidence be retrieved without reinjecting the entire archive?
• Does DONE require a real product-level verification path?

The lesson

We started with a reasonable belief: a precise JIT capsule should keep an agent focused.

The production incident refined it:

A JIT capsule is only as strong as the context runtime carrying it.

The best retrieval system cannot compensate for dormant spillover, pathological compression thresholds, or a summarizer trapped below its own output ceiling.

Large context windows are useful. But the moat is not how much context an agent can hold.

It is how deliberately the system decides what the agent must see now.

The JIT Context OS architecture, benchmarks, and source are available on GitHub, with a permanent research record at CERN Zenodo.