Why We Killed Chat History in Autonomous Coding Agents (and Gained a 2.44x Speedup)
If you run autonomous coding agents in multi-turn production loops (Claude Code, Hermes Agent, OpenCode, Aider), you have almost certainly watched this tragedy unfold:
Around Turn 8, your agent is flying. It understands the codebase, writes surgical edits, and runs tests.
Around Turn 22, something shifts. The prompt has swollen to 65,000 tokens of raw bash logs, previous tool outputs, and historical chat turns.
By Turn 35, the agent is hallucinating nonexistent helper functions, re-reading the same file three times in a row, hitting provider rate limits (HTTP 429: Rate limit exceeded), and—worst of all—treating its own past speculative hallucinations as established architectural facts.
We call this The Haystack Tax. And the industry’s default answer—“just give models a 1-million-token context window and dump everything in”—is mathematically and economically broken.
Here is why we completely eliminated accumulated conversation history in our autonomous coding runtime, replaced it with a Just-In-Time (JIT) Epistemic Context OS, and what happened when we benchmarked both approaches on a live 45-module production codebase.
The Three Fatal Bugs of the Accumulating Agent Loop
When you let an autonomous agent loop blindly append each turn’s output to the prompt, three failure modes emerge:
1. The Lost-in-the-Middle Attention Degradation
Frontier LLMs have massive context windows, but attention is non-uniform. As context expands beyond 15,000–30,000 tokens, the model’s ability to recall subtle domain invariants and type signatures drops sharply. The agent begins guessing interfaces it already inspected ten turns earlier.
2. Epistemic Self-Poisoning (The Confidence Laundering Loop)
In Turn 5, an assistant might speculate: “Perhaps auth_provider defaults to Redis.”
In Turn 12, that speculation sits inside the conversation history with role: "assistant".
By Turn 18, the model re-reads its own past statement, assumes it is a ground truth, and writes 200 lines of code against an imaginary Redis dependency. An LLM cannot reliably distinguish between its own past hypotheses and verified runtime reality unless the harness enforces epistemic weights.
3. Prompt Cache Thrashing & Rate-Limit Meltdown
When the prompt accumulates dynamic timestamps, changing terminal outputs, and shifting history at every turn, provider prompt caching breaks down. Instead of paying cached-read rates ($0.03/1M), you pay full uncached input rates ($3.00/1M) on every turn, burning through 5-hour quota windows in under 20 minutes.
The Solution: JIT Context OS & The Non-Accumulating Loop
Instead of dragging a growing haystack across turns, we redesigned the harness around a non-accumulating turn cycle:
- The prompt NEVER accumulates unbounded history. Each turn receives only three things:
- The static System Prompt with tool schemas (100% prompt-cache hit).
- The immediate previous tool execution result (the latest observation).
- A lean, dynamically compiled
<ONA_CONTEXT>capsule (400–1,200 tokens) generated just-in-time.
- Sub-3ms L0 Hot-Path (SQLite WAL):
Every mutating tool execution (write_file) and verification run (pytest,vitest) intercepts exit codes and sinks hard cryptographic facts (file hashes, exit codes, concise traceback needles) directly into a local SQLite database running with Write-Ahead Logging (PRAGMA synchronous = NORMAL). Zero network latency; true Read-Your-Own-Writes (RYOW). - Iron Epistemic Invariants (The Authority Hierarchy):
- Direct User Instructions: Authority =
1.0(Invariant I1: Direct user intent overrides all history). - Runtime Tool Verification (
exit: 0, test pass): Authority =1.0. - Tool Observations (
read_file, API responses): Authority =0.9. - Assistant Outputs & Speculations: Authority =
0.0. An assistant’s own thoughts cannot create root facts without a verifying tool execution.
- Direct User Instructions: Authority =
Empirical Proof: The Synthapse E2E Production Benchmark
We didn’t test this on toy leetcode puzzles. We ran a paired, head-to-head empirical benchmark on Synthapse—a complex, 45-module real-time Web Audio/WebGL techno synthesizer codebase with strict DSP invariants and 199 Vitest suites.
Two isolated agent instances were dispatched with identical objectives: implement full Web MIDI bi-directional hardware feedback, LED state duplexing, and parameter clipping guards.
| Dimension | Baseline (Standard Haystack Accumulation) | JIT Context OS (Sub-1.2k tok Dynamic Capsule) | Delta / Empirical Advantage |
|---|---|---|---|
| Wall-Clock Duration | 2,752.6s (~45.8 min) | 1,127.3s (~18.8 min) | 2.44x faster (-59.0% delivery time) |
| Total LLM Turns | 171 turns | 66 turns | -61.4% turns (105 API calls avoided) |
| Tool Execution Churn | 314 tool executions | 119 tool executions | -62.1% tool churn |
| Test Failures & Regressions | 7 runtime regression loops | 0 test failures (100% first-turn pass) | Zero error loops |
| Prompt Input Volume | >54,000 tokens (Turn 30+) | 482 – 1,140 tokens (strictly clamped) | >98% prompt token reduction |
| Cache Read Ratio | <18% (frequent cache invalidation) | >88% stable prefix cache hits | ~74% API cost reduction |
| Final Verification | 199/199 passing (after manual rescues) | 199/199 passing (autonomous, clean) | Identical quality, zero collateral drift |
The Core Finding
The baseline agent spent over 60% of its time recovering from errors it introduced itself—editing unrelated audio mixer modules because they were mentioned in old turns.
The JIT-governed agent operated with surgical focus: it knew the exact target files, the active invariants, and the latest verified test outputs. It made the changes, ran the tests, received verified exit: 0 proofs, and finished cleanly.
The Local Edge Duel: $599 Mac Mini vs Google Gemini 3.8 Flash (EXP-009)
Can small, open-weight models on consumer hardware compete with frontier cloud models if their context is properly governed?
We ran an autonomous multi-module engineering duel on an Apple Mac Mini M2 Pro (16GB RAM) fixing 4 distinct root causes across 3 interconnected Python modules (event_pipeline.py, retry_policy.py, storage.py), verified by a blind pytest suite:
| Competitor | Hardware & Runtime | Total Turns | Wall-Clock Time | Pytest Verification | API / Hardware Cost | Privacy Guarantee |
|---|---|---|---|---|---|---|
| 🥇 Local Qwen 3.8 9B + JIT | Apple Mac Mini M2 Pro (Ollama 32k) | 4 turns | ~2 min (126s) | 4/4 PASSED (exit 0) | $0.00 | 100% Local / Zero Data Leak |
| 🥈 Gemini 3.8 Flash + JIT | Google Cloud Frontier API | 9 turns | 17.7s | 4/4 PASSED (exit 0) | Paid Cloud API | Cloud API payload |
| 🥉 Gemini 3.8 Flash (No JIT / Chat) | Google Cloud Frontier API (Haystack) | 10 turns | 34.5s | ❌ 0/4 FAILED (lost in tests/) | Paid Cloud API | Cloud API payload |
What Happened:
- Turn 1 (Parallel Discovery): In one turn, the local 9B model fired 4 parallel
read_filecalls. - Turn 2 (Surgical Multi-File Fix): Guided by an ~1.8k token working-set capsule, it emitted 3 parallel
write_filecalls repairing all 4 bugs across 3 files simultaneously. - Turn 4 (Verification): Ran tests -> 4/4 passed (100% clean exit code 0) on the very first verification run.
Meanwhile, Google’s flagship Gemini 3.8 Flash without JIT drowned in chat history, drifted into modifying tests rather than fixing source code, and failed completely. Small models aren’t incapable of complex coding; they have been suffocated by conversational context bloat.
Where to Explore
We published the architectural specification, formal invariants, and empirical benchmark suites on GitHub and CERN Zenodo:
- GitHub Repository: github.com/wojciechwiesner/jit-context
- CERN Zenodo Permanent Record: DOI: 10.5281/zenodo.22649542
- Live System Telemetry: Running locally via macOS launchd and accessible through the JIT Observatory dashboard.
If you are designing enterprise multi-agent factories or high-reliability coding harnesses, stop expanding your context windows. Start governing your context runtime.