Same Session, Three Modes: What JIT and a Live JEV Call Actually Changed
The useful question is not “does a bigger window fix the agent.” It is “what changes when you stop dumping the whole trace into the next turn, and what changes again when a second model ranks the files.”
We ran that question once, on the same ten fixtures, in one session. Three modes, interleaved per task: no JIT, JIT with heuristic file ranking, and JIT with a live call to JEV. The scorecard is on theones.io/benchmarks/jit-jev. The raw log is the JSON.
This is not official SWE-bench. The harness builds isolated multi-file trees from inline fixtures plus decoy files. It does not check out Django, Flask, Requests, Astropy, Sympy, or pytest.
The totals
Agent model: gemini-3.8-flash. JEV model: typesafe/jev-1.13-20260917. Date: 2026-09-23.
| Mode | Pass | Wall | Turns | Tokens | Discovery | Live JEV |
|---|---|---|---|---|---|---|
| No JIT | 10/10 | 207.6s | 69 | 57,989 | 40 | 0 |
| JIT, no JEV | 9/10 | 138.5s | 52 | 63,202 | 26 | 0 |
| JIT + live JEV | 10/10 | 132.0s | 49 | 64,797 | 23 | 10/10, fallback 0 |
JIT plus a live JEV call is 132.0 / 207.6 = 1.57x on wall clock against the no-JIT run. Turns fell from 69 to 49. Discovery calls fell from 40 to 23.
Tokens moved the other way: 64,797 against 57,989. The capsule adds prompt. It does not shrink the bill. Gemini dollar cost is absent from the API response, so it is not printed. The JEV usage field recorded $0.000261 for the ten live calls.
What JEV added on top of JIT
Against heuristic JIT, the live JEV path saved 6.5 seconds and 3 turns. That is the whole gap. It is not a different product.
JEV was slower on django-11848 (30.62s vs 23.42s) and flask-4045 (11.07s vs 6.72s). It was faster on the rest, except pytest-5221, where the comparison is dirty.
The heuristic 9/10 is not a failed pytest. On pytest-5221 the model returned a body the harness could not parse (Expecting value: line 1 column 1). The tests never ran. Do not read that row as “JEV rescued a bug the ranker missed.”
Every JIT+JEV task logged HTTP 200, model typesafe/jev-1.13-20260917, fallback count 0. Average JEV latency on an earlier live-only run of the same fixtures was about 505ms. This comparison log records the call, the model, and the usage cost. It does not claim a sub-millisecond ranker.
What this does not say
An older fixture log (150.9s raw, 133.9s on a JEV-labeled path, 9/10) is a different session. The JEV path in that log did not record whether the call was live or a heuristic fallback. It is not this table.
The 2.44x figure is the Synthapse paired run: 45 audio modules, 199 Vitest suites, 9m 27s to 3m 52s. It is a different codebase and a different harness. Putting it on this chart would be a lie.
Pass rate is not the story. No JIT already passed 10/10. The measured change is time, turns, and how many files the agent had to go find.
The visual page, with per-task bars and the caveats in the same view, is here.