We Gave Our Agent a Conscience and a Judge. The Plumbing Fixed More.

We spent a week building cognition for our agent runtime. A conscience gate (we call it Sumienie) that checks whether an answer is grounded in what the tools actually returned. A judge (Rozwaga) that runs a second attempt when the conscience is not satisfied and picks between the two. A sensory layer and a planner in front of the worker.

Then we measured it on GAIA Level 1, the 53 validation tasks. The result is not the one we wanted to write about, so this is the one we are writing about.

The runs

Worker: gemini-3.8-flash. One run per row. “Exact” is a normalized string match against the ground truth.

Run	Exact	Avg time per task
Before fixes, 8 turns	34/53	29 s
Sensory layer and planner added	33/53	31 s
Loop guard and forced final answer, 8 turns	40/53	26 s
Same, 15 turns	45/53	36 s
Plus conscience gate and judge	47/53	62 s

From 34 to 45 is plumbing. From 45 to 47 is the cognition layer, and as shown below, even that is within single-run noise. The sensory layer and the planner did nothing measurable, so they are coming out.

What the plumbing was

Fourteen of the 53 answers in the first run were empty. The model had done the research. It was just never allowed to answer.

When the loop ran out of turns, the runner asked for a final synthesis but sent the history without the tool declarations. Gemini replied with a tool call anyway, the parser did not expect one, and an except: pass swallowed it. Fourteen empty strings, silently scored as wrong.

The other fixes were just as unglamorous:

• The turn budget was hardcoded to 8 in one place and never passed to the worker from the command line. Raising it to 15 was worth 5 tasks.
• A loop guard now skips exact repeat calls and flags searches that return nothing new, then tells the model to change strategy.
• The last turn disables tools, so the model has to answer instead of searching once more.

None of this is intelligence. It is making sure the intelligence we already paid for reaches the scorer.

What the judge actually did

The judge ran on 16 of 53 tasks, the ones the conscience did not verify. It changed the answer 3 times: twice to the correct one, once to a wrong one.

Compared with the 15-turn run without it, the final score moved +4 and -2. With one run each, that is inside the noise of a model sampling at a non-zero temperature. The three switches do not show a clean pattern either: one came from the LLM comparison and was right, two came from preferring the only candidate the conscience verified, and those split one right, one wrong. A grounded answer is not always a correct answer. On half of the 16 tasks, both attempts agreed and the judge had nothing to decide.

It also cost 1.7x the time per task.

What a 9B model taught us about our own harness

We ran the same first 10 tasks on qwen3.8:jit, a 9B model on a Mac mini. It scored 3/10, where Gemini scored 10/10 on those same tasks.

The easy conclusion was that small models cannot do GAIA. Then the error logs showed four HTTP 400 responses that ended tasks early, and we had not logged the body. Once we did: “Cannot have 2 or more assistant messages at the end of the list.” When the model answered in plain text without a tool call, our runner appended that turn and asked again. Gemini tolerated it. Ollama rejected it.

So part of that 3/10 was our bug. We fixed it and reran the same 10 tasks with the judge on. The 9B model scored 8/10, all eight verified by the conscience. No HTTP 400s this time. It did hit four HTTP 500s: llama-server returned tool-call arguments cut off mid-JSON. That is a local serving problem, not ours, but it still cost one task an empty answer.

	qwen3.8:jit, before the fix, no judge	qwen3.8:jit, fix + judge	gemini-3.8-flash + judge
Exact	3/10	8/10	10/10
Avg time per task	107 s	316 s	37 s
API errors	4 HTTP 400 (our bug)	4 HTTP 500 (llama-server)	0

The fix and the judge went in together, so this run cannot say how much each one did. The log says this much: the judge ran on 5 of the 10 tasks. Twice the first attempt came back empty and the second was right and verified. Once it chose a verified short “3” over a full sentence. Once it chose a verified but wrong paper title, the same failure we saw with Gemini. Once both attempts were empty. The other three gains happened without the judge.

Ten tasks, one run each, and the local model is 8.5x slower per task than Gemini. It is a signal that a 9B model on a Mac mini can do real GAIA work once the plumbing is right, not a score.

Four rules we are keeping

1. Measure the plumbing before the brain. Count empty answers, API errors, and turns that hit the budget before touching reasoning layers. Our biggest gain came from a missing tool declaration.
2. A judge is only as good as its evidence. Being grounded in tool output is necessary, not sufficient: the conscience verified one wrong answer that the judge then picked.
3. Record provenance from the source. Our result files had the commit and the version hardcoded to an old release. They now read git rev-parse and pyproject.toml. A number you cannot tie to a commit is only an anecdote.
4. Say what the number is. These 53 tasks are the same ones we tuned the harness on. This is a development-set score from single runs, not a leaderboard entry. Before we publish a headline number we need the full 165 validation tasks, three runs each, and the official scorer.

The code, the raw result JSON, and the changelog are in the jit-context repository. This series of posts starts with The One Who Is Many.

If you are building agents and want to compare harness notes, join the community. If you run a product and want this kind of measurement on your own agent, get in touch.