In agent benchmarks, the harness moves the score as much as the model, sometimes more. The word has two senses: the agent harness (the scaffold around the model, which must be disclosed) and the eval harness (the test runner, which must stay constant). Zhang et al. (arXiv, May 2026) show it often outweighs the model itself on long-horizon tasks.
Two things called 'harness'
Before you compare scores, separate the two things people call a 'harness'. The agent harness (or scaffold) is the code, the prompts, the tools, and the loop around the model, everything the agent uses to actually do the task. The eval harness is a different thing: it's the test rig that runs the benchmark and scores it, like SWE-bench's Docker runner or EleutherAI's lm-evaluation-harness. One you design to improve the result. The other has to stay fixed for the comparison to mean anything. Conflating the two breaks any comparison, because you can't tell whether the score changed from the model, the scaffold, or the rig.
Same model, different scores
The proof that the scaffold moves the score came as a number. LangChain ran the same model, gpt-5.2-codex, on Terminal Bench 2.0 and, changing only the harness, went from top 30 to top 5: from 52.8 to 66.5, a gain of 13.7 points (February 17, 2026). Same model, different scaffold, materially different result. And it isn't a one-off. Per Scale and Digital Applied, on the Pro variant of SWE-bench the scaffold alone accounts for about a 5.2-point range, and swings of 10 to 20 points are attributed to the harness. Those numbers are theirs, not our measurement.
The two senses, side by side
| Term | What it is | In the benchmark |
|---|---|---|
| Agent harness (scaffold) | Code, prompts, tools, and loop around the model that the agent uses to do the task | Changes the score; must be disclosed for the comparison to hold |
| Eval harness | The test rig that runs the benchmark and scores it (SWE-bench Docker runner, EleutherAI lm-evaluation-harness) | Must be held constant across the models being compared |
| Model | The weights that generate the tokens | One variable, not the only one; the scaffold can outweigh it on long tasks |
| Terminal-Bench 2.0 | Benchmark of terminal and engineering tasks (tbench.ai) | Where LangChain changed only the harness and gained 13.7 points |
| SWE-bench | Benchmark of resolving real GitHub issues (SWE-bench/SWE-bench) | On the Pro variant, the scaffold alone accounts for about a 5.2-point range (Scale, Digital Applied) |
Why leaderboards mislead
If the harness moves the score this much, comparing two models without disclosing the harness doesn't say much. That's the argument in Zhang et al., 'Stop Comparing LLM Agents Without Disclosing the Harness' (arXiv, May 7, 2026): on long-horizon tasks the harness is 'often more influential than the underlying language model', and leaderboard comparisons without harness disclosure are 'incomplete and potentially misleading'. In practice, when you read that model A beat model B, ask whether the eval harness was the same and which agent harness each one used. If that isn't in the table, the comparison doesn't isolate the model.
What the benchmark teaches builders
The lesson is practical. If the harness moves the score, the harness is where you invest engineering, not just in picking the model. That's what Capiva works on. We solve the coding layer with capivaOS, the open-source harness on top of Claude Code. And to understand the whole discipline, from the model to the operational harness, the place to start is What is an AI Harness.
Frequently asked questions
Why does the same model score differently on benchmarks?▼
Because the agent harness (the scaffold) around it changes. With the same weights, a better scaffold reads the repository better, uses sharper tools, and self-corrects more. LangChain showed this: same gpt-5.2-codex, different harness, 13.7 more points on Terminal Bench 2.0.
What are the two senses of 'harness' in a benchmark?▼
The agent harness (or scaffold) is what the agent uses to do the task: code, prompts, tools, loop. The eval harness is the rig that runs the test and scores it. The first changes the score and must be disclosed; the second must be held constant for the comparison to be fair. Conflating the two breaks any comparison.
What is an eval harness?▼
It's the infrastructure that runs the benchmark and computes the score. Examples: SWE-bench's Docker runner, which runs the real issues and checks whether the patch passes the tests, and EleutherAI's lm-evaluation-harness. It has to be identical across the models being compared, otherwise you're measuring the rig, not the model.
How much does the harness change the score?▼
It depends on the task, but the effect is large. LangChain got +13.7 points on Terminal Bench 2.0 just by swapping the harness. Per Scale and Digital Applied, on the Pro variant of SWE-bench the scaffold alone accounts for about a 5.2-point range, and swings of 10 to 20 points are attributed to the harness.
How do you compare agents fairly?▼
Hold the eval harness constant, disclose each agent's harness, and report both. Zhang et al. argue that leaderboard comparisons without disclosing the harness are incomplete and potentially misleading. Without that disclosure, the score gap could be the scaffold, not the model.
Does the harness matter more than the model?▼
On long-horizon tasks, Zhang et al. say the harness is often more influential than the model. Not always, and the model still counts. The point is you can't isolate the model without fixing the harness. Both are variables; on long tasks, the scaffold tends to dominate.
Hold the eval harness constant, disclose the agent harness. Without that, the score isn't about the model, it's about the scaffold.