A controlled swap, with a rerun yardstick
Most harness comparisons run each configuration once and report the gap. Liu and Han's design adds the control those comparisons usually skip: ten rerun pairs of the exact same configuration, so a harness gap can be read next to the movement a score shows when nothing changes at all. That yardstick matters, because, as the paper notes, ten reruns of one configuration on SWE-bench Verified already spread by 2.2 to 6.0 points.
The authors excluded eight tasks from the original 500-task benchmark, leaving 492: a 45-task hard subset and a separate 447-task pool. All five models ran the hard subset; the 447-task pool was run with the two Qwen models, Qwen3.6-35B-A3B and Qwen3.8-27B. The cross-harness experiments used five models and did not test Claude, GPT or Gemini across the three harnesses. Claude Opus 5 appears separately as a Claude Code-only reference, not as evidence about harness swapping.
In the configurations tested, most measured pass-rate differences between harnesses were small relative to the variation seen when the same setup was rerun. On the 447-task pool, the authors found Claude Code and mini-SWE-agent equivalent within a five-point margin for the two Qwen models. That does not establish identical performance or rule out smaller differences. The measured gaps were small: mini-SWE-agent finished 1.8 points ahead of Claude Code on Qwen3.6-35B-A3B and 1.4 points ahead on Qwen3.8-27B. On the hard 45 tasks, the three harnesses spread by two to five tasks per model, and rerunning a single harness moved it by up to three. The tasks one harness won in a run were not the tasks it won in the next. Even a vendor's own harness running its own model scored inside the range of the other harnesses.
The one effect that cleared the noise was a loss. OpenCode trailed by up to nine points on the 447-task pool, and on Qwen3.8-27B about half of that gap sat in runs cut short when the model hit its output limit mid-step, before a patch was written: 33 OpenCode trials ended that way on that model, against one for Claude Code, which re-prompts the model to continue. The authors' summary is blunt: “Every harness effect we can name is a way to lose a task, and none we measured is a way to win one.”
Follow the preamble
The bill is a different story, and it is set early. Per call, the three harnesses hand the model roughly the same amount of tool output. What differs is what each harness carries into every step: a fixed preamble of system prompt and tool schemas, 16,581 tokens for Claude Code against 829 for mini-SWE-agent, paid again on each of up to 300 steps per trial. Claude Code also keeps the model's own reasoning in the transcript, so the transcript the next step resends grows far faster than the other harnesses'. On GLM-5.3-Flash, that per-step weight becomes the bill: a trial costs 1.81 yuan under Claude Code, 0.97 under mini-SWE-agent and 0.56 under OpenCode. Those figures are the paper's calculations under its stated price list — the token cost of running the model — not subscription prices anyone pays for Claude Code or OpenCode themselves.
Prompt caching softens this. Between 96% and 99% of input tokens in these runs hit the provider's cache. Under the cache-pricing scenarios evaluated in the paper, changing cached-input prices did not reverse the relative cost ranking for a given model.
There is one reversal worth keeping. On DeepSeek-V4-Flash, the lightest harness took 141 model calls per trial to Claude Code's 104, and its growing transcript erased the preamble advantage: mini-SWE-agent ended up moving the most tokens and costing slightly more, at 1.02 yuan per trial against Claude Code's 0.98 at DeepSeek's list price. Per-step weight and step count trade against each other, and which one wins depends on the model. In this DeepSeek-V4-Flash configuration, mini-SWE-agent's greater call count outweighed its smaller fixed preamble, making it slightly more expensive despite being the lightest harness by prompt size.
What a claimed gain has to clear
The rerun data also set the resolution any harness comparison can claim. At the discordance the authors observed, 45 tasks catch a 13-point gap only half the time, and 447 tasks resolve about five points. The paper notes that many published harness gains sit below the first of those numbers, and some below the second. Under the authors' observed discordance assumptions, their power analysis indicates that even a 447-task comparison has limited ability to resolve differences much below five percentage points. A cited vendor claim of a 2.4-point gain would therefore need a more sensitive evaluation design or additional evidence to establish that the improvement exceeds run-to-run variability.
For buyers, the practical reading is narrow. A harness pitch that quotes a pass-rate gain should come with reruns, a task count large enough to resolve the claimed gap, and a token bill under a stated price list. The numbers worth asking for are the ones the authors end on: the size of the first call, the growth per step, and the step count. Those three decide what an agent costs. The score is the separate question, and in the configurations tested it is the one a single run cannot answer.
Sources
- [1] Liu & Han, “What Does a Harness Buy? Tokens, Mostly.” — Yangze Liu and Zhongyi Han, Shandong University, arXiv 2610.04433 (submitted October 3, 2026; HTML full text read first-hand by ALFW) — the three-harness comparison on SWE-bench Verified, the rerun yardstick and power analysis, the preamble and cost accounting under one price list, the OpenCode output-cap loss on the 447-task pool, and the paper's citation of a vendor's advertised 2.4-point gain over Claude CodeRead source
- [2] VentureBeat, “A coding agent's harness can triple its cost without making it more accurate, study finds” (October 23, 2026)Read source
- [3] The Decoder, “Fancy harnesses make coding agents more expensive but not better, study finds”Read source
- [4] AI Weekly, “Shandong Study: Fancier Coding Harnesses Triple Cost, Not Accuracy”Read source