Coding-agent harness costs diverge more than most test scores
A study holding language models fixed across three coding-agent harnesses found task costs could vary threefold. Claude Code and mini-SWE-agent returned similar results on the larger task pool, while OpenCode fell behind. Repeated trials showed how small score gaps can blend into ordinary run variation. The findings mainly concern one software-repair benchmark and the tested configurations.
Artificial Intelligence··Midday
Repeated instructions raised the bill
With the model, tasks and price list held fixed, task costs varied by up to threefold. Instructions and tool definitions resent at each step occupied about 16,600 tokens in Claude Code, 7,000 in OpenCode and fewer than 1,000 in mini-SWE-agent. Additional steps could offset a shorter initial prompt. The findings mainly concern one benchmark, Researchers also made a small comparison on Terminal-Bench 2.1. Testing of Claude Opus 5 was confined to Claude Code.[1]
The same models ran inside three harnesses
Shandong University researchers Yangze Liu and Zhongyi Han investigated how much the software surrounding a coding agent changes its results. The Decoder’s October 24 coverage describes findings from their October 3 preprint. This surrounding software, called a harness, supplies a language model with system instructions, tools and conversation management.[1]
The researchers tested five models in Claude Code, OpenCode and mini-SWE-agent. They used 45 difficult tasks and a separate pool of 447 from SWE-bench Verified, a software-repair benchmark. mini-SWE-agent provides only a Bash command tool; the other two offer specialized tools and automatic history compression.[1]
Reruns shifted the apparent score gaps
For the two models tested on the larger pool, the score gap between Claude Code and mini-SWE-agent stayed below two percentage points. The statistical comparison placed them within a five-point equivalence band. On the difficult tasks, an unchanged setup could solve up to three more or fewer tasks when run again. OpenCode’s shortfall reached nine points in the larger test. For one model, roughly half its deficit was linked to a limit on output, while tighter time limits for local models complicated the comparison.[1]
