Terminal-Bench audit finds defects that shift coding-agent scores
Researchers examined how defects in Terminal-Bench tasks affect the scores assigned to coding agents. Their audit and earlier findings identify problems in twenty-eight of eighty-nine tasks, including inconsistent instructions and tests, changing dependencies and resource limits. Reevaluation with nineteen agents leaves the broad ranking strongly related to the earlier one while changing some nearby positions. Execution time also affects the comparisons.
Artificial Intelligence··Night
Defects affect twenty-eight tasks
Researchers affiliated with Stanford and Cornell universities examined Terminal-Bench, a collection of practical terminal tasks for coding agents. Their audit and previous findings identify issues in twenty-eight of its eighty-nine tasks. Some instructions and grading tests disagree; elsewhere external dependencies change or the execution environment fails. A grading condition requiring a file to remain untouched can reject valid work when the instructions never state that condition.[1]
Human maintainers decide which repairs ship
The workflow brings task files, earlier execution traces and community reports together. An auditor proposes a defect classification; a separate verifier reruns the known-correct solution and challenges the repair. Human maintainers approve changes. Agreement between auditing models is not a correctness guarantee. Because this workflow helped assemble the eventual repair set, agreement with that set measures retrospective consistency rather than independent discovery accuracy.[1]
Budgets also change agent comparisons
Reevaluation with nineteen agents preserves a strong relationship between the broad rankings while shifting some close positions. Longer time limits also narrow certain gaps between agent setups. Resource repairs assess the binding time, memory or processor constraint separately and allow headroom above the known-correct solution’s requirements. The preprint discusses this maintenance process without establishing that every repair first became available on its publication day.[1]
