More code or more information?
A program’s source is unavailable, but its executable and documentation remain. For a developer rebuilding its behaviour, the first problem is identifying what the specification still leaves unknown. The new Cohere experiment uses ProgramBench to create that situation. Workers can probe the reference program and return their findings to a lead before implementation. Extra agents therefore add two potentially different resources: capacity to write code and information about the behaviour that code must reproduce. The design lets those explanations compete rather than assuming every additional worker contributes in the same way.[1]
The ProgramBench preprint specifies hidden behavioural tests for reconstructed programs. Neither architecture nor implementation language has to match the original. Cohere averages the share of tests passed across 200 tasks, rather than reporting the share of programs fully resolved. That distinction governs the result’s meaning. A higher score can indicate better reconstruction of some behaviours without establishing a complete application. The finite test suite also cannot supply an exhaustive specification covering every possible input. This is a useful relative endpoint for team design, with a narrower interpretation than software ready for unrestricted use.[1]
Cohere reports a substantial contribution from planning when it compares planning, implementation and verification support separately on ProgramBench. Most main configurations were repeated three times on the same tasks, with paired comparisons to solo agents. The revealing observation precedes the score: additional planners probe more reference behaviour and return larger reports. Additional implementers receive smaller pieces of code, without a comparable increase in checks against the reference. That pattern supports the information-gathering explanation. It does not establish that different delegation prompts or coordination arrangements must produce the same outcome.[1]
When the specification is supplied
The secondary test changes the task structure. Commit0-Lite supplies repository skeletons, documentation and unit tests for 16 Python libraries. Cohere reports the greatest contribution from implementation support there. The job still involves producing software, but what to build is more explicit. A change in the useful delegation stage within the same research programme weakens the claim that planning workers are universally preferable. The solo agent already passes every test on 12 tasks, leaving limited headroom in this small secondary sample.[1]
The contrast alone does not settle causality. Program families and difficulty also change; the researchers have not varied only information availability within identical tasks. That is a viable alternative explanation for the planning advantage. Some fixed-worker configurations were run only once, further limiting interpretation of small score differences. Still, the observed probing behaviour and the direction of the secondary test are mutually informative. The narrower conclusion is that the value of assistance needs to be measured together with the conditions of the task.[1]
For a development team, the experiment suggests a practical opening question: is work stalled on discovering behaviour, implementing known behaviour or validating the combined pieces? The lead remains the decision point that integrates workers’ reports in Cohere’s setup. I find that useful for naming the bottleneck before choosing team size. Comparing identical tasks with an explicit specification and with only the executable could test the information-gathering explanation more directly. That would distinguish where planning effort helps without converting one benchmark’s favourable stage into a general rule for software teams.[1]