Learning behavior before writing code
On ProgramBench, an agent must reconstruct a program without seeing its code. It has an executable and documentation and learns behavior by probing. In Cohere’s experiment, planners attached to a Sol lead multiply those probes: increasing planners from two to 16 raises total attempts from 15 to 109. The information collected by collaborators precedes implementation. That task structure is the center of the finding for me. Expecting the same result in every software workflow treats behavior that must be rediscovered and source code that can already be read as equivalent starting conditions.[1]
The endpoint matters too. Across ProgramBench’s 200 tasks, the reported score averages the fraction of hidden tests passed. It is not the fraction of programs completely solved. Solo Sol reaches 57.7% under a plan–implement–verify workflow, while a blocking team reaches 63.1%. The increase shows that more reconstructed behavior agrees with the tests. It does not establish that every program is ready for use. For a team interpreting success as a finished deliverable, this distinction in the denominator comes before the number of planners.[1]
The same team, a different starting point
Cost comparisons show that choosing where to delegate is itself part of the experiment. Asynchronous planning alone reaches 63.6% at $249. Collaboration across all stages costs $510 for 63.2%. A fixed team joining a strong lead to 16 inexpensive Luna planners reaches 61.8% at $155. I cannot read that last result as evidence that cheap planners always preserve quality: fixed teams ran only once. The uncertainty assessment from three repetitions in most dynamic configurations cannot simply be transferred to that single run.[1]
Commit0-Lite, where source code already exists, tests the idea under a different starting condition. In eight of ten team configurations, dividing implementation works better than dividing planning. That supports the interpretation that missing information in a black-box task increases the value of planning. It need not be the sole explanation. The second set contains only 16 Python repositories, and solo Sol already solves 12. Repository selection and limited remaining headroom may also affect the ordering of strategies. Task type and sample size change together, so their contributions need to be separated.[1]
A measurable basis for dividing work
A strength of the experiment is that collaboration is not left as a single on-or-off choice. Blocking and asynchronous helpers, planning and implementation assignments, and stronger or smaller models are compared separately. That makes the configuration behind a score increase visible. The results belong to Cohere’s researchers; they do not demonstrate replication by another team. Within those boundaries the finding is useful: gathering information through executable probes and modifying known code present different division-of-labor problems.[1]
The decision criterion I take from this study is to track both the helper’s activity and the lead’s improvement in completing the task. 109 probes are not automatically 109 useful discoveries. Planning gains become clearer when configurations use the same programs at comparable cost and separately report completely solved tasks. The experiment suggests that extra agents help particularly where behavior must be discovered. Its boundary is visible in the small second sample with code available at the outset. Measuring the initial information gap offers a firmer basis for choosing team size than assuming a general advantage from collaboration.[1]