The gap beside the result
An additional 19 repairs sounds like a persuasive reason to choose a coding agent. The ContrastRepair comparison in Bintang Pratama’s November 15 review reports a difference of that size across 337 software bugs. What catches my attention is the missing information beside the gain: without knowing where the two methods disagree, their totals cannot determine the uncertainty of the comparison. Whether the baseline fixes the same bugs as the added mechanism belongs in the mathematics of that decision. A table of aggregate success rates leaves it out.[1]
The review maps 151 journal articles, codes 67 empirical studies and appraises 11 in full text. Seven studies provide ten matched contrasts; nine report magnitudes, all favoring the added mechanism. Relative gains of about 6–56% are sensibly left without a pooled average: correctness criteria differ, and some figures are the best result or a range across configurations. The scientific target is the additional number of independently correct repairs on the same bugs, using the same model and a comparable budget. Counts of candidates passing tests answer only part of that question.[1]
Comparison at the bug level
The review makes the missing table concrete with an explicitly hypothetical example. 30 bugs fixed only by the intervention and 11 fixed only by the baseline yield 19 additional repairs. So do 60 and 41. In the second case, the methods disagree on many more bugs. The review constructs these illustrative cells to show why the same aggregate difference permits different uncertainty. Publishing paired outcomes for each bug exposes both gains and losses. That is the information the review could not extract from its eligible comparisons.[1]
Even that table needs a carefully named outcome. Letting the tests used to generate or select patches also deliver the final verdict gives a system an opportunity to fit those tests. More attempts may also increase success; an alternative explanation is that the added mechanism produces genuinely better repairs. Holding the model and candidate or token budget constant, then applying an independent correctness check outside the generation loop, helps distinguish these possibilities. Such a design can measure effort alongside repair correctness. Equal aggregate success can conceal different costs of obtaining it.[1]
Apply the standard to the review
The review itself deserves the same scrutiny. Conference proceedings are excluded, although a companion search finds 119 of them. Full-text appraisal favors accessible studies and is not independently duplicated by a second reviewer. The protocol was not preregistered. These limits constrain any claim to a complete account of the field. Yet the unavailable paired cells remain a concrete reporting problem for the comparisons selected. Publication preference for positive results is another possible explanation for the uniformly favorable contrasts; a shared direction alone cannot establish a generalizable effect size.[1]
My decision rule for a research team evaluating a repair method would be to show what the gain costs on the same bugs. The review cannot calculate cost per independently correct repair alongside a matched baseline cost. Reporting money, tokens and time separately, each with the correct-repair denominator, would help. Paired bug outcomes, equal attempt budgets and independent validation would give the number 19 more substance for a research decision. They would make it possible to discuss whether the apparent advantage comes from more candidates or a stronger repair mechanism.[1]