Eigen RadarAI
Analysis

Software repair review finds missing data for comparing correct fixes

Bintang Pratama’s software repair review separates patches that satisfy existing tests from fixes judged correct through additional checks. It maps 151 journal articles and examines 11 full texts in detail. Seven studies offer matched comparisons, but missing bug-level paired results prevent a pooled estimate of the benefit from retrieval and tool-using systems.

Artificial Intelligence··Morning
Paired patch pages on a review wall, with gaps where a pair's second outcome is missing.

Paired outcomes are missing

Seven studies in Bintang Pratama’s new software repair review offered ten comparisons sharing a base model and bug benchmark. Nine reported the size of a gain, all favoring the added mechanism. Yet the missing paired results for individual bugs and missing variance information prevented a pooled effect estimate. The November 15 publication is a single review of existing research, examining how retrieval and tool-using iteration are evaluated in language-model repair systems.[1]

Passing tests and establishing correctness

A patch can satisfy the available tests while leaving behavior outside those tests incorrect. The review therefore separates test-passing candidates from repairs subjected to independent correctness checks. It also distinguishes an improvement associated with additional context from one bought with more candidate attempts. Supplying the exact fault location changes the task again. These conditions affect what a repair count measures, even when the compared systems use the same underlying model.[1]

Three levels of evidence

The review maps 151 journal articles from 2022–2026, contains 67 empirical studies in its core set, with 11 papers receiving full-text technical appraisal. Searches relied on OpenAIRE and Crossref; conference proceedings were excluded. Abstract coding and detailed appraisal remain separate layers, with different coverage. The proposed reporting set includes bug-level paired results, equal candidate budgets, independent correctness criteria and checks for exposure to benchmark fixes during training. Its scope is bounded by journal selection and accessible metadata.[1]

References

  1. News sourceJournal of Engineering and High TechnologySoftware repair review separates passing patches from independently correct repairs↩1↩2↩3