Nine attempts counted as failures
A research agent can reach the wrong conclusion. Its evaluator can also reject valid work. The detail that stands out in the FrontierPhysics preprint is that the second possibility becomes countable: experts reviewing 168 single-agent GPT-6 Astra attempts reclassified 9 apparent failures as successes. The pass rate rose from 14.9% to 20.2%. The model did not learn new physics during that correction. The judgment of its work changed. Measuring research ability requires treating the measuring instrument itself as a possible source of error.[1]
Those rates belong to one particular audit. They provide no justification for adding the same correction to every model. The study covers 56 tasks contributed by 42 physicists, with different model configurations assessed by the same automated reviewer. Detailed expert inspection concentrates on the selected model’s attempts. That asymmetry matters: the broad comparison table and the deep human audit have different coverage. A correction to one model’s score cannot tell us how much the entire ranking changes. Revising tasks and remeasuring affected results is a valuable step by the authors because it makes that uncertainty visible.[1]
Validity decisions and quality scores
The evaluation’s structure explains why this error matters. An attempt must pass both an executable verifier and every blocking requirement for scientific validity. Other criteria, including aspects of presentation or coverage, receive scores. A well-written result cannot compensate for failing a blocker. That distinction makes scientific sense: persuasive communication and a valid solution are different properties. But when the reviewer misinterprets a blocking requirement, the error is more than a small adjustment to a quality score. It can change whether the work counts as successful at all.[1]
My conclusion is to make the reasons for rejection auditable, rather than loosen the blocking requirements. An alternative explanation remains explicit: some disagreements may originate in ambiguous task instructions or criteria rather than the model’s limitations. The discovery and correction of task-design problems in the preprint makes that possibility concrete. A single failure label provides incomplete information when it merges an incorrect calculation with a misunderstood assignment. Separate reasons give model developers a clearer basis for investigating physics knowledge, working procedure or the evaluation definition itself.[1]
A second reviewer and external validation
Using another model as reviewer supplies a useful comparison. Regrading with Claude Opus 5.5 produces the same pass decisions for almost all GPT-6 Astra attempts, but the reviewer is more lenient toward work from its own model family. Agreement between two automated reviewers therefore does not establish that they replace expert judgment. Knowing which solutions break that agreement is more informative. Leniency varying by model family raises a specific question about whether the same measuring instrument treats every competitor consistently.[1]
High rubric agreement with 33 reference works supplied by task authors is also useful but limited. The same expert community defines what the task requires and supplies the example solution. This supports the rubric’s ability to work sensibly within its own context. It does not directly measure recognition of a different but valid solution from an outside team. That gap matters in scientific research because there is no single correct way to present every solution. Agreement with a reference does not cover the entire ability to judge an unexpected approach fairly.[1]
Reviewer auditing beside the score
The table that earns more confidence from me therefore shows how expert inspection changes reviewer decisions alongside the pass rate. An evaluation teaches more when outside experts inspect successful and unsuccessful decisions, distinguish reasons for disagreement and identify which results were affected by task revisions. This is not a forecast of the next result. It specifies the evidence that increases my confidence. A higher score alone does not reduce uncertainty in the instrument used to produce it.[1]
FrontierPhysics is valuable for exposing limits in research agents and in their reviewer together. The preprint is a serious evaluation of specified physics tasks, rather than validation of the entire scientific-discovery process. An expert audit changing the score does not make the study worthless. It reveals which parts of the table are correctable. My reading criterion is this: when explaining a failure rate, how close does the evaluation come to distinguishing solutions that were wrong from solutions that were judged wrongly?[1]