Which question came before the diagnosis?

The advantage of a completed case

A diagnostic question can contain something as valuable as the correct answer: an earlier decision about which information needed collecting. The November 10 Hx2Dx ARENA preprint removes that advantage. Instead of a completed case narrative, a model begins with the patient’s age, sex and main complaint. It must decide which questions to ask, which investigations to request and when to stop. All seven models perform less accurately under those conditions. The interesting contribution is less the identification of a diagnostic champion than an attempt to measure how much work the assembled evidence has already done.^[1]

The comparison uses 1000 educational cases from the New England Journal of Medicine. Diagnosis and discussion are removed in the static condition. In the interactive condition, a patient role answers only what is asked, while an investigation role returns findings from the original record. GPT Luna 6 scores 83.9% with complete cases and 47.4% in conversation, a gap of 36.5 percentage points. The loss appears when access to information changes within the same case collection. That matching is useful: it reduces the problem of presenting different patients or hospitals as one comparison. The measurement still does not come from a clinical trial involving real patients.^[1]

The experiment that changes the interviewer

The researchers then give other models the transcripts collected by the strongest interviewer, GLM 5.3 Flash. Its final diagnosis is removed; questions, answers and obtained investigations remain. Another model can therefore diagnose from the same encounter without gathering the evidence itself. GPT Luna 6 rises to 56.3%, and the spread between models narrows from 32.5 to 11.4 percentage points. This intervention is more informative about evidence acquisition than the initial ranking alone. When the same system improves with better-selected findings, it becomes harder to attribute the entire failure to the reasoning that produces the final diagnosis.^[1]

Transcript transfer is not a perfect isolation of clinical questioning skill. A more orderly or compatible transcript could itself improve results, and using the same model family for patient and examiner roles makes the measurement environment consequential. Performance also remains below the static condition after transfer. Even the strongest interviewer may not have recovered every discriminating finding in the original narrative. My conclusion is therefore that diagnosis synthesis and evidence acquisition deserve separate evaluation. The findings support that distinction without establishing that interviewing is the sole bottleneck in clinical practice. They do not yet measure its effect on real patients.^[1]

The partial diagnosis inside a success

Another distinction sits inside the word success. The primary measure combines correct and partially correct diagnoses. GLM 5.3 Flash scores 54.4% interactively under that rule, but 36.6% when only exact diagnoses count. The authors have an understandable rationale: the reference answer in a teaching case may include a subtype requiring biopsy or molecular testing beyond identifying the disease category. Finding the right category can guide subsequent investigation. Yet these percentages answer different questions. Whether either is sufficient for a clinical decision depends on the decision and the consequences of error; a leaderboard cannot settle that requirement by itself.^[1]

What I would want from the next evaluation is evidence that this distinction holds beyond selected rare teaching cases. The collection represents verbally communicating patients, uses English text and stops encounters after 50 turns. Possible training exposure to decades of published cases also remains unresolved. These limitations define where the finding can travel without erasing the method’s value. In a new patient population, separately recording question selection, recovered discriminating findings and exact diagnosis would be a useful next step. The completeness of the file handed to a clinician could also be measured. That would make it harder to count a system as successful merely because its final answer looks plausible despite omitted questions.^[1]