Fixed cases, changing instructions
A model’s willingness to act on the same clinical text changes when it is assigned a different physician persona. The study published in npj Health Systems on November 14 reports a 42.8% affirmative rate without a persona across 20 open-weight models. The lowest and highest persona rates are 36.9% and 46.4%. That 9.5-point span appears without adding a clinical finding to the case. The peer-reviewed publication, following a January preprint, makes the system instruction a measurable part of the decision configuration.[1]
The design’s strength is its fixed clinical information. Three ethical orientations—duty, care and utility—are crossed with intuitive, integrative and analytic styles. A changed answer therefore cannot readily be attributed to new information about the patient. This does not establish that a model develops ethical commitments resembling a physician’s. The authors use these labels for regularities in outputs. Compliance with the wording of short personas and a forced binary response format remains another plausible mechanism for the observed shifts.[1]
The repetitions inside the decision count
Two corpora test the reach of the result: 2500 simulated emergency cases and 2500 real discharge summaries. Models answer questions about safety, autonomy, treatment, resource use and follow-up on the same texts. There are no tools, retrieval, memory or self-correction. The investigators report 85.7% directional agreement and an effect-size correlation of 0.82 between the corpora. That supports similar sensitivity across two kinds of clinical text, with both comparisons still conducted by the same team inside the same experimental setup.[1]
The measurement denominator needs clarification. The methods multiply 5000 texts by 5 questions, 10 repeats and 20 models to report 5 million decisions. They also describe nine personas and a tenth baseline condition, without showing how that condition factor enters the total. This ambiguity does not by itself overturn the reported difference, but it complicates appeals to the total output count as evidence of precision. Repeatedly querying a model about one case also adds no new patient; it samples output variability for that case.[1]
The comparator needed for correctness
The more fundamental limit concerns what yes means. The baseline is not a comparison set labelled with correct answers. Guideline concordance and outcome correctness were not adjudicated. Moving away from 42.8% therefore does not establish a higher or lower error rate. More affirmative answers on autonomy questions can carry a different clinical meaning from more affirmative answers on safety questions. One model’s exclusion from susceptibility analysis after producing 100% affirmative answers under every condition further illustrates why low variability alone cannot certify good decisions.[1]
A concrete evaluation design follows: track persona changes on matched cases while separately identifying correct and incorrect interventions. Prespecified expert answers, case-level paired comparisons and explicit condition counts would help test whether a threshold shift brings benefit. The next stage should include the actual workflow: tools and clinician oversight were absent here and might attenuate or alter the effect. Retaining the system-instruction version and reassessing the same cases after a change would make decision behaviour visible even when the model name stays constant.[1]