Two standards for oversight

Correcting some of an AI system’s mistakes does not establish that the combined decision is better than a clinician’s unaided decision. That distinction is the most useful contribution of Nadine Schlicker and colleagues’ heart-sound experiment. If oversight succeeds whenever people outperform the machine, recovering part of the loss from a malfunctioning system can count as success. Requiring the team to preserve the human baseline sets a different standard. In evaluating clinical AI oversight, I would want that second threshold included as well.[1]

The preregistered experiment randomly assigned 48 participants to two groups. Each classified 40 heart-sound recordings. One group saw AI advice first; the other classified the sound independently before seeing advice and being allowed to revise its decision. Researchers reduced recommendation accuracy across four blocks. Participants offset some errors at low reliability, but combined performance fell below the unaided performance measured in the independent-first group. Requiring an initial judgment did not provide the expected protection in this setting.[1]

What matters to me is measuring human participation and effective oversight separately. An independent first judgment cannot protect the outcome if the clinician abandons a correct assessment after seeing advice. By retaining both initial and final classifications, the experiment makes that transition visible. For people designing clinical workflows, the outcome of interest needs to extend beyond a confirmation click to include correct decisions displaced by erroneous recommendations.[1]

The limits of deciding first

The researchers observed that confidence in participants’ own classifications declined earlier than trust in the system. That offers a clue about the authority attributed to advice. It does not isolate the mechanism. The high accuracy described at the outset may have shaped trust, while the fixed sequence of declining reliability leaves time and order effects in play. I would therefore be cautious about assigning the entire result to automation bias.[1]

In the final block, advice accuracy fell to zero and the audio and visual display were deliberately mismatched. Changing both together makes their separate contributions difficult to identify. The sample also included final-year medical students, and the task lacked patient interviews, additional clinical information and real treatment decisions. The findings show fragile oversight in this classification experiment. They do not estimate harm rates in patients or the effect of independent first judgments across all clinical fields.[1]

In a follow-up test, I would vary the reliability sequence and the audio-visual mismatch in separate conditions. A comparator preserving unaided assessment would also allow investigators to track correct decisions lost after advice. Both oversight thresholds should then be measured again with additional clinical information and realistic workload. That design would clarify the conditions in which requiring an initial judgment helps. It could also help a hospital define measurable oversight responsibilities beyond the person named on the final screen.[1]