The right label and the right decision
What stops me in Magnus Lynch’s small image experiment is how one comparison answers two different questions. Fine-tuned Inception v3 gives the correct first diagnosis for 125 of 200 lesions; GPT-6 Astra and Claude Fable 5.1 give 84 and 85. That is a classification result on a particular collection of images. Lynch also asks whether the patient could be discharged without further intervention. Naming a lesion correctly and avoiding discharge of a dangerous lesion are different objectives. Using the first ranking to answer the second question changes the decision being measured. The distinction matters before making any comparison about safety.[1]
For the discharge question, the network’s preset threshold misses 18 of 30 melanomas, against 3 and 4 for the language models. This makes behavior visible that a reader counting only correct diagnoses would overlook. It does not establish that the language models are clinically safer. The network’s operating point and the way it is matched to the language models’ decisions also enter the comparison. I read this table as a reason to specify the evaluation objective explicitly. It is insufficient as a definitive ranking of model identities because changing the question changes what counts as success.[1]
The choice carried by a threshold
When Lynch matches thresholds, the network’s missed-melanoma count falls to two or one. Different operating points of the same model produce results that a single accuracy number cannot contain. Catching more dangerous lesions and avoiding unnecessary investigation of benign ones are objectives that need to be considered together. At the initial threshold the network discharges 76 of 100 safe cases, against 32 and 22 for the language models. Those figures describe the costs of different decisions. We cannot calculate the safe-discharge count at matched thresholds from these initial numbers, and should leave that missing comparison unfilled.[1]
The network’s initial melanoma misses might reflect its image representation; the chosen decision threshold could also explain part of the same pattern. The change after threshold adjustment gives a concrete reason to examine that second explanation. We can learn this without making a sweeping judgment about architecture. The comparator is also an older baseline network. Its advantage over language models on a particular metric does not establish parity with current specialist systems. Naming the comparator does not diminish the finding. It fixes the question the experiment actually answers and helps separate a model’s representation from the policy applied to its output.[1]
The denominator of the next comparison
The sample was assembled from 100 unsafe and 100 safe lesions; the benign cases had also undergone biopsy. That balance cannot be read as disease prevalence in an ordinary clinic. There are only 30 melanomas and each language model is run once on each case. I therefore do not infer a hospital-level harm rate or repeatability from this experiment. Prior exposure to the public data also remains unresolved for the language models. These limitations do not erase the results. They constrain the patient populations and uses to which those results can reasonably be carried.[1]
The next test I would ask for fixes the decision objective and threshold before examining test images. In a separate patient group, reporting both missed dangerous lesions and unnecessary investigation of benign cases at the same sensitivity would make the choice clearer. This is my methodological proposal, not work Lynch has completed. Waiting, additional investigation and missed disease do not impose the same burden on patients. For that reason, I want the report to show which decision is obtained at which cost, rather than end with one winner. That is the useful question this small experiment leaves open.[1]