Start with the case Lynch opens with. An invasive melanoma on an ear, less than 0.8 mm thick. Both frontier models called it a seborrhoeic keratosis, a harmless warty growth, and said it could be discharged. The old network called it a melanoma. One case proves nothing on its own. The pattern across 200 cases is the story, and it has two halves that point in opposite directions.
This was a task-specific comparison: a fine-tuned dermatology classifier against two general-purpose models run once per case using a single prompt.
The naming test
Lynch drew 200 lesions at random from MILK10k, a public dataset released through the International Skin Imaging Collaboration in 2025. Every lesion was biopsied, which makes the set harder than a clinic's usual stream: a mole somebody decided to cut out is a mole that worried a clinician. Half the set should never be sent home, including 30 melanomas. The other half is benign.
The two LLMs — the models Lynch's report names as GPT-6 Astra and Claude Fable 5.1 — got both photographs of each lesion and a clinician-style prompt: name the most likely of six diagnoses, give a confidence and a second choice, and say whether the lesion could be safely discharged. Neither refused a case. The CNN, Inception v3, was fine-tuned on 68,011 dermatology images from the ISIC Archive, with the test lesions and every other photograph of the same lesion or patient removed from its data, including its validation set, before training. It had never seen the cases it was graded on, and they were not used to choose the model or set its threshold.
It won clearly. First choice: CNN 125 of 200, Claude Fable 5.1 85, GPT-6 Astra 84. Allowing a second choice widened the same gap: 173 against 138 and 134. The lead held on the harmless lesions: the CNN recognised 37 of 50 benign moles and 26 of 50 seborrhoeic keratoses, against 16 and 13 for GPT-6 Astra and 33 and 6 for Claude Fable 5.1. On melanoma it led too, 13 of 30 against 11 and 8, which is another way of saying that every system missed most melanomas as a first choice, and the winner missed 17. In a second case Lynch highlights, a low-pigment invasive melanoma, GPT-6 Astra called it a basal cell carcinoma at 95% confidence; Claude Fable 5.1 and the CNN called it a melanoma.
The errors had habits. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, and 14 of the 30 melanomas went into that bucket. Claude Fable 5.1 called 53 lesions actinic keratosis when 20 were. Of the 17 melanomas the CNN got wrong, it called eight a benign mole and six a seborrhoeic keratosis.
In a separate re-scoring of Lynch's released replies, Tensorust reported that Astra assigned at least 80% stated confidence to 56 of its 116 incorrect first-choice diagnoses, compared with two of Fable's 115. These are model-reported confidence values, not independently calibrated probabilities.
The discharge table
Diagnosis was only half the prompt. The systems were also asked whether each lesion could be sent home with no treatment, no biopsy and no follow-up. At its preset threshold, the CNN kept 77 of the 100 unsafe lesions and would have discharged 18 of the 30 melanomas; it discharged 76 of the 100 safe lesions. The LLMs were far more cautious: GPT-6 Astra kept 97 of the 100 unsafe lesions and Claude Fable 5.1 kept 96, and they would have discharged three and four melanomas. But the same table carries the matched row: with its threshold adjusted to the LLMs' sensitivity, the CNN kept 98 or 99 of the 100 unsafe lesions and would have discharged one or two melanomas. Matching sensitivity is not free, though — a threshold set that cautious discharges fewer of the safe lesions too. The 76 of 100 safe discharges belongs to the preset setting, not to the matched one.
A threshold is not a technical footnote. It is the version of the model a user meets.
The danger comes when the result travels as a single winner-and-loser headline. Lynch's own table shows why that is incomplete: classification accuracy and discharge decisions are different measurements, and the CNN's apparent triage performance changes sharply with its threshold.
In a separate patient-style test using clinical photographs alone, Lynch reports that both models recommended seeing a doctor for almost every case, including benign lesions. In a further test Lynch ran on 100 more biopsy-proven melanomas, the two models advised seeing a doctor in 99 and 98 of the cases. That is the safest behaviour in the whole test, and also the least informative: a system that sends everyone to a clinic has not triaged anyone.
What the test does not show
Lynch lists his own limits, and they are real. Two hundred lesions show a pattern; they do not settle close rankings, and 30 melanomas leave wide uncertainty around the number that matters most. Each LLM case ran once, and answers vary between runs. The set is all biopsied lesions, a harder stream than ordinary practice: on ordinary archive images, the CNN's separation of lesions that should and should not be discharged was far better, an area under the ROC curve of 0.96 against 0.86 on this test set. MILK10k has been public since 2025, so the LLMs may have seen it in training, which would have helped them, not hurt them; the CNN had not seen the test lesions.
Two more limits deserve plain statement. Inception v3 is a baseline from 2015, chosen deliberately; Lynch notes that state-of-the-art skin systems, and the regulated products built on them, perform considerably better. And the CNN had a home advantage of a specific kind: it was fine-tuned on tens of thousands of images drawn from the same archive tradition as the test set, for exactly this six-way job. A specialist beating a generalist at the specialist's own task is evidence about fit, not about which kind of model is better in general.
Lynch also discloses his own position and tooling: he designed the test and made the clinical rulings, the analysis and first drafts were produced with Claude Code, and one of the two models being graded is Anthropic's. Scoring was done by script against the histology. The cases, prompts, every LLM reply and the scoring code are in one public download, and the CNN itself runs in a phone browser, with nothing uploaded. Read the study for what it is: a single-investigator technical demonstration, not a blinded, independently replicated clinical validation.
None of the three systems was clinically validated by this comparison for autonomous diagnosis or discharge decisions. The results describe a retrospective image test, not evidence that any model should decide whether a patient can safely leave without assessment.
The practical reading is narrow. In clinician mode, the LLMs named the right diagnosis in fewer than half the cases; in patient mode, they send nearly everyone to a doctor anyway. And when a small specialist model beats a frontier one, ask which score the headline used. Naming a lesion and deciding who goes home are different jobs. This test measured both.
Sources
- [1] Lynch Lab (Dr Magnus Lynch), "A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis", October 25, 2026Read source
- [2] Lynch Lab test data: cases, prompts, every LLM reply and the scoring codeRead source
- [3] Tensorust, "AI skin cancer detection: a 2015 CNN beat GPT-6 and Fable 5", October 25, 2026Read source
- [4] MILK10k dataset, MILK study team, International Skin Imaging Collaboration (2025), as linked from the primary source.