Two denominators behind one headline

The version travelling through the morning digests says Microsoft's AI diagnoses “up to four times more accurately than human doctors”. That formulation has a source. In Microsoft's own announcement the comparison reads: “On complex NEJM cases, MAI-DxO achieved 83% diagnostic accuracy. Physicians achieved 22% without external reference resources and 36% with them, making MAI-DxO about 3.7 and 2.3 times more accurate, respectively.” Those are Microsoft's reported ratios — and they join two differently sized evaluation sets.

The 83.2% in the Science paper is MAI-DxO's score across all 304 NEJM clinicopathological conference cases in the study: SDBench's own documentation states that published results are reported on the combined validation and test sets. The physician study was smaller by design. It ran on 56 cases held out from engineering efforts during development, and no participant worked the whole set — physicians completed an average of 36.4 cases closed-book and 34.8 open-book, producing 764 and 662 diagnoses respectively. Their averages were 22.3% without references and 36.1% with search and reference tools; the best individual physician reached 51.8%.

What neither the paper's main text nor Microsoft's announcement reports is MAI-DxO's accuracy on those same 56 cases. The matched head-to-head therefore cannot be computed from the published figures, and the 3.7x and 2.3x figures should be read as what they are: Microsoft's reported ratios over unlike populations. The paper does publish one explicitly same-case comparison, and it is behavioural rather than diagnostic: on the same cases, MAI-DxO asked a mean of 25.5 questions before ordering its first test, against 5.6 for the physicians.

The second dataset narrows the margin without closing it. On 325 emergency-department cases drawn from MIMIC-IV-ED, MAI-DxO scored 80.3% against the physicians' 46.8% — “more than 1.7 times higher”, in Microsoft's reported ratio. The paper's main text does not state how many of the 325 cases the physician average covers; the best individual emergency physician is reported at 62.5% over 40 cases. And the paper draws its own boundary around all of it: the comparisons “do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice”, and the authors expect the gap to narrow against specialists.

A different study with the same name

An older set of numbers needs keeping separate. In the June 2025 preprint of this work, the system was paired with o3 and reached 85.5% in its maximum-accuracy configuration, while 21 physicians averaged 20% in a setup that barred search, textbooks, colleagues and AI. The Science paper is a later publication from the same research programme with a changed evaluation — GPT-5 in the main results, a GPT-5.5 re-run before publication, an added open-book physician arm, and the emergency-department dataset added. It is not the preprint with a journal logo added, and its figures should not be merged with the preprint's.

What the orchestration adds, and what the dollars are

MAI-DxO is not a model. It is a coordinator that makes a model behave like a panel: agents propose diagnoses, select the next question or test, and challenge the leading hypothesis before the system commits. Across every model tested, that orchestration added an average of 7.9 percentage points on the NEJM cases and 4.4 points on the emergency cases over the same models given a basic prompt — an average across models, not any single model's improvement. For the strongest pairing, TechCrunch's read of the study data puts GPT-5 on its own, with a short basic prompt, at 78.9% on the complex cases; MAI-DxO with GPT-5 reached 83.2%. The panel is worth roughly four points there — and a separate, larger difference in estimated testing costs, set out below.

Every dollar figure in the study is a mean estimated testing cost, assigned by SDBench from CPT-based pricing in the Kaiser Permanente Washington chargemaster. None of them is a bill, a payment, a reimbursement or a saving. On that basis, GPT-5 alone ran at about $8,400 per complex case in TechCrunch's read of the study data, against MAI-DxO's $5,431 — about $3,000 lower in estimated costs within the benchmark. The unconstrained system was still not the cheapest participant: open-book physicians averaged $4,725 in estimated testing costs, and closed-book physicians $3,049. MAI-DxO's cheaper configurations are tuned ones, trading accuracy for a smaller workup: 80.9% at $2,241 and 79.6% at $1,303, against 84.5% at $8,945 for the configuration aimed at diagnostic excellence. On the emergency cases, the system's $4,979 compares with the physicians' $3,400.

The workup styles differ as much as the scores. In SDBench, questions are nearly free — they fold into a fixed physician-visit cost — while tests are priced item by item, which rewards exactly the strategy MAI-DxO follows: ask 25.5 questions first, then test selectively. In a real clinic the scarce currency is time, and the physicians spent 11.9 minutes per case closed-book and 18.1 open-book. A benchmark that prices tests but not minutes will favour the system that can ask questions at machine speed.

Who built the test, and who grades it

The benchmark uses a physician-authored scoring rubric, but diagnosis adjudication is performed by a Judge agent within the evaluation framework developed by the research team. The Gatekeeper also supplies some synthetic findings. Publishing the code improves inspectability; it does not by itself establish independent validation. SDBench's code — benchmark, Gatekeeper, Judge and cost estimator — is released under the MIT License, so other groups can run the same evaluation; the NEJM-derived dataset itself is available from NEJM Group for noncommercial research under licence.

The disclosures sit together in the paper. All of the authors are Microsoft employees and hold equity in the company. Eight of them are inventors on US patent application 19/722,651, submitted by Microsoft, relating to an automated evaluation framework for sequential diagnosis. And the paper states that no funds external to Microsoft were acquired for these studies. Peer review examined the work; it does not change who built the test the system happens to lead.

The limits are stated just as plainly. The cases are retrospective and written. The Gatekeeper supplies findings from the record — and, where a requested finding is absent, generates diagnostically neutral synthetic results. No patient was examined, and no imaging, waveform or pathology slide was interpreted directly. The rubric judges whether the diagnosis was right, not whether the treatment that would follow is safe or appropriate, and nothing in the design measures longitudinal management or outcomes.

The test that has not happened yet

The follow-up is the right shape. Through a partnership with Beth Israel Deaconess Medical Center in Boston, Microsoft will evaluate DxO's clinical reasoning across hospital settings. As King explained the design to TechCrunch, patients receive standard treatment — triage, time with a specialist, treatments, investigations and a discharge decision — while consenting patients in parallel also engage with DxO, and Microsoft and its partners compare the system's recommendations against the human doctors'. Whether that study can settle the clinical question will depend on its protocol, its outcome measures and who runs the evaluation being public in advance.

Until it reports, the honest summary is narrow. A peer-reviewed benchmark shows a well-built orchestration system outscoring individual generalist physicians on hard written cases, by margins whose exact size depends on denominators the headline version leaves out. That is a real result about a test. It is not yet evidence that any patient's diagnosis arrives sooner, safer or cheaper with the system in the building.

Sources

  1. [1] Nori et al., “Sequential diagnosis with language models”, Science, 22 October 2026, DOI 10.1126/science.aeh7930 — primary source, read in full first-hand for this article (abstract, Results, Discussion, competing interests, data and code availability): the 83.2% / 80.3% / 36.1% / 22.3% / 46.8% accuracy figures; the 304 NEJM and 325 MIMIC-IV-ED case counts; the 56-case physician study (764 closed-book and 662 open-book diagnoses; best physician 51.8%); all cost values and their CPT/chargemaster basis; the 25.5 vs 5.6 questions statistic, stated on the same cases; the +7.9 / +4.4 average orchestration boost; the Judge rubric and Gatekeeper synthetic findings; the stated limits, including that the comparisons do not establish superiority over specialists, multidisciplinary teams or real-world clinical practice; and the disclosures — all authors Microsoft employees holding equity, eight inventors on US patent application 19/722,651 relating to an automated evaluation framework for sequential diagnosis, and no funds external to Microsoft acquired.Read source
  2. [2] Microsoft AI, “Building Medical Superintelligence”, Microsoft AI news, 22 October 2026 — the company's own announcement, read in full first-hand: “On complex NEJM cases, MAI-DxO achieved 83% diagnostic accuracy. Physicians achieved 22% without external reference resources and 36% with them, making MAI-DxO about 3.7 and 2.3 times more accurate, respectively”; 80% vs 47% and “more than 1.7 times higher” on emergency cases; the 81%-at-less-than-half-cost configuration; and the open-sourcing of SDBench for select NEJM cases. The post does not mention the 56-case physician subset.Read source
  3. [3] SDBench project site and public repository, Microsoft (read first-hand): the benchmark's documentation states that published results are reported on the combined validation and test sets, and its reproduction instructions for the Science paper baselines use the validationandtest case set; the source code (Gatekeeper, Judge, Cost Estimator, baseline orchestration) is under the MIT License. Neither the site nor the repository publishes a test-set-only MAI-DxO accuracy figure.Read source
  4. [4] MIMIC-IV-Ext-SDBench dataset record, PhysioNet (read first-hand): the released emergency-department dataset splits 488 case records into train 163, validation 254 and test 71; the published SDBench evaluations use validation + test = 325 cases, on which it reports the GPT-5 baseline agent at 76.6% and the DxO agent without budget constraints at 80.3%, with mean estimated workup costs of about US$4,240 and US$4,980.Read source
  5. [5] Rebecca Bellan, “Microsoft's Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases ‘medical superintelligence’”, TechCrunch, 22 October 2026 — read in full first-hand: GPT-5 on its own with a short, basic prompt scored 78.9% on complex cases at about $8,400 per case, and DxO cut that estimated cost by about $3,000 while raising accuracy by about four points (TechCrunch's read of the study data); the Beth Israel Deaconess partnership and study design as explained by King — standard treatment in parallel, consenting patients also engaging with DxO, recommendations compared against the human doctors'; and the Mayo Clinic healthcare foundation-model plan.Read source
  6. [6] Nori et al., “Sequential Diagnosis with Language Models”, arXiv preprint, June 2025 — used only for the clearly labelled preprint paragraph: the earlier o3-era evaluation and its closed-book-only physician setup. Its figures are not merged with the Science paper's anywhere in this article.Read source