83%: The Diagnostic Accuracy Gap in New Microsoft AI Study

In a simulation of difficult medical scenarios, a new artificial intelligence system achieved an 83 percent correct diagnosis rate. This figure stands in sharp contrast to the 36 percent accuracy recorded by generalist physicians working individually with reference materials. The findings were published in the journal Science on Thursday, marking a significant data point in the ongoing debate over AI capabilities in healthcare.

The system, identified as MAI-DxO, functions by coordinating multiple AI agents to act like a panel of specialists. These agents question patients, order tests, and debate diagnoses. The study utilized complex cases from the New England Journal of Medicine. While these cases are typically handled by specialists or teams who were not included in the comparison, the gap in performance between the AI and generalists was substantial. Microsoft measured both accuracy and the estimated cost of running the AI for each case.

Dominic King, vice president of Health at Microsoft AI, described the results as substantial and exciting. He noted that the primary challenge in healthcare delivery is the growing mismatch between the demand for and supply of medical expertise. Mustafa Suleyman, CEO of Microsoft AI, called the result a step toward what he terms medical superintelligence. Suleyman leads Microsoft's effort to build its own frontier AI and previously co-founded DeepMind.

The underlying data suggests that much of the improvement stems from more powerful AI models, with the orchestration system adding further gains. Microsoft tested the system with models from several companies, including OpenAI, Google, Anthropic, Meta, and DeepSeek. The best results were achieved using OpenAI's GPT-5 and GPT-5.5. With GPT-5.5, a model released in April of this year, the orchestration system raised accuracy from 72 percent to nearly 87 percent on complex cases at a slightly lower cost.

However, the gains were less pronounced in emergency department cases, where accuracy hovered around 85 percent regardless of the orchestration layer. The system helped most with less capable models. For instance, Gemini 2.5 Flash saw its score rise from 46.4 percent to nearly 69 percent with the assistance of the new system. Microsoft researchers noted that accuracy gains naturally shrink as models approach theoretical limits, but the orchestration helps optimize both accuracy and cost.

Microsoft is working with the Mayo Clinic to train a foundation model for healthcare, with a release planned for next year. Suleyman believes this will enable enormous leaps in the field. He also highlighted a new partnership with Beth Israel Deaconess Medical Center in Boston to evaluate the system in real-world settings. Patients will receive standard treatment while consenting to parallel engagement with the AI. Microsoft and its partners will then compare the AI recommendations against those of human doctors.

This study coincides with a broader manifesto from Microsoft on building medical superintelligence. The company aims to move beyond benchmarking individual tasks to integrating with clinicians for better outcomes. Suleyman stated that the company is committed to accelerating the introduction of this technology, viewing it as a potential differentiator among frontier labs. Microsoft already sells dictation and radiology tools to clinicians and sees this new system as an expansion of those efforts.