Microsoft’s AI diagnosis tool beats doctors on past cases in Science study

Microsoft says its multi-agent AI system diagnosed medical cases far more accurately than doctors in tests published in Science, though it has not yet been tried with real patients.

1 min read

The Verge

At a glance

AI accuracy on NEJM case reports
~83%
doctors with references, same cases
~36%
AI accuracy on emergency cases
80%
What’s new
Microsoft’s Diagnostic Orchestrator, which uses several AI agents working like a panel of experts, outperformed physicians on cases from a medical journal and hospital emergency records, according to a peer-reviewed paper in Science.
Who it affects
Doctors, hospitals and patients, along with health regulators who would oversee any clinical use.

The system asks questions, weighs possible causes and chooses which tests to order before giving a diagnosis, much as a doctor would. Cases came from The New England Journal of Medicine and from emergency admissions at Boston’s Beth Israel Deaconess Medical Center, where human doctors averaged 47 percent. It also tended to reach answers at a lower estimated cost. The tool runs on OpenAI’s GPT-5, and Microsoft plans to open-source its SDBench benchmark. Microsoft’s AI health lead, Dr. Dominic King, says hospital testing with Beth Israel Deaconess will likely begin in the next couple of months.

Why it matters

Beating doctors on written case files is not the same as treating patients, and Microsoft AI chief Mustafa Suleyman says the tool still needs clinical validation. The paper also leaves out the newest models, so results may shift. The real test is the hospital trial: if accuracy holds in live settings, regulators will face pressure to define how AI diagnosis can be used and who is accountable.

Microsoft says its AI tool is better than doctors at diagnosing conditions

The Verge

theverge.com

Summary by Second Order AI. Full reporting is at the original source.