Artificial intelligence is increasingly being tested as a partner in medical diagnosis. A model can read a patient history, consider laboratory results and suggest conditions a busy clinician might otherwise overlook. Newer reasoning-focused models promise to do this more carefully by working through the evidence before producing an answer.
But a new peer-reviewed study has identified a critical complication. When researchers compared reasoning-enabled models with their conventional counterparts using real hospital records, the reasoning versions generally identified the primary diagnosis more accurately but also produced more than 34% additional false-positive diagnoses.
Repeated evaluations of the same cases also produced less consistent answers. A follow-up experiment involving 15 physicians found that a structured approach combining the model’s broad suggestions with clinical verification improved primary-diagnosis accuracy by around 20% and reduced false positives by 44%.
Published on 29 October 2026 in Nature Communications, the research offers a more complex answer than either enthusiasm or alarm about AI in medicine. More elaborate reasoning can help an algorithm find the right disease, yet the extra diagnoses it proposes may also create new risks if no qualified person checks them.
The difference between finding a diagnosis and diagnosing safely
In a hospital, a patient rarely arrives with every relevant fact neatly summarized. Clinicians must interpret symptoms, medical history, vital signs, laboratory results and imaging reports, sometimes while several conditions could plausibly explain the same findings.
A useful diagnostic assistant might identify the most likely condition and raise other serious possibilities that warrant investigation. However, every additional suggestion has consequences. An incorrect diagnosis may lead to more tests, inappropriate medication, anxiety or distraction from the actual problem.
This creates two separate questions. How often does the system identify the principal disease correctly? And how many incorrect additional diseases does it introduce while doing so? A model that improves the first measure can still perform worse on the second.
Shuai Wu, Linwei Chen and an international research team examined precisely that tension. Their investigation compared AI systems operating with and without additional reasoning capabilities, rather than assuming that longer explanations necessarily meant better clinical judgment.
The models were assessed on retrospective patient records and a controlled physician experiment. They were not used to make autonomous treatment decisions for patients in a live hospital, a distinction that is essential when interpreting the findings.
Nearly 3,000 real hospital cases formed the benchmark
The researchers built a benchmark called AIMS-LDR, drawing on de-identified electronic health records from two established medical research datasets. The material represented 2,978 hospital and intensive-care cases across the combined evaluation.
One component, LDR-MIMIC, contained 1,157 admissions from the MIMIC-IV dataset. The other, LDR-eICU, included 1,821 intensive-care cases from a multicentre United States database.
Records included clinical details such as histories, investigations, observations and relevant notes. The team removed diagnosis-revealing information, including discharge diagnoses, to reduce the possibility that an algorithm would simply read the answer rather than infer it.
The evaluation covered 12 medical specialties and allowed the models to consider a range of possible conditions. This made the benchmark more involved than a short multiple-choice medical examination based on a neatly written question.
It was still a controlled research setting. The data reflected past clinical encounters, and the algorithms were not responsible for examining patients, obtaining consent, ordering tests or acting on the diagnoses they proposed.
Why the researchers compared matched AI models
The study evaluated six matched model pairs, comparing reasoning-enabled operation with a corresponding standard or non-reasoning mode. Systems came from several major model families, including OpenAI, Google, Anthropic, DeepSeek and Qwen.
Matching the configurations helped the team examine what happened when a model was allowed to perform more explicit reasoning, rather than attributing every difference to an entirely unrelated model architecture or training dataset.
Each system was asked to assess clinical information, prioritize its likely diagnosis and consider relevant alternatives. The researchers examined primary-diagnosis accuracy, the usefulness of differential diagnoses, false positives and whether repeated runs produced consistent conclusions.
They also varied prompting approaches, model settings and evaluation methods to determine whether the pattern depended on a particular test arrangement. These robustness checks matter because a result arising from one unusual prompt may not indicate a general tendency.
Even so, the tested configurations represent particular model versions available for this investigation. Their observed performance should not be presented as a permanent property of every newer or future AI system.
Reasoning improved the primary answer
On the MIMIC component, reasoning-enabled systems achieved an average 52.8% primary-diagnosis accuracy, compared with 38.7% for their non-reasoning counterparts.
On the independent eICU component, the corresponding averages were 35.1% and 28.9%. Although both figures were lower in that dataset, the direction of the primary-diagnosis difference remained favorable to reasoning-enabled systems.
These percentages describe benchmark results, not how often a clinician would necessarily make the correct diagnosis when supported by such a tool in ordinary practice. Electronic records, coding practices and the format of the task can all influence measured accuracy.
Reasoning models also performed better on some differential-diagnosis measures. Generating a clinically relevant list of alternative conditions can be valuable when a case is complex and a clinician needs to consider explanations beyond the most obvious possibility.
However, being more capable of producing plausible alternatives does not guarantee that every alternative is correct, proportionate or useful enough to justify additional investigation.
The price of broader diagnostic suggestions
The same systems that improved primary-diagnosis accuracy generated more incorrect additional conditions. In the reported MIMIC comparison, reasoning models averaged 1.47 false-positive diagnoses per case, compared with 1.09 for non-reasoning versions.
That represents an increase of approximately 34.8% in the number of false-positive diagnoses. In the eICU comparison, the figures were 5.47 versus 5.01, again favoring the conventional models on this measure.
A false-positive diagnosis is not necessarily equivalent to an actual clinical error. In this benchmark, it meant that the model named a condition not supported by the reference information for that case. In a real hospital, some suggestions might prompt useful verification, while others might generate unnecessary testing or treatment.
The researchers also considered whether models identified the complete set of important conditions correctly, rather than merely naming the leading one. On one benchmark comparison, the reasoning-enabled systems performed worse by that stricter criterion, at 9.1% versus 10.9%.
The important insight is that improvements in a headline accuracy score can conceal a deterioration in the quality of the overall diagnostic list. A system intended to support medical decisions should be judged on more than its top answer.
Repeated questions did not always produce the same answer
Another concern involved reproducibility. A reliable diagnostic support system should not change its judgment dramatically when the same case is evaluated again under comparable conditions.
The researchers repeated evaluations and found that reasoning-enabled systems were generally less consistent between runs. In a reported three-run comparison, the same primary diagnosis appeared in 52.3% of reasoning-model cases, compared with 69.0% among non-reasoning systems.
The paper summarizes this wider reproducibility disadvantage as approximately 40% lower reproducibility. That headline concerns the study’s broader evaluation; it should not be calculated simply by subtracting the two percentages from the example comparison.
Unstable outputs can make a clinical recommendation difficult to audit. If repeated analyses of an unchanged record produce different diagnoses, a medical team needs to know which findings can be relied on and whether further confirmation is necessary.
That does not mean human medical judgment is perfectly consistent. The research instead highlights reproducibility as a relevant safety measure that should accompany diagnostic accuracy when AI systems are evaluated.
Why extra false positives may matter in practice
Some incorrect suggestions carry little immediate consequence; others could lead to invasive investigations, medications or a delay in addressing the most important problem.
The researchers examined the potential implications of incorrect diagnoses using retrospective clinical information. This offered a way to consider possible downstream resource use rather than simply counting whether a model’s answer matched a label.
But simulated implications are not the same as actual harm. No patient in this benchmark was shown to receive an unnecessary procedure because of the model’s output, and the study did not track real-world health outcomes after AI-directed treatment.
The safety issue is therefore one of potential risk exposed by the benchmark. A model offering extra unsupported diagnoses may create additional work for a clinical team and could influence decisions if those suggestions are accepted without checking the evidence.
Developers evaluating diagnostic systems should therefore report false-positive burden, consistency and appropriate referral behavior alongside sensitivity or top-answer accuracy.
What happened when physicians checked the AI
To test a possible safer workflow, the researchers included a 15-physician validation experiment. The clinicians assessed a controlled selection of 150 clinical cases while working with model-generated information.
The structured collaboration gave the AI a role in broadening the diagnostic possibilities while retaining the physician’s responsibility for checking which suggestions were medically justified.
Under that arrangement, reported primary-diagnosis accuracy improved by approximately 20%, while false positives declined by around 44%. These changes indicate that clinical oversight helped preserve useful information while filtering out unsupported diagnoses in the experiment.
The design is encouraging but limited. Fifteen physicians working through a defined benchmark are not equivalent to a large multicentre clinical trial. The result does not establish how a similar workflow would affect emergency-department throughput, patient safety or healthcare costs under routine conditions.
Reviewing long AI reasoning traces may also take more time than checking short outputs. Any operational benefit would have to account for the clinician’s time, the way evidence is presented and whether the AI makes verification easier rather than harder.
A better model is not automatically a safer workflow
Healthcare AI discussions often focus on which model scores highest on a benchmark. The new findings suggest that this is too narrow a question for medical diagnosis.
A useful workflow must address how a clinician sees the model’s reasoning, how uncertain claims are identified, and what happens when the model names several conditions without sufficient supporting evidence.
In related Research Today coverage of machine learning for depression-treatment matching, a model could predict symptom severity but did not reliably select who would benefit more from a particular intervention. That study concerned a different medical task, yet it illustrates why predictive performance and clinical usefulness should not be confused.
Another Research Today investigation of an AI-supported lung-nodule pathway addressed changes in service access and follow-up. It highlights a complementary issue: an algorithm’s value depends partly on the care system into which it is introduced.
Similarly, research on AI-enabled diabetic eye screening among young patients shows why screening completion and diagnostic accuracy must be considered separately. In each setting, the correct evaluation measures depend on what the technology is actually expected to do.
What the findings do not establish
The benchmark used historical electronic health records rather than prospective clinical encounters. The labels used to judge answers depended on recorded diagnoses and data-processing choices, which can contain uncertainty or coding errors.
The systems did not observe patients directly, request additional examinations or adapt to new information in real time. A diagnostic assistant used interactively by clinicians might behave differently from a model given a retrospective case and asked to produce a list of answers.
The models and configurations were also specific to the research period. Subsequent changes to model architectures, medical training data or safety systems could alter the balance between primary-diagnosis accuracy and false positives.
The 15-physician validation was relatively small and used structured cases. It was not designed to establish that AI assistance improves survival, reduces diagnostic delays or lowers health spending across a real healthcare network.
Finally, the work did not show that the safest choice is always the model with the fewest suggestions. Omitting an important alternative diagnosis can also be harmful. The goal is appropriately broad clinical consideration combined with disciplined verification, not simply making a diagnostic list as short as possible.
What hospitals and AI developers should test next
Before reasoning-enabled AI is used for high-stakes decisions, researchers need prospective evaluations involving clinicians, patients and real workflow constraints. Such studies should measure whether the systems change decisions, introduce unwanted investigations or improve patient outcomes.
Developers also need to test consistency across repeated runs, provide clear evidence for suggested diagnoses and distinguish unsupported speculation from findings that warrant urgent attention.
Clinician oversight is a promising safeguard, but it must be designed deliberately. A human reviewer should have enough information to challenge the algorithm rather than merely approving a persuasive-looking explanation.
The strongest lesson from the study is not that sophisticated reasoning is bad for medicine. It is that finding the right answer more often does not automatically justify every extra answer a model generates.
Accuracy must be accompanied by verification
Across nearly 3,000 retrospective hospital cases, reasoning-enabled AI generally identified the primary diagnosis more accurately than matched conventional versions. Yet those gains came with more false positives and less consistent outputs.
The physician experiment showed that a structured process of clinical verification could reduce the safety trade-off under controlled conditions. It did not prove that autonomous diagnostic AI is ready for widespread deployment.
For hospitals considering these tools, the practical question is therefore not simply whether a reasoning model is more intelligent. It is whether the full combination of algorithm, clinician and clinical process produces decisions that are more accurate, appropriately cautious and safer for patients.
Looking for studies on this topic? Use the Research Today Research Paper Finder to search scholarly research by topic, question, author or keyword.
Source Information
Study title: The accuracy-safety trade-off of large reasoning models in clinical diagnostics
Authors: Shuai Wu, Linwei Chen, Yunxiang Fu, Xiaoman Zhang, Luyang Luo, Meng Lou, Yangxi Huang, Guanhua Chen, Ying Fu, Ya Xing Wang, Qian Di, Yuyin Zhou, Xiang Li, Tien Yin Wong, Yizhou Yu and Hong-Yu Zhou
Journal: Nature Communications
Publication date: 29 October 2026
Study design: Retrospective diagnostic AI benchmarking using six matched reasoning and non-reasoning model pairs across 12 specialties, repeated-output robustness tests and a controlled 15-physician validation experiment
Sample or scope: 2,978 retrospective hospital cases, comprising 1,157 MIMIC-IV admissions and 1,821 eICU cases; additional clinical evaluation with 15 physicians across 150 cases
Key finding: Reasoning-enabled models generated over 34% more false-positive diagnoses despite higher primary-diagnosis accuracy; structured physician verification improved diagnostic accuracy by approximately 20% and reduced false positives by 44% in the controlled validation experiment
