In recent years, the integration of artificial intelligence (AI) within the medical field has spurred considerable debate regarding the efficacy and reliability of these technologies in clinical reasoning. A study conducted by researchers at the University of Alberta, published on November 24, 2025, reveals that while AI has shown remarkable capabilities in answering multiple-choice questions, it continues to lag behind human doctors in the essential skills required for clinical reasoning.

The Study Overview

The research, spearheaded by Liam McCoy, a neurology resident at the University of Alberta, focused on assessing the performance of ten prominent AI models in clinical reasoning tasks. The study utilized a benchmark tool based on script concordance testing, which is traditionally employed in medical education to evaluate the soft skills of medical and nursing students as they navigate through complex clinical scenarios.

Objectives of the Research

  • Evaluate large language models' ability to analyze symptoms and order the appropriate tests.
  • Assess the models’ flexibility in updating judgments based on new and changing clinical information.
  • Determine the models’ capability to differentiate between relevant and irrelevant data in clinical contexts.

Key Findings

The findings of McCoy's research underscore the limitations of current AI systems, particularly in adapting to dynamic clinical situations. Specifically, he observed:

Finding Detail
Performance Comparison AI models matched the performance of first- and second-year medical students but did not reach the standards of senior residents or attending physicians.
Judgment Updating Models demonstrated significant difficulty in adjusting their analysis based on new and relevant clinical information.
Irrelevant Information Models often failed to recognize irrelevant information, with about 30% of the data presented in test scenarios being red herrings.

Clinical Reasoning vs. AI Performance

McCoy's observations highlight a profound distinction in the approach to clinical reasoning between humans and AI. While AI may excel in recalling factual information, the nuanced process of synthesizing data and developing a clinical narrative remains a challenging feat for these models. McCoy elaborates:

“Large language models display impressive academic performance, yet the nuances involved in clinical reasoning are still beyond their capabilities. For a patient to receive safe and accurate diagnosis through AI assistance still requires considerable human oversight.”

The Relevance of Script Concordance Testing

Script concordance testing plays an integral role in medical training by allowing students to earn points based on their ability to align with established clinical reasoning patterns recognized by experts. This method assesses students in scenarios ranging from surgery to pediatrics and emergency medicine. McCoy emphasized the importance of developing these scripts:

  • New medical learners follow simple scripts that grow more sophisticated with experience.
  • Advanced practitioners can adapt their scripts according to changing clinical presentations and information.
  • AI models falter in this adaptability, often sticking to rigid interpretations of presented data.

Implications for Future AI Development in Medicine

These findings present a critical insight into the future trajectory of AI in clinical applications. McCoy believes that while AI can enhance clinical tasks, the technology must evolve to better support healthcare professionals:

  • Researchers must prioritize improving AI’s ability to handle dynamic situations that require clinical acumen.
  • Collaboration between AI developers and healthcare professionals is essential to ensure the technology aligns with patient-centered care needs.
  • Ethical considerations must guide the integration of AI technology to avoid jeopardizing patient safety.

Conclusion

In conclusion, while AI continues to advance and may assist in various medical functions, it is still a long way from replacing human expertise in clinical reasoning. Results from the University of Alberta's study highlight both the potential and the current limitations of AI in medicine. As this technology matures, it will be imperative to enhance its practical applications while maintaining the irreplaceable human elements that define the physician-patient relationship.

References

[1] McCoy, L. G., et al. (2025). Assessment of Large Language Models in Clinical Reasoning: A Novel Benchmarking Study. NEJM AI.

[2] Science X. (2025). Doctors still outperform AI in clinical reasoning, study shows. Retrieved from Science X.

Explore Further

For related stories on the impact of AI in healthcare, visit: