Autonomous Medical AI 'MIRA' Outperforms Specialists on Diagnostic Accuracy in Virtual Care
Nature study: MIRA beats specialists on virtual diagnosis; real-patient validation pending
※ This article was translated by AI.
The care process of the autonomous medical AI 'MIRA'. MIRA is designed to autonomously perform a series of clinical tasks in a virtual EHR environment, from patient history-taking to test requests and result review, diagnosis, drug prescriptions, surgery requests and admission decisions. (Image: recreated with ChatGPT from data in the Nature paper.). Recreated in English from the original slide.
[MedigateNews] An autonomous medical artificial intelligence (AI) that performs the entire care process on its own, from patient history-taking to test orders, diagnosis, drug prescriptions and admission decisions, recorded higher diagnostic accuracy and guideline adherence than specialists in a virtual care environment. The study offers early evidence that medical AI can carry out a continuous care process rather than isolated clinical tasks, but verifying its performance and safety in real patients remains a task for the future.
A research team that included Heidelberg University Hospital and the Technical University of Dresden in Germany evaluated the diagnostic and treatment decision-making abilities of an autonomous medical AI agent, 'MIRA (Medical Intelligence for Reasoning and Action)'. The study, 'Towards autonomous medical artificial intelligence agents,' was published in the international journal Nature.
In the same 311 patient cases, MIRA recorded a diagnostic accuracy of 87.8%, higher than the 78.1% of the specialist group. It also outperformed specialists on adherence to clinical practice guidelines, and it also performed post-diagnosis tasks, including drug prescriptions, surgery requests and admission decisions.
From History-Taking to Tests and Diagnosis: MIRA Performs the Care Process in a Virtual EHR
Previous large language model (LLM) research in medicine has focused mainly on evaluating performance on individual tasks, such as answering medical questions or diagnosing diseases.
The researchers said, "Existing limited approaches to using medical AI have not fully tapped the potential of LLMs across diverse medical tasks." They added, "A gap still remains in integrating medical AI agents into existing clinical work. The performance and safety of AI agents across the entire care process, from communication with patients through diagnosis, treatment decisions and admission, have also not yet been evaluated."
The team therefore developed MIRA, which links multiple tasks, from patient history-taking to test orders, interpretation of results, diagnosis and treatment planning. MIRA is designed not only to suggest a direction of care to clinicians but also to carry out follow-up tasks, such as drug prescriptions and surgery and admission requests, in a virtual electronic health record (EHR) system.
The researchers built a virtual emergency department environment based on the medical records of 574 real patients to evaluate MIRA's performance. The eight target conditions were appendicitis, cholecystitis, diverticulitis, pancreatitis, pulmonary embolism, pneumonia, pancreatic cancer and urinary tract infection.
In the experiment, an AI that answers based on a real patient's symptoms and medical history played the role of the patient. MIRA and the physicians used for comparison questioned the same AI patients to collect the information they needed and make a diagnosis.
In the diagnostic accuracy evaluation, MIRA recorded an accuracy of 88.9% across all 574 patient cases. This was assessed by how closely MIRA's diagnoses matched the real patients' discharge diagnoses.
In the 311 patient cases directly compared with physicians, MIRA's diagnostic accuracy was 87.8%. A group of four specialists recorded 78.1%, and a group of four residents and two specialists recorded 71.1%. The differences in diagnostic accuracy between MIRA and each physician group were all statistically significant (P<0.001).
By condition, MIRA's diagnostic accuracy for pancreatitis was 95.2%, higher than the 78.6% of the specialist group and the 61.9% of the mixed-experience physician group. For pancreatic cancer, MIRA and the specialist group showed the same level of accuracy.
Higher Guideline Adherence Than Specialists; Drug Prescriptions and Admission Decisions Also Evaluated
MIRA also outperformed the physician groups in treatment decisions after diagnosis.
In the researchers' evaluation of adherence to clinical practice guidelines for drug therapy, MIRA's adherence was on average 35 percentage points higher than the specialist group and 36 percentage points higher than the mixed resident-specialist group. The differences from both physician groups were statistically significant (P<0.001).
In test selection, MIRA tended to order some additional blood tests compared with the physicians. The additional tests were concentrated mainly in relatively low-cost items such as electrolytes and inflammatory markers, and MIRA did not show a tendency to systematically order more of the higher-cost imaging tests. The researchers assessed, "AI-led information gathering does not necessarily lead to overuse of medical resources."
In surgery requests, MIRA showed a high level of agreement with the actual medical records. It requested laparoscopic appendectomy in all 124 cases in which the real patients underwent that operation, and recorded a 90.6% agreement rate for laparoscopic cholecystectomy.
High accuracy was also confirmed in drug prescribing. The researchers analyzed 468 prescriptions MIRA wrote in 56 patient cases and found that 467 (99.8%) contained clinically useful and accurate free-text dosing and administration instructions. Accuracy by detailed item was 97.6% for drug dose, 98.3% for dose unit, 99.1% for administration frequency and 97.0% for administration route. A separate drug safety evaluation found no severe drug interactions, dose adjustments inappropriate for kidney function, risk of QT interval prolongation or dangerous opioid prescriptions.
The researchers also examined MIRA's ability to judge whether patients need admission. In patients with pneumonia and pulmonary embolism, MIRA missed no patients who needed admission in either condition, recording 100% sensitivity. Its specificity in correctly identifying patients who did not need admission was 86% for pneumonia and 70% for pulmonary embolism.
However, some limitations were found in treatment decisions for individual patients. In antibiotic treatment for some conditions, MIRA made decisions that did not match the guidelines, and in drug prescribing, the accuracy of the administration route was lower than other items. For patients with pulmonary embolism, MIRA showed a statistically significant tendency to decide on admission even when it was not needed.
The researchers emphasized, "Even if MIRA showed higher guideline adherence than physicians when all patients and treatment items are combined, some patients may have received treatment that deviated from best clinical practice." They added, "Providing safe, guideline-concordant treatment to every patient requires patient-level safeguards, continuous monitoring and iterative system improvement."
"Validation in Real Patients Not Yet Done…Phased Use Under Clinician Supervision Needed"
MIRA recorded high diagnostic accuracy and guideline adherence in the virtual care environment, but it has not been confirmed whether it can show the same performance in real patients.
In this study, the AI patients were built from the symptoms and medical history contained in real patients' discharge summaries.
The researchers explained, "Virtual patients' answers may be more systematic than the speech of real emergency department patients, and may not fully reflect speech disfluency, missing information or inconsistencies."
The scope of evaluation was also limited. The study focused on eight conditions, and the physician groups used to compare MIRA's performance consisted of four specialists and a mixed group of six residents and specialists.
The possibility that the AI model's training data and evaluation data partly overlapped must also be considered. It has not been confirmed whether the MIMIC-IV data used in the study were included in training the language model underlying MIRA.
The researchers explained, "The sources of the model's training data are not disclosed, so the possibility of overlap cannot be fully ruled out." They added, "The performance observed in this study may overestimate its ability to generalize to other patient cases."
For the early stage of adoption in actual care, the researchers proposed starting with areas where the scope of work is clearly defined, such as checking medications taken before admission, composing test orders, drafting consultation requests between departments and suggesting guideline-concordant prescriptions.
The researchers said, "This experiment provides early evidence that a single AI agent can carry out the care process from checking a patient's history to making an admission decision." They added, "In actual clinical settings, we expect adoption to begin with tasks that are limited in scope and performed repeatedly, under the explicit review of clinicians."
by Jiwon Lee
※ This article was translated by AI. If you find an error in the English, please let us know at news@medigatenews.com.