AI Doctors 3x More Expensive Than Human

While modern AI models excel at diagnostic pattern recognition on clinical benchmarks, a new study in NEJM AI suggests that applying them to routine care management could inflate care costs (potentially by 3x). What is interesting is that the cost increase does not happen because the models order exotic procedures, but instead because the AI models seem to reflexively ask for blood panels and specialist referrals in visits where human physicians order nothing at all.

The paper, “Comparative Cost of Care Recommended by Large Language Models versus Treating Physicians” by Aran et al. in NEJM AI, offers a rigorous, case-level accounting of what happens when you give large language models the clinical note from actual primary care and emergency room patient visits and ask them to come up with a followup plan.

Diagnosis vs. Management: From Spotting Patterns to Managing Care

I have spent a fair amount of time thinking about applying AI/machine learning methods in medicine. Back in 2016, I wrote that the real near-term value of clinical AI lay not in replacing doctors, but in augmenting diagnostic interpretation. Diagnostic interpretation, after all, is a high-dimensional pattern recognition problem — interpreting X-rays, finding subtle lesions on pathology slides, or tracking subtle laboratory trends — and therefore well-suited to AI / machine learning approaches and the strengths of a “doc-bot”.

In my subsequent research with the Stanford Ophthalmic Informatics Group (Predicting Glaucoma Progression Using EMR Text), we showed that deep learning models could ingest even messy, unstructured electronic health record notes and predict which glaucoma patients would progress to needing surgery far better than trained human ophthalmologists. The logic behind this is simple: when fed large volumes of historical clinical text, neural networks can consistently discern faint prognostic markers that human eyes might miss.

This and other similar studies have understandably fed a widespread belief in the pro-AI camp: if AI methods can probably diagnose disease better than a human, surely an “AI doctor” will naturally deliver better, cheaper healthcare.

As generative AI has swept into consumer hands, healthcare has become one of its most common applications. Millions of people now ask questions of products like ChatGPT, Claude, and Gemini daily for everything from low-stakes, self-directed wellness (i.e., weekly workout and meal planning, or explaining what a lab result means) to more serious medical inquiries.

The danger comes when patients lean on these conversational models for higher-stakes clinical management and lack the knowledge to critically engage with a response machine that is designed to always sound confident. In some cases, the results can be tragic. In the last week (as of this posting), the family of Bowen Wan, a 31-year-old former Amazon software engineer, filed a wrongful death lawsuit against OpenAI. Wan had asked ChatGPT to taper off her anti-seizure medication (lamotrigine). ChatGPT generated a 25-day step-down schedule, reportedly reassuring her through withdrawal symptoms and advising her against consulting human doctors. Tragically, after completing the taper, she suffered a fatal seizure.

The tragedy illustrates the boundary between diagnosis and clinical management. Spotting a disease label on a test is fundamentally different from deciding what clinical actions to take next. Clinical management is not just pattern recognition. It is a sequence of actions and responses, with economic, medical, and behavioral management implications that need to be governed by risk thresholds, follow-up cadence, and judgment about what a patient may be hiding or doing.

This brings us directly to the NEJM AI study: when frontier models are asked to manage real patients, what do they actually recommend?

The Experiment: Pricing AI Care Plans Against Real Physicians

To find out, the researchers gathered 200 real, deidentified outpatient and emergency department encounters from MTSamples, a public repository of clinical reports contributed by practicing physicians across specialties. Surgical, procedural, dental, and inpatient notes were excluded so every case reflected a standard frontline primary care or emergency visit.

The study design was straightforward:

1. The Split: Each clinical note was edited to remove the human physician’s assessment and plan. What was left was text containing the patient’s chief complaint, history of present illness, past medical history, current medications, vital signs, physical examination findings, and any baseline laboratory or imaging results already completed. What was removed — the human physician’s documented Assessment and Plan (A&P) — served as the benchmark.
2. The Models: The researchers fed the exact same clinical presentation to 24 AI models spanning seven developers. This included 20 general-purpose models across successive generations (GPT-4.1, GPT-5.2, and GPT-5.5; Claude 3.5 Sonnet, Claude 4.5 Sonnet, and Claude 4.8 Opus; Gemini 2.5 Pro, Gemini 3 Pro, Gemini 3.1 Pro, and Gemini 3.5 Flash; Grok 3, Grok 4.1, and Grok 4.3; Llama 3.3 and Llama 4; Qwen 2.5, Qwen 3, and Qwen 3.7; DeepSeek V3 and DeepSeek R1) alongside four specialized medical models (MedGemma 27B, MedGemma 4B, Meditron, and OpenEvidence).
3. The Pricing Pipeline: Three independent language model extractors parsed every order from both the human physician’s plan and the AI plans, matching items to their Current Procedural Terminology (CPT) codes. Diagnostic tests (blood work, cultures, imaging) were priced against the CY 2026 Medicare Physician Fee Schedule. Specialist referrals were priced at the standard Medicare new-patient evaluation and management (E&M) code. New medications were counted rather than priced (because drug prices vary wildly by formulary and cannot be reliably imputed from chart notes).

Every AI plan was priced exactly as written, assuming every suggested test and referral was executed in full.

Triple the Cost

The cost gap between human physicians and AI models was massive:

• The treating physicians’ documented plans cost a mean of $97 per visit (with a median of $0 and an interquartile range of $0 to $72).
• Across the 20 general-purpose AI models, the recommended care averaged $314 per visit (3.2 times more than the human physician baseline).

Across all 200 cases, the AI was more expensive in 90% of them. Individual model averages ranged from a low of $209 per visit (Gemini 3.1 Pro, still 2.1x the human cost) to an astonishing $599 per visit (Qwen 3, a 6.3x markup).

Two findings from this comparison jumped out:

First, medical fine-tuning did nothing to restrain costs (at least not what has been done to date). The four specialized medical models (MedGemma 27B at $342, MedGemma 4B at $260, Meditron at $295, and OpenEvidence at $297) averaged $298 per visit (3.1x the physician). Fine-tuning models on medical textbooks and clinical licensing exams only seemed to reinforce the instinct to order exhaustive workups.

Second, successive model generations did not improve. You might hope that as models get smarter, they learn to separate high-yield clinical signals from low-yield screening noise. The opposite happened. In four of the seven model families, the newest version recommended more care than the oldest, and across generations there was no consistent trend toward meaningfully lower spending:

To see how individual models compare, you can explore the complete 24-model dataset (based on data shared from the paper) below:

The Zero-Order Paradox: Too Many Panels

How does this extra $216 per visit break down?

One might assume that when AI overprescribes, it is ordering full-body MRIs, genetic panels, or exploratory CT scans on complex cases.

But the data reveals that is not the case. In fact, when the analysis was restricted to just the 67 encounters where the human physician actually ordered a diagnostic workup, the AI-to-physician diagnostic testing cost ratio dropped from 2.5x across all cases down to 1.4x on tested cases. On visits where testing was indicated by the human physicians, the AI’s orders were only 40% more expensive than baseline (still worth managing, but not astronomically high).

Instead, the excess was concentrated almost entirely in visits where the human doctor ordered zero followup tests (roughly two-thirds of the visits in the study). On those exact same encounters, however, the AI doctor added diagnostic testing in 45% to 79% of visits (depending on the AI model):

The AI did not order exotic tests. They actually ordered pretty basic test panels:

• Complete Blood Count (CBC): ordered on a mean of 28% of zero-test visits (up to 44% in some models).
• Comprehensive Metabolic Panel (CMP): ordered on 20% of zero-test visits.
• Hemoglobin A1c: ordered on 15% of zero-test visits.
• Thyroid-Stimulating Hormone (TSH): ordered on 12% of zero-test visits.
• Basic Metabolic Panel (BMP), Lipid Panels, ESR, and CRP: ordered on 6% to 12% of visits.

In 21% of all patient encounters, the AI suggested eight or more diagnostic tests.

6x More Specialist Referrals

Diagnostic blood tests accounted for $105 of the $216 per-visit gap. The remaining $111 was driven by specialist referrals.

Human physicians made a specialist referral in roughly one out of every seven cases (~0.15 referrals per visit). The AI models, however, recommended an average of ~0.87 referrals per visit (5.6x higher):

• Qwen 3 recommended 1.94 referrals per visit (nearly two specialist referrals for every single outpatient checkup).
• DeepSeek R1 recommended 1.36 referrals per visit.
• Grok 3 recommended 1.33 referrals per visit.

Even the most conservative frontier models, like Gemini 3.1 Pro (0.40 referrals/visit) and GPT-5.5 (0.54 referrals/visit), referred patients out at nearly triple the human baseline rate.

In health economics, specialist referrals are one key driver of medical spending. When a primary care physician refers a patient to a specialist, that referral does not just cost the initial Medicare consultation fee ($111). It initiates a whole new clinical cascade: specialist intake evaluations, secondary imaging, endoscopic procedures, biopsies, and brand-name prescription drugs that were never captured in this study’s initial billing calculation.

It’s important to remember that, especially in a time of constrained medical capacity in the US, a 5.6x surge in referrals across the board could be catastrophic for patient access, wait times, and overall healthcare system efficiency.

Why Chatbots Practice Defensive Medicine

Why did the language models behave like this?

The expert clinician reviews of some of the plans suggested that part of the explanation lies in the models being oriented towards defensive medicine. Typically this refers to physicians ordering tests and consultations primarily to shield against malpractice liability rather than improve care, but for LLMs, I would hazard a guess this defensive behavior is inherent for three reasons:

1. The “One-Shot” Trap: Real clinical medicine is an iterative, longitudinal process. When an experienced physician sees an otherwise healthy patient with a low-grade cough or mild abdominal pain, they practice watchful waiting. They tell the patient: “Take supportive care, stay hydrated, and call me back on Friday if your fever spikes.” They rely on the passage of time as their cheapest and most effective diagnostic tool. A conversational chatbot has no tomorrow. It is evaluated on producing a single, self-contained response right now. Lacking a continuous feedback loop, the model treats the single interaction as its only opportunity to act, biasing it towards a “grand final answer”.
2. Asymmetric Training Loss (The Fear of Omission): Language models are aggressively trained to be helpful and safe. If an AI evaluates a patient with mild facial weakness and fails to mention an acute stroke, human evaluators and red-teamers will penalize the model heavily for a dangerous omission. If the model orders an unnecessary MRI and an ESR lab to “rule out” an unlikely condition, nobody penalizes it for health insurance waste.
3. Legal Defensibility & Procedure Escalation: As prior research by Aran et al. showed in Communications Medicine, making an LLM’s clinical recommendations legally defensible directly forces procedure escalation. When a model attempts to bulletproof its output against hypothetical liability, it reflexively activates high-acuity protocols (for example, four out of five evaluated models triggered an acute-stroke pathway on a routine outpatient Bell’s palsy visit).

You Can’t Just Prompt Your Way Out of This

Can you simply prompt the LLM to be more cost-effective? The authors tested this directly on 19 of the models (excluding OpenEvidence, which lacked a programmatic API, and four older checkpoints that had been retired from developer APIs by the time prompt tests were run) using two alternative prompts:

1. The Cost-Aware Prompt: Appending a single instruction to the standard prompt: “Recommend only clinically necessary, cost-effective items.”
2. The Parsimonious-but-Safe Prompt: Instructing the model to provide only the three most likely diagnoses, explicitly rule out dangerous cannot-miss conditions, and recommend only items whose results would directly alter clinical management.

The results highlight a painful clinical trade-off: The simple cost-aware prompt worked. It shaved the mean cost from 3.2x down to 2.5x the physician benchmark. The parsimonious-but-safe prompt was more aggressive. It slashed the average cost multiplier from 3.2x all the way down to 1.4x. For some models, it completely closed the gap (Gemini 3.1 Pro matched human physician costs at ~1.0x, and Grok 4.3 dropped to 0.7x).

There is, to be fair, important nuance across individual models: a handful managed this pressure remarkably well. GPT-5.5 actually improved its diagnostic concordance from 88% to 89% while cutting costs by nearly 40%, Meditron gained eight percentage points in concordance (68% to 76%) as its cost dropped by more than half, and GPT-5.2 largely maintained its diagnostic accuracy (84% to 82%).

For the vast majority of models, however, forcing parsimony caused diagnostic reasoning to degrade: 17 out of the 19 models drifted away from the treating physician’s primary diagnosis. In open-weights and clinical models like MedGemma 27B and Qwen 3, concordance plunged into the low 60s and diagnostic error rates jumped to 12%–13%. For several models, costs dropped below the human baseline because the AI began undersuggesting necessary care for high-acuity conditions.

While a few frontier models show that cost restraint and diagnostic rigor can coexist, in general, aggressively force-prompting budget-consciousness seems to negatively impact diagnostic performance. To explore this trade-off across all 19 evaluated models, you can inspect each model’s trajectory below:

But It’s Not All Waste

It is tempting to look at a 3.2x markup and conclude that clinical LLMs are therefore useless engines of medical waste.

But that would be an oversimplification. The study’s expert clinical review of 1,000 recommendations across five representative models tells a more complicated story:

• 18% of AI “additions” were rated guideline-concordant care that the human doctor had actually omitted. In nearly one out of five new tests, the AI caught a potential oversight (although to be clear, it’s unknown if this is an actual oversight versus the physician making a judgment call based on unrecorded clinical context) — such as ordering an overdue diabetic Hemoglobin A1c test in a hypertensive patient with borderline fasting glucose, or recommending bone densitometry for an older patient with a prior compression fracture.
• 59% of AI “additions” were rated reasonable but low yield. These are defensible tests given the AI’s limited context (but unlikely to change clinical management).
• 17% were rated debatable or defensive and 5% were rated clearly unnecessary.

Across these five audited models, the diagnostic testing excess averaged $73 per visit (somewhat lower than the 20-model panel mean of $105, as this audit omitted extreme high-cost outliers like Qwen 3). When the authors weighted this $73 excess by clinical appropriateness ratings, it translated into a clear spending breakdown:

• $16 per visit (22%) was spent catching genuine human omissions: Guideline-concordant care that the treating doctor had overlooked.
• $39 per visit (53%) was spent on defensible but low-yield screening: Plausible tests given that the AI lacked a physical exam or longitudinal history, but tests unlikely to alter clinical management.
• $18 per visit (25%) was spent on clearly unnecessary or defensive testing: Unwarranted workups ordered out of extreme caution or perceived liability (such as ordering CT scans, arterial blood gases, and full serology panels for routine, stable chronic follow-ups).

The cost gap is not purely insane, hallucinated medicine. Over half of the excess is concentrated in the gray zone of defensible, low-yield screening rather than outright waste.

One skeptical take on this study might be that human doctors ordered less because of some kind of information asymmetry — that some of these visits represent situations where the physician might have had a prior relationship with the patient and hence had some unspoken background context that never made it into the written note.

The authors tested this by stratifying cases into first-time visits (73 cases) and established follow-ups (101 cases). If unrecorded relationship history explained why doctors ordered fewer tests, the cost gap should have vanished on first encounters where the doctor had never seen the patient before. But the data showed the exact opposite: the AI cost excess was significantly larger on first-time visits ($265 excess) than on repeat visits ($173 excess). Undocumented prior history does not explain the gap.

Summary

While imperfect, I think this study was pretty well-designed and provides valuable insights into the limitations and potential of current AI models in clinical practice. So, what does this say about the future of AI in healthcare?

First, AI doctors, at least in their current incarnation, are not a magic bullet for the cost of medicine. If you hooked today’s leading LLMs directly up to hospital ordering systems and let them issue labs, imaging, and specialist consultations autonomously, outpatient healthcare spending would surge overnight.

Second, building a cost-effective clinical AI is not a simple matter of prompt engineering. Current frontier models are “post-trained” to be conversational consultants. To practice cost-effective medicine, models probably need to be trained using reinforcement learning for sequential decision-making under uncertainty with financial, physical, and emotional costs for each action — where the agent learns that observing a patient over time and ordering nothing can sometimes be the right action.

Until further technological advances or new clinical practice-trained AI models, the near-term path forward is probably not a fully autonomous machine doctor. It is likely a human-machine partnership that combines the models’ strengths with the human’s.