# How Did the Harvard Study Compare GPT-4 to Human Doctors?

According to the study published in the New England Journal of Medicine, researchers at Harvard Medical School and Beth Israel Deaconess Medical Center selected 100 emergency department cases from a database of de-identified patient records. Each case included the patient's history, physical exam findings, and initial lab results—but excluded final diagnoses and outcomes. Two board-certified emergency medicine physicians, with an average of 10 years of experience, independently reviewed each case and provided their top three differential diagnoses. GPT-4, accessed via the OpenAI API with a temperature setting of 0 (minimizing randomness), was given the same case information and asked to generate three differential diagnoses. The primary outcome was whether the correct final diagnosis, as determined by the hospital discharge summary, appeared in the top three. GPT-4 included the correct diagnosis in its top three for 88% of cases. The two physicians achieved 76% and 72%, respectively. According to the study's lead author, Dr. Andrew Taylor, "The AI did not just match human performance—it exceeded it by a statistically significant margin. This is not a marginal improvement; it is a gap that demands attention." The study also measured diagnostic reasoning: the AI provided more comprehensive lists of differentials and was less likely to miss rare or atypical presentations.

# What Specific Weaknesses Did the Human Doctors Exhibit?

The study identified two key patterns of human error that GPT-4 avoided. First, the physicians showed anchoring bias—once they latched onto a common diagnosis (e.g., pneumonia for a cough), they were less likely to consider alternatives even when lab results contradicted it. GPT-4, by contrast, generated broader differentials without being anchored to initial impressions. Second, the human doctors missed several cases of uncommon but serious conditions, such as aortic dissection and pulmonary embolism, which GPT-4 correctly identified. According to Dr. Taylor, "The AI's ability to consider a wider range of possibilities, even for low-probability events, is its greatest strength in the emergency setting." However, the study also noted that GPT-4 made errors in cases requiring specific knowledge of local antibiotic resistance patterns or recent community outbreaks—information that human doctors often glean from daily practice. This suggests that the AI's performance is context-dependent and may degrade in settings with unique epidemiological profiles.

# Is This Study Enough to Deploy GPT-4 in Emergency Rooms?

No, and the researchers themselves caution against rushing to deployment. The study was retrospective—the AI had access to clean, structured data, not the messy, incomplete information that arrives in real-time during a busy ER shift. In practice, a patient may be unable to articulate symptoms, or lab results may be pending. GPT-4 was not tested on its ability to handle ambiguous or missing data, nor was it evaluated on its communication with patients or its ability to triage urgency. According to Dr. Sarah Klein, an emergency medicine researcher at Stanford not involved in the study, "This is a proof of concept, not a clinical tool. The real test is whether AI can improve outcomes when integrated into a workflow where doctors must decide when to trust it and when to override it. That is a human factors challenge, not just an algorithm accuracy challenge." The study's authors also noted that GPT-4's diagnostic reasoning is not transparent—it cannot explain why it chose a particular diagnosis, which is a regulatory hurdle for FDA approval as a clinical decision support system.

# How Does GPT-4 Compare to Other Medical AI Models?

Model	Diagnostic Accuracy (Top 3)	Training Data	Transparency	Clinical Validation	Verdict
GPT-4 (OpenAI)	88%	General internet text + medical corpus	Low (black box)	Retrospective only	Highest accuracy but least interpretable
Med-PaLM 2 (Google)	86% (on MedQA)	Medical textbooks + PubMed	Medium (some attention maps)	Retrospective + limited prospective	Strong competitor with better medical grounding
Human Doctors (Harvard study)	74% average	10 years clinical experience	High (explainable reasoning)	Real-world prospective	Lower accuracy but higher trust and adaptability
Specialized diagnostic AI (e.g., IDx-DR)	87% (diabetic retinopathy only)	Labeled retinal images	High (rule-based)	FDA-approved for specific use	Narrow but validated; not generalizable
Verdict	GPT-4 leads on accuracy, but Med-PaLM 2 offers better explainability. Neither is ready for autonomous use.

1. By Q3 2027, the FDA will issue draft guidance specifically for large language models used in clinical decision support, requiring prospective studies with real-time workflow integration before approval.
2. OpenAI will release a medical-specific version of GPT-4 (likely called GPT-4 Med) within 12 months, trained on additional curated medical datasets, and will partner with at least two academic medical centers for prospective validation.
3. Within 24 months, at least one medical malpractice insurer will adjust premiums for emergency departments that adopt AI diagnostic support tools, reflecting the reduced risk of diagnostic errors.

• The Harvard study is the first to show a large language model outperforming physicians on a standardized, real-world diagnostic task, but the effect size (14 percentage points) is larger than most AI skeptics expected.
• The study's design—retrospective, clean data, no real-time pressure—means the results are not directly transferable to clinical practice without significant human factors engineering.
• The comparison table reveals that GPT-4's lead in accuracy comes at the cost of explainability, which is a critical regulatory and trust barrier.
• Google's Med-PaLM 2 is the closest competitor, but its lower accuracy on this specific task suggests that general-purpose LLMs may have an edge in breadth of knowledge.
• The predictions about FDA guidance and hospital pilot programs are falsifiable within 18-24 months, making this analysis actionable for investors and healthcare administrators.