Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“It will not be trustworthy and complete enough to give reliable medical advice, despite devouring a large fraction of the Internet.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4 will not be trustworthy and complete enough to give reliable medical advice.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
This prediction is judged on GPT-4 as released (March 2023) and evaluated during 2023. TRUE: credible evidence shows GPT-4 was not reliable enough to give medical advice without supervision. This can be peer-reviewed or independent studies documenting clinically significant errors, hallucinations or omissions in medical answers, or OpenAI's own documentation warning against relying on it for medical advice. MOSTLY TRUE: GPT-4 performed strongly on many medical tests, for example exam-style benchmarks, but notable reliability failures in real advice settings were still documented. MOSTLY FALSE: most evaluations found GPT-4's medical advice broadly reliable, with only isolated problems. FALSE: credible evaluations found GPT-4 gave consistently reliable and complete medical advice, comparable to clinicians, and it was treated as trustworthy for that purpose. Passing medical licensing exams alone does not count as reliable medical advice.
Evidence
- GPT-4 language model gives limited quality medical answers in study MedTech Dive · 5 July 2023
“The GPT-4 AI gave responses that matched 15 of the 25 recommendations.”
- ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases medRxiv · 23 January 2025
“ChatGPT 4 is more successful than ChatGPT 3.5 (Wilcoxon signed rank test, alpha=0.05, p<0.004) but still far from the performance of the common cases. 40% of all cases are solved with the first suggested diagnosis while it takes 8 or more suggestions to reach a 90% diagnostic accuracy.”
- ChatGPT as a medical doctor? A diagnostic accuracy study on common and rare diseases medRxiv · 23 January 2025
“The results for ChatGPT 4 are even superior and 100% of all cases were solved in all 3 prompt versions within 3 suggested diagnoses (top3 accuracy).”
- GPT-4 System Card - cdn.openai.com OpenAI · 13 May 2025
“As noted above in 2.2, despite GPT-4's capabilities, it maintains a tendency to make up facts, to double-down on incorrect information, and to perform tasks incorrectly.”
Reasoning
An independent 2023 study reported that GPT-4's answers matched only 15 of 25 guideline-based medical recommendations (E1). OpenAI's own GPT-4 system card warns that the model tends to make up facts, double down on incorrect information and perform tasks incorrectly (E4). A diagnostic accuracy study found GPT-4 did well on common cases, solving all of them within its top three suggestions (E3). On rare diseases, though, its first suggestion was right in only 40% of cases (E2). That pairs strong performance on some tests with documented reliability failures, which matches the mostly-true level of the criteria.
Cite this claim
Gary Marcus, “GPT-4 will not be trustworthy and complete enough to give reliable medical advice.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_6baf6d86ea1a, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_6baf6d86ea1a
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.