Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
19 claims
“Some of its advice will be head-scratchingly bad.”
Gary Marcus, author 27 July 2025 What to Expect When You're Expecting … GPT-5 (Marcus on AI)
Some of GPT-5's advice will be very bad.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
GPT-5 was released in August 2025. Judge within about 12 months of release. True: credible public sources document specific instances of GPT-5 giving clearly bad advice, meaning advice that is obviously wrong, harmful or absurd (for example in health, legal, financial or practical matters, or in simple reasoning), beyond trivial factual slips. Mostly true: such instances are documented, but they are few or only moderately bad rather than 'head-scratching'. Mostly false: the only documented errors are minor inaccuracies, or the bad-advice examples come from other models or only from after the window. False: independent evaluations within the window show no instances of clearly bad advice from GPT-5.
Evidence
- New model, old risks: sociodemographic bias and adversarial hallucinations vulnerability in GPT-5 - npj Digital Medicine npj Digital Medicine (Nature Portfolio) · 4 April 2026
“Adversarially, GPT-5 elaborated (hallucinated) on planted fabrications in 65% of runs under the standard prompt (95% CI 61.1–68.7).”
- Most medical AI chatbots vulnerable to attacks that trigger harmful advice, study finds KoreaBioMed · 5 January 2026
“The study found that even top-tier AI models, such as GPT-5 and Gemini 2.5 Pro, were fully exposed to such attacks, at times recommending drugs known to cause fetal abnormalities to pregnant patients.”
- Clinical Accuracy and Safety Concerns Following GPT-5 Public Demonstration in Cancer Care PubMed Central (peer-reviewed article) · 29 November 2025
“To put this into perspective, an 85% concordance rate, as we found, means that roughly 1 in 6 recommendations could be incorrect.”
- The fragile intelligence of GPT-5 in medicine Nature Medicine (Comment; authors at Stanford University) · 16 October 2025
“Although GPT-5 shows reduced hallucination rates compared with earlier models, clinical safety research suggests that the risk remains notable.”
Reasoning
Independent sources from within roughly 12 months of GPT-5's August 2025 release record clearly harmful advice from GPT-5. E2 reports that under attacks designed to trigger harmful advice, GPT-5 at times recommended drugs known to cause fetal abnormalities to pregnant patients. E3 found that about 1 in 6 of GPT-5's cancer-care recommendations could be incorrect, and E1 found that GPT-5 elaborated on planted medical fabrications in 65% of runs. Most of these errors came from adversarial setups or were reported as overall error rates, not as instances of GPT-5 giving absurd advice unprompted, so the verdict is mostly true rather than fully true.
Cite this claim
Gary Marcus, “Some of GPT-5's advice will be very bad.” (said 27 July 2025). Verdict: mostly true. Reliability API, claim c_69e54a064061, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_69e54a064061
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.