Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“Hallucinations (which should really be called confabulations) will continue to haunt generative AI.”
Gary Marcus, Substack author 1 January 2025 25 AI Predictions for 2025, from Marcus on AI (Marcus on AI)
Hallucinations will continue to afflict generative AI in 2025.
Resolved true The record shows it happened or was so.
What would count as true or false
Judged at the end of 2025. True: throughout 2025 leading generative AI systems still produced hallucinations or confabulations in a meaningful way. Evidence can include hallucination benchmarks, vendor system cards reporting non-trivial hallucination rates, and documented incidents such as fabricated citations in court filings or false outputs from AI products. A fall in rates that leaves hallucinations widespread still counts as true. Mostly false: hallucinations had become rare and marginal in leading systems by 2025. False: hallucination was effectively solved in the major systems during 2025.
Evidence
- Are AI Hallucinations Getting Better or Worse? We Analyzed the Data ScottGraffius.com (Scott M. Graffius) · 7 January 2026
“OpenAI’s o3 series, for example, experienced hallucination rates of 33-51% on PersonQA and SimpleQA.”
- Are AI Hallucinations Getting Better or Worse? We Analyzed the Data ScottGraffius.com (Scott M. Graffius) · 7 January 2026
“Several top models dropped below 1%, including Google’s Gemini-2.0-Flash at roughly 0.7%, with OpenAI and Gemini variants clustering around 0.8–1.5% (Vectara, 2025; AllAboutAI, 2025).”
- California issues historic fine over lawyer’s ChatGPT fabrications CalMatters · 22 September 2025
“The fine appears to be the largest issued over AI fabrications by a California court and came with a blistering opinion stating that 21 of 23 quotes from cases cited in the attorney’s opening brief were made up.”
- OpenAI o3 and o4-mini System Card OpenAI · 16 April 2025
“The o4-mini model underperforms o1 and o3 on our PersonQA evaluation. This is expected, as smaller models have less world knowledge and tend to hallucinate more.”
Reasoning
OpenAI's April 2025 o3/o4-mini system card says o4-mini underperforms on its PersonQA hallucination evaluation, and that smaller models "tend to hallucinate more" (E4). An independent review reports hallucination rates of 33–51% for the o3 series on PersonQA and SimpleQA (E1). In September 2025 a California court fined a lawyer after finding that 21 of 23 case quotes in a brief were made up (E3). Some top models had hallucination rates below 1% in figures citing Vectara (E2), but that does not show hallucination was solved or rare in leading systems, so the criteria's "true" condition is met.
Cite this claim
Gary Marcus, “Hallucinations will continue to afflict generative AI in 2025.” (said 1 January 2025). Verdict: resolved true. Reliability API, claim c_75716b453035, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_75716b453035
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.