Marcus said he would be truly astonished if hallucinations in large language models were brought down to human expert levels by the end of 2024.
Unresolvable The public record could not settle it.
Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 31 Dec 2024. The claim is that LLM hallucination rates were not brought down to the level of human experts by end-2024. True: by end-2024, leading LLMs (e.g. GPT-4o, o1, Claude 3.5, Gemini) still hallucinated at rates clearly above human experts on comparable factual tasks, as shown by benchmarks (e.g. SimpleQA, hallucination leaderboards), domain studies (legal, medical citations) or expert consensus. Mostly true: leading models matched experts only in narrow, specific settings (e.g. grounded summarisation) while hallucinating well above expert levels in general use. Mixed: the evidence is genuinely split, with credible studies showing expert-level parity in several major domains and others showing clear gaps. Mostly false: by end-2024 most credible evaluations found leading models at or near human-expert error levels across most tasks, with gaps only at the margins. False: broad, credible evidence by end-2024 showed LLM hallucinations at or below human expert levels in general.
Set before the research began.
Evidence (4)
-
E1 OpenAI o1 System Card arXiv (OpenAI system card) · 5 December 2024 Own record
“| SimpleQA | hallucination rate | \(0.61\) | \(0.44\) | \(0.44\) | \(0.90\) | \(0.60\) |”
-
E2 OpenAI o1 System Card arXiv (OpenAI system card) · 5 December 2024 Own record
“All models underperform the consensus (57%) and median (42%) expert baseline.”
-
E3 Measuring short-form factuality in large language models OpenAI
“SimpleQA is created to be challenging for frontier models (e.g., GPT-4o and Claude both score less than 50%).”
-
E4 Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools alphaXiv (preprint by Magesh et al., Stanford RegLab) · 30 May 2024
“we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.”
Reasoning
The evidence shows that leading models and legal AI tools hallucinated at substantial rates in 2024 (SimpleQA rates of 0.44 to 0.90 in E1, 17% to 33% for legal tools in E4). No item gives a human-expert hallucination rate on a comparable task to measure these against; E2's expert baselines are for an unnamed benchmark that is not a hallucination measure.
Cite this claim
Gary Marcus, “Marcus said he would be truly astonished if hallucinations in large language models were brought down to human expert levels by the end of 2024.” (said 31 March 2024). Verdict: unresolvable. Reliability API, claim c_9be207f4aafc, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_9be207f4aafc
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More prediction claims by Gary Marcus
-
Mostly true
“Some of its advice will be head-scratchingly bad.”
-
Mostly true
“GPT-5 will surely be better, a lot better than GPT-4.”
-
Resolved true
“GPT-5 really is about to drop, no foolin’.”
-
Resolved true
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
GPT-5 will still make a significant number of obvious errors that are hard to predict.
-
Mostly true
“Neurosymbolic AI will become much more prominent.”
Verdicts are made automatically; how verdicts are made.