AI model hallucination will not be fully solved in 2024, though upcoming versions will be much better.
Unresolvable The public record could not settle it.
Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
The prediction has two parts: (a) hallucination is not fully solved in 2024, and (b) upcoming model versions are much better on hallucination. True: as of end-2024, leading OpenAI models still hallucinate measurably (shown by benchmarks, OpenAI system cards or credible reporting), AND OpenAI models released in 2024 (e.g. GPT-4o, o1) show a clear, documented fall in hallucination rates compared with GPT-4 on published factuality or hallucination evaluations. Mostly true: (a) holds and the improvement is real but modest or mixed across benchmarks. Mixed: (a) holds but evidence shows no meaningful improvement, or improvement in some evaluations matched by regression in others. Mostly false: (a) holds but 2024 models hallucinated as much as or more than predecessors on most published evaluations. False: hallucination was credibly shown to be fully solved in 2024 (highly unlikely, but this is the test).
Set before the research began.
Evidence (4)
-
E1 OpenAI o1 System Card OpenAI (arXiv) · 5 December 2024 Own record
“| SimpleQA | hallucination rate | \(0.61\) | \(0.44\) | \(0.44\) | \(0.90\) | \(0.60\) |”
-
E2 OpenAI o1 System Card OpenAI (arXiv) · 5 December 2024 Own record
“According to these evaluations, o1-preview and o1 hallucinate less frequently than GPT-4o, and o1-mini hallucinates less frequently than GPT-4o-mini.”
-
E3 Study suggests that even the best AI models hallucinate a bunch | TechCrunch TechCrunch · 14 August 2024
““At present, even the best models can generate hallucination-free text only about 35% of the time.””
-
E4 Study suggests that even the best AI models hallucinate a bunch | TechCrunch TechCrunch · 14 August 2024
“The results suggest that models aren’t hallucinating much less these days, despite claims to the contrary from OpenAI, Anthropic and the other big generative AI players.”
Reasoning
The evidence shows that leading models still hallucinated in 2024 [E3], but none of it compares OpenAI's 2024 models with GPT-4, as the criteria require. The only claim of improvement comes from OpenAI itself and compares o1 with GPT-4o [E2], while the independent report suggests models are not hallucinating much less [E4], so the evidence does not settle the size of the improvement.
Cite this claim
Sam Altman, “AI model hallucination will not be fully solved in 2024, though upcoming versions will be much better.” (said 18 March 2024). Verdict: unresolvable. Reliability API, claim c_ebb9f47a6e82, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_ebb9f47a6e82
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More prediction claims by Sam Altman
-
Mostly true
“on many coding tasks, o3-mini will outperform o1 at a massive cost reduction!”
On many coding tasks, OpenAI's o3-mini will outperform o1 at a massive cost reduction.
-
Mostly true
“And it’s going to be sad because GPT-5 is going to launch and there’s going to be another big spike in demand and we’re not going to be able to serve it.”
-
Mostly true
“AGI will feel within reach to many people in the industry”
By 2025, AGI will feel within reach to many people in the AI industry.
-
Resolved true
“We will get net-gain nuclear fusion working at prototype scale”
By 2025, net-gain nuclear fusion will be working at prototype scale.
-
Resolved true
“I believe her.”
Agrees with Kellyanne Conway that Trump is a man of action who is just getting started.
Verdicts are made automatically; how verdicts are made.