Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
42 claims
“The problem of hallucinations is not solved; reliability is not solved; planning on complex tasks is (as the authors themselves acknowledge) not solved.”
Gary Marcus, Substack author 24 March 2023 The Sparks of AGI? Or the End of Science? (Marcus on AI)
GPT-4 has not solved hallucinations, reliability or planning on complex tasks, as the paper's authors acknowledge for planning.
Resolved true The record shows it happened or was so.
What would count as true or false
Judged as of 2023-03-24. The claim has three parts: GPT-4 had not solved (1) hallucinations, (2) reliability, and (3) planning on complex tasks. It also says the paper's authors acknowledged (3). True: evidence from that time shows GPT-4 still produced hallucinations and unreliable outputs and failed at complex planning, and the paper discussed (e.g. Bubeck et al.) acknowledges limits in planning. OpenAI's own GPT-4 report admitting hallucination and reliability limits counts toward (1) and (2). Mostly true: all three limits are confirmed but the attribution is slightly off, e.g. the authors acknowledged the planning limit only in qualified or different terms. Mixed: one of the three limits is shown to have been substantially solved at that time, or the attribution is plainly wrong while the substantive limits hold. Mostly false: evidence from that time shows two of the three were substantially solved. False: evidence shows GPT-4 had effectively eliminated hallucinations and unreliability and could plan complex tasks reliably.
Evidence
- GPT-4 OpenAI · 14 March 2023
“Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).”
- GPT-4 Technical Report arXiv (OpenAI authors)
“Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).”
- Sparks of Artificial General Intelligence: Early experiments with GPT-4 arXiv (ar5iv rendering)
“Throughout the paper we emphasize limitations whenever we found one, but we also dedicate Section to an in-depth analysis of the lack of planning, likely a direct consequence of the autoregressive nature of GPT-4’s architecture.”
- GPT-4 Technical Report arXiv (OpenAI authors)
“GPT-4 significantly reduces hallucinations relative to previous GPT-3.5 models (which have themselves been improving with continued iteration).”
Reasoning
OpenAI's GPT-4 announcement and technical report (E1, E2) say GPT-4 "still is not fully reliable" and that it "hallucinates" facts and makes reasoning errors, which supports parts (1) and (2). The report also says GPT-4 significantly reduces hallucinations compared with GPT-3.5 (E4), but that is a reduction, not a solution. The Sparks of AGI paper by Bubeck et al. (E3) gives "an in-depth analysis of the lack of planning" in GPT-4, which supports part (3) and the claim that the authors acknowledged it.
Cite this claim
Gary Marcus, “GPT-4 has not solved hallucinations, reliability or planning on complex tasks, as the paper's authors acknowledge for planning.” (said 24 March 2023). Verdict: resolved true. Reliability API, claim c_65baa675ff5a, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_65baa675ff5a
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.