Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“I suspect that what we will see is a familiar pattern: immense initial buzz, followed by a more careful scientific inspection, followed by a recognition that many problems remain.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
Marcus suspected that GPT-4's release would follow a pattern of immense buzz, then scientific scrutiny, then recognition that many problems remain.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge this against the period after GPT-4's release. True: GPT-4's launch drew immense initial buzz, was then followed by careful scientific evaluation (research papers, independent tests), and that work led to broad recognition that substantial problems remained, such as hallucination, reasoning errors and unreliability. Mostly true: the three stages happened, but in a slightly different order or the stages overlapped. Mixed: there was buzz, but scrutiny produced mixed conclusions, with no broad recognition that many problems remained. Mostly false: the buzz was modest, or the scrutiny mostly concluded that the major problems had been solved. False: GPT-4 was never released, or its release was followed by little buzz and the scrutiny found no significant remaining problems.
Evidence
- The AI that sparked tech panic and scared world leaders heads to retirement Ars Technica · 30 April 2025
“Its release created a wave of immense hype—and existential panic—about AI’s ability to imitate human communication and composition.”
- roundups & rapid reactions Science Media Centre · 15 March 2023
“Although it doesn’t solve the problems of hallucinations, the OpenAI benchmarks show it is more factual by more than 40% over GPT-3.”
- GPT-4 OpenAI · 14 March 2023
“Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).”
- How Is ChatGPT's Behavior Changing Over Time? (Chen, Zaharia & Zou 2024) AI for Science (record summarising Chen, Zaharia & Zou, Harvard Data Science Review; arXiv:2307.09009, 2023)
“The same service name returned substantially different behaviour within three months: GPT-4's accuracy at identifying prime versus composite numbers fell from 84% (March 2023) to 51% (June 2023).”
Reasoning
E1 describes GPT-4's release as creating "a wave of immense hype," which covers the buzz stage. Outside academics later measured GPT-4's behaviour in 2023 and found that its accuracy on a prime-number task fell from 84% to 51% between March and June (E4), which documents unreliability. However, the stages overlapped rather than following one another: the launch-day expert reaction already said GPT-4 doesn't solve hallucinations (E2), and OpenAI's own release page said it "still is not fully reliable" (E3). Because all three stages happened but recognition of the problems came at launch rather than only after scrutiny, the criteria point to mostly true.
Cite this claim
Gary Marcus, “Marcus suspected that GPT-4's release would follow a pattern of immense buzz, then scientific scrutiny, then recognition that many problems remain.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_bfa4544a48b1, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_bfa4544a48b1
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.