Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“GPT-4 will still, like its predecessors, be a bull in a china shop, reckless and hard to control.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4 will still be reckless and hard to control, making a significant number of stupid errors.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
A prediction about GPT-4 once released (March 2023); judge it at roughly one year after release (2024-03). The claim is that GPT-4, like its predecessors, would be reckless and hard to control and would make a significant number of stupid errors. True: evaluations and documented use show GPT-4 still frequently produced confident errors and hallucinations, was jailbroken or behaved unpredictably despite controls, and committed clear basic blunders. OpenAI's own GPT-4 technical report or system card acknowledging such limitations counts. Mostly true: errors and controllability problems persist and are well documented, but are clearly reduced compared with predecessors, so 'still like its predecessors' is somewhat overstated. Mixed: errors persist but controllability is reported as largely improved, or the reverse. Mostly false: there are only occasional errors and GPT-4 is broadly reported as reliable and controllable. False: GPT-4 is shown to make few basic errors and to be reliably controllable.
Evidence
- GPT-4 System Card - cdn.openai.com OpenAI · 13 May 2025
“As noted above in 2.2, despite GPT-4’s capabilities, it maintains a tendency to make up facts, to double-down on incorrect information, and to perform tasks incorrectly.”
- GPT-4 System Card - cdn.openai.com OpenAI · 13 May 2025
“OpenAI has implemented various safety measures and processes throughout the GPT-4 development and deployment process that have reduced its ability to generate harmful content. However, GPT-4 can still be vulnerable to adversarial attacks and exploits or, “jailbreaks,” and harmful content is not the source of risk.”
- Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis Journal of Medical Internet Research · 22 May 2024
“The hallucination rates were, respectively, 39.6% (55/139), 28.6% (34/119), and 91.4% (95/104) for GPT-3.5, GPT-4, and Bard (*P*<.001).”
- GPT-4’s successes, and GPT-4’s failures Gary Marcus (Substack) · 15 March 2023 Own record
“And when GPT-4 is integrated in Bing (which incorporates current search results), hallucination remains rampant, e.g., in this example from last week when Bing apparently was already quietly using GPT-4 in the background.”
Reasoning
OpenAI's GPT-4 System Card says GPT-4 keeps a tendency to make up facts, double down on incorrect information and perform tasks incorrectly (E1). It also says GPT-4 can still be jailbroken (E2), which the criteria count toward the claim. However, the same card says safety measures reduced GPT-4's ability to generate harmful content (E2). A JMIR study found a 28.6% hallucination rate for GPT-4, still high but lower than GPT-3.5's 39.6% (E3). So the errors and control problems persisted and are well documented, though one study measured a lower hallucination rate for GPT-4 than for GPT-3.5 [E3] and OpenAI reports reduced ability to generate harmful content [E2], which makes 'still, like its predecessors' somewhat overstated.
Cite this claim
Gary Marcus, “GPT-4 will still be reckless and hard to control, making a significant number of stupid errors.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_151a009cf527, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_151a009cf527
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.