Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
Gary Marcus, author 27 July 2025 What to Expect When You're Expecting … GPT-5 (Marcus on AI)
GPT-5 will still make a significant number of obvious errors that are hard to predict.
Resolved true The record shows it happened or was so.
What would count as true or false
This prediction is judged on GPT-5 as released (August 2025) and used through the end of 2025. TRUE: after release, many obvious, elementary errors by GPT-5 were documented by users, journalists or evaluators, and they occurred unpredictably on simple tasks. Examples include basic counting, spelling, simple arithmetic or logic, mislabeled maps or diagrams, and confident factual blunders. MOSTLY TRUE: such errors were documented but described as notably less frequent than in earlier models, though still a recognized issue. MOSTLY FALSE: only a few isolated obvious errors were reported, and evaluations found them rare. FALSE: credible evaluations found GPT-5 essentially free of such simple errors, or the errors were too rare to count as significant.
Evidence
- GPT-5 is a joke. Will it matter? Blood in the Machine (Substack newsletter) · 12 August 2025
“GPT-5 couldn’t count the number of ‘b’s in blueberry, couldn’t identify how many fingers were on a picture of a human hand, got basic arithmetic wrong, and so on.”
- OpenAI Faces Backlash for Retiring Older Models With GPT-5 Launch PCMag · 8 August 2025
“GPT-5 is rolling out now to paid and free users. But despite the intelligence upgrade, some have spotted the AI model still making boneheaded mistakes. For example, it can give the wrong answer when you ask how many bs are in the word blueberry.”
- Sam Altman set the hype, but did GPT-5 deliver? Social Samosa · 18 August 2025
“The correct answer, x = 0.79, requires nothing more than basic decimal subtraction. Yet GPT-5 initially responded with x = -0.21.”
- OpenAI's GPT-5 rollout faces user backlash | VentureBeat VentureBeat · 11 August 2025
“Altman and others at OpenAI claimed the “autoswitcher” went offline “for a chunk of the day,” making the model seem “way dumber” than intended.”
Reasoning
Independent outlets reported obvious errors by GPT-5 on simple tasks soon after its release. PCMag [E2] reported that it miscounted the letter 'b' in "blueberry". Blood in the Machine [E1] described failures at letter counting, counting fingers in a picture of a hand, and basic arithmetic. Social Samosa [E3] documented a wrong initial answer to a basic decimal equation, which is an error on a simple task. OpenAI's explanation that a failed routing system made the model seem dumber at launch [E4] is a partial caveat, but it does not show the errors were rare, and no evidence suggests they were notably less frequent than in earlier models.
Cite this claim
Gary Marcus, “GPT-5 will still make a significant number of obvious errors that are hard to predict.” (said 27 July 2025). Verdict: resolved true. Reliability API, claim c_2cdf0a9efee2, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_2cdf0a9efee2
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.