Claims

Prediction · AI capabilities

“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”

Gary Marcus, author · 27 July 2025 · What to Expect When You're Expecting … GPT-5 (Marcus on AI)

GPT-5 will still make a significant number of obvious errors that are hard to predict.

Resolved true The record shows it happened or was so.

What would count as true or false

This prediction is judged on GPT-5 as released (August 2025) and used through the end of 2025. TRUE: after release, many obvious, elementary errors by GPT-5 were documented by users, journalists or evaluators, and they occurred unpredictably on simple tasks. Examples include basic counting, spelling, simple arithmetic or logic, mislabeled maps or diagrams, and confident factual blunders. MOSTLY TRUE: such errors were documented but described as notably less frequent than in earlier models, though still a recognized issue. MOSTLY FALSE: only a few isolated obvious errors were reported, and evaluations found them rare. FALSE: credible evaluations found GPT-5 essentially free of such simple errors, or the errors were too rare to count as significant.

Evidence

1. GPT-5 is a joke. Will it matter? Blood in the Machine (Substack newsletter) · 12 August 2025

   “GPT-5 couldn’t count the number of ‘b’s in blueberry, couldn’t identify how many fingers were on a picture of a human hand, got basic arithmetic wrong, and so on.”

2. OpenAI Faces Backlash for Retiring Older Models With GPT-5 Launch PCMag · 8 August 2025

   “GPT-5 is rolling out now to paid and free users. But despite the intelligence upgrade, some have spotted the AI model still making boneheaded mistakes. For example, it can give the wrong answer when you ask how many bs are in the word blueberry.”

3. Sam Altman set the hype, but did GPT-5 deliver? Social Samosa · 18 August 2025

   “The correct answer, x = 0.79, requires nothing more than basic decimal subtraction. Yet GPT-5 initially responded with x = -0.21.”

4. OpenAI's GPT-5 rollout faces user backlash | VentureBeat VentureBeat · 11 August 2025

   “Altman and others at OpenAI claimed the “autoswitcher” went offline “for a chunk of the day,” making the model seem “way dumber” than intended.”

Reasoning

Independent outlets reported obvious errors by GPT-5 on simple tasks soon after its release. PCMag [E2] reported that it miscounted the letter 'b' in "blueberry". Blood in the Machine [E1] described failures at letter counting, counting fingers in a picture of a hand, and basic arithmetic. Social Samosa [E3] documented a wrong initial answer to a basic decimal equation, which is an error on a simple task. OpenAI's explanation that a failed routing system made the model seem dumber at launch [E4] is a partial caveat, but it does not show the errors were rare, and no evidence suggests they were notably less frequent than in earlier models.

Cite this claim

Gary Marcus, “GPT-5 will still make a significant number of obvious errors that are hard to predict.” (said 27 July 2025). Verdict: resolved true. Reliability API, claim c_2cdf0a9efee2, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_2cdf0a9efee2