GPT-5 will still make a significant number of obvious errors that are hard to predict.

This prediction is judged on GPT-5 as released (August 2025) and used through the end of 2025. TRUE: after release, many obvious, elementary errors by GPT-5 were documented by users, journalists or evaluators, and they occurred unpredictably on simple tasks. Examples include basic counting, spelling, simple arithmetic or logic, mislabeled maps or diagrams, and confident factual blunders. MOSTLY TRUE: such errors were documented but described as notably less frequent than in earlier models, though still a recognized issue. MOSTLY FALSE: only a few isolated obvious errors were reported, and evaluations found them rare. FALSE: credible evaluations found GPT-5 essentially free of such simple errors, or the errors were too rare to count as significant.

Set before the research began.

1. E1 GPT-5 is a joke. Will it matter? Blood in the Machine (Substack newsletter) · 12 August 2025

   “GPT-5 couldn’t count the number of ‘b’s in blueberry, couldn’t identify how many fingers were on a picture of a human hand, got basic arithmetic wrong, and so on.”

2. E2 OpenAI Faces Backlash for Retiring Older Models With GPT-5 Launch PCMag · 8 August 2025

   “GPT-5 is rolling out now to paid and free users. But despite the intelligence upgrade, some have spotted the AI model still making boneheaded mistakes. For example, it can give the wrong answer when you ask how many bs are in the word blueberry.”

3. E3 Sam Altman set the hype, but did GPT-5 deliver? Social Samosa · 18 August 2025

   “The correct answer, x = 0.79, requires nothing more than basic decimal subtraction. Yet GPT-5 initially responded with x = -0.21.”

4. E4 OpenAI's GPT-5 rollout faces user backlash | VentureBeat VentureBeat · 11 August 2025

   “Altman and others at OpenAI claimed the “autoswitcher” went offline “for a chunk of the day,” making the model seem “way dumber” than intended.”

Independent outlets reported obvious errors by GPT-5 on simple tasks soon after its release. PCMag [E2] reported that it miscounted the letter 'b' in "blueberry". Blood in the Machine [E1] described failures at letter counting, counting fingers in a picture of a hand, and basic arithmetic. Social Samosa [E3] documented a wrong initial answer to a basic decimal equation, which is an error on a simple task. OpenAI's explanation that a failed routing system made the model seem dumber at launch [E4] is a partial caveat, but it does not show the errors were rare, and no evidence suggests they were notably less frequent than in earlier models.