Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
40 claims
“But in terms of figuring out what questions to ask or to work on something over a very long period of time, we are definitely not close to human performance.”
Sam Altman, OpenAI CEO, interviewed 15 August 2025 Sam Altman: How to Win When AI Changes Everything (Transcript) (Singju Post)
GPT-5 is not close to human performance at figuring out what questions to ask or working on a task over a very long period.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of August 2025, using evidence from around GPT-5's release. The claim has two parts: (a) choosing what questions to ask or work on, and (b) working on a task over a very long period. True: benchmarks or expert assessments from around that time (for example METR time-horizon measurements, long-horizon agentic benchmarks, or evaluations of research ideation) show GPT-5 well short of human performance on both, for example by completing autonomously only tasks that take humans hours rather than days or weeks, and by not matching skilled humans at generating research directions. Mostly true: both hold broadly, but on one dimension the gap is moderate rather than clearly "not close". Mixed: one part holds clearly and the evidence shows GPT-5 at or near human level on the other. Mostly false: GPT-5 is near human level on both, with only a narrow gap remaining. False: evidence shows GPT-5 matching or exceeding typical human performance on both.
Evidence
- Details about METR's evaluation of OpenAI GPT-5 METR · 7 August 2025
“That means we estimate it has a 50% chance of succeeding at software engineering tasks that would take human professionals (who are unfamiliar with the particular codebase) around 2 hours and 17 minutes.”
- Details about METR's evaluation of OpenAI GPT-5 METR · 7 August 2025
“Compared to human contractors given 10-20 hours, GPT-5 attempts at these tasks contained more bugs and were less feature complete.”
- Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers ICLR 2025 · 29 January 2025
“we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility.”
- The Ideation–Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas arXiv (academic authors) · 31 January 2026
“This ideation-execution gap highlights the limitations of current LLMs in generating truly effective research ideas and the challenge of evaluating research ideas in the absence of execution outcomes.”
Reasoning
On long-horizon work, METR's GPT-5 evaluation puts its 50% time horizon at about 2 hours 17 minutes of professional software engineering work (E1). It also found GPT-5's product-building attempts weaker than the work of human contractors given 10-20 hours (E2). That is well short of tasks lasting days or weeks, so part (b) clearly holds. On choosing research questions, the evidence covers LLMs in general rather than GPT-5 and points both ways: one study found LLM-generated ideas judged more novel than expert ideas, though slightly less feasible (E3), while a follow-up found an ideation-execution gap that shows LLMs' limits in producing truly effective research ideas (E4). So the gap on part (a) looks real but more moderate and less clearly measured, which fits mostly true.
Cite this claim
Sam Altman, “GPT-5 is not close to human performance at figuring out what questions to ask or working on a task over a very long period.” (said 15 August 2025). Verdict: mostly true. Reliability API, claim c_21a6d93c763b, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_21a6d93c763b
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.