Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 72 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
70 62 78 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 2 promises resolved, 5 needed for a score.
All types combined: 75 (likely 69 to 80, high confidence), from 79 claims: 35 factual claims, 44 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Scott Alexander's claims
26 claims
“I expect scaling to continue to produce gains, but this is less obvious than it’s been the past five years.”
Scott Alexander, blogger 20 February 2023 Grading My 2018 Predictions For 2023 (Astral Codex Ten)
Scaling AI models will continue to produce gains, though less obviously than in the past five years.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
The claim has two parts. (a) Scaling continued to produce gains: from February 2023 onward, larger models or more training compute kept giving measurable capability improvements, shown in benchmark results, technical reports or published scaling-law studies of frontier models released after the claim. (b) The gains were less obvious than in 2018-2023: credible sources (researcher statements, analyses, press coverage of diminishing returns, debate over a pre-training 'wall') show that the returns from scaling became more contested or harder to see than in the GPT-2 to GPT-4 era. True: both (a) and (b) hold. Mostly true: (a) clearly holds and (b) holds only weakly or is mildly disputed. Mixed: one part holds and the other clearly fails. For example, scaling gains were as obvious and undisputed as before, or the gains came mainly from methods other than scale while pure scaling stalled. Mostly false: scaling gave only marginal gains that most expert sources call negligible. False: scaling stopped producing gains or reversed. Judge on evidence from February 2023 up to the time of evaluation.
Evidence
- Trends in AI Epoch AI · 5 February 2026
“Training compute for frontier language models has been growing at 5× per year since 2020.”
- Trends in AI Epoch AI · 5 February 2026
“Following the introduction of the first reasoning models (OpenAI’s o1-mini and o1-preview) in September 2024, the Epoch Capabilities Index (ECI) frontier has advanced linearly by 14 points per year, compared with 6 points per year for non-reasoning models.”
- AI Scaling Wall Wikipedia · 30 September 2026
“One 2025 analysis estimated that each tenfold rise in compute yields only one to two percentage points of MMLU-Pro accuracy.”
- The $19.6 billion pivot: How OpenAI's 2-year struggle to launch GPT-5 revealed that its core AI strategy has stopped working | Fortune Fortune · 25 February 2025
“These media reports indicated that Orion was better at some language tasks than GPT-4, but that its abilities at coding and mathematics was not significantly improved on the previous model.”
Reasoning
Epoch AI reports that frontier training compute kept growing about 5× per year (E1). It also reports that frontier capability, measured by its Capabilities Index, kept rising, by about 6 points per year for non-reasoning models and faster for reasoning models (E2). A 2025 analysis estimated that each tenfold rise in compute still adds one to two points of MMLU-Pro accuracy (E3), so scaling kept producing gains, but small ones. Fortune reports that OpenAI's Orion model was better than GPT-4 at some language tasks but not significantly improved in coding and mathematics (E4). This supports the 'less obvious' part, while E3 shows that added compute still produced measurable, if small, gains, and E2 shows the capability frontier advancing faster after reasoning models were introduced than for non-reasoning models.
Cite this claim
Scott Alexander, “Scaling AI models will continue to produce gains, though less obviously than in the past five years.” (said 20 February 2023). Verdict: mostly true. Reliability API, claim c_6690cfadae41, method 0.1.0+8bea457b. https://epistacklabs.com/claims/c_6690cfadae41
Claims without a verdict
Is a verdict here wrong, or do you speak for Scott Alexander? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.