ClaimsScott Alexander

“I expect scaling to continue to produce gains, but this is less obvious than it’s been the past five years.”

Scott Alexander, blogger 20 February 2023 Grading My 2018 Predictions For 2023 (Astral Codex Ten) archived copy

Prediction AI capabilities

Scaling AI models will continue to produce gains, though less obviously than in the past five years.

Mostly true True in substance, wrong in a detail.

Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?

The second judge said resolved true; the verdict shown is the first judge's.

What would count as true or false

The claim has two parts. (a) Scaling continued to produce gains: from February 2023 onward, larger models or more training compute kept giving measurable capability improvements, shown in benchmark results, technical reports or published scaling-law studies of frontier models released after the claim. (b) The gains were less obvious than in 2018-2023: credible sources (researcher statements, analyses, press coverage of diminishing returns, debate over a pre-training 'wall') show that the returns from scaling became more contested or harder to see than in the GPT-2 to GPT-4 era. True: both (a) and (b) hold. Mostly true: (a) clearly holds and (b) holds only weakly or is mildly disputed. Mixed: one part holds and the other clearly fails. For example, scaling gains were as obvious and undisputed as before, or the gains came mainly from methods other than scale while pure scaling stalled. Mostly false: scaling gave only marginal gains that most expert sources call negligible. False: scaling stopped producing gains or reversed. Judge on evidence from February 2023 up to the time of evaluation.

Set before the research began.

Evidence (4)

  1. E1 Trends in AI Epoch AI · 5 February 2026

    “Training compute for frontier language models has been growing at 5× per year since 2020.”
  2. E2 Trends in AI Epoch AI · 5 February 2026

    “Following the introduction of the first reasoning models (OpenAI’s o1-mini and o1-preview) in September 2024, the Epoch Capabilities Index (ECI) frontier has advanced linearly by 14 points per year, compared with 6 points per year for non-reasoning models.”
  3. E3 AI Scaling Wall Wikipedia · 30 September 2026

    “One 2025 analysis estimated that each tenfold rise in compute yields only one to two percentage points of MMLU-Pro accuracy.”
  4. E4 The $19.6 billion pivot: How OpenAI's 2-year struggle to launch GPT-5 revealed that its core AI strategy has stopped working | Fortune Fortune · 25 February 2025

    “These media reports indicated that Orion was better at some language tasks than GPT-4, but that its abilities at coding and mathematics was not significantly improved on the previous model.”

Reasoning

Epoch AI reports that frontier training compute kept growing about 5× per year (E1). It also reports that frontier capability, measured by its Capabilities Index, kept rising, by about 6 points per year for non-reasoning models and faster for reasoning models (E2). A 2025 analysis estimated that each tenfold rise in compute still adds one to two points of MMLU-Pro accuracy (E3), so scaling kept producing gains, but small ones. Fortune reports that OpenAI's Orion model was better than GPT-4 at some language tasks but not significantly improved in coding and mathematics (E4). This supports the 'less obvious' part, while E3 shows that added compute still produced measurable, if small, gains, and E2 shows the capability frontier advancing faster after reasoning models were introduced than for non-reasoning models.

Cite this claim

Scott Alexander, “Scaling AI models will continue to produce gains, though less obviously than in the past five years.” (said 20 February 2023). Verdict: mostly true. Reliability API, claim c_6690cfadae41, method 0.1.0+8bea457b, dataset release v1.1.2. https://epistacklabs.com/claims/c_6690cfadae41

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Scott Alexander, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More prediction claims by Scott Alexander

  1. Mostly true Prediction 3 April 2025
    “Our scenario focuses on coding in particular because we think coding is what starts the intelligence explosion.”

    The authors of the scenario think coding is what will start the intelligence explosion.

  2. Mostly true Prediction 3 April 2025
    “We expect that as the AI labs become more capable, they tell the government about this because they want government contracts, they want government support.”

    In the AI 2027 scenario, as AI labs become more capable they tell the government, seeking government contracts and support.

  3. Mostly true Prediction 3 April 2025
    “In our scenario, that starts with cyber warfare, the government sees that these AIs are now as capable as the best human hackers, but can be deployed at humongous scale.”

    In the AI 2027 scenario, the government sees via cyber warfare that AIs are as capable as the best human hackers but deployable at huge scale.

  4. Resolved true Prediction 3 April 2025
    “So they become extremely interested and they discuss nationalizing the AI companies.”

    In the AI 2027 scenario, the government becomes extremely interested in AI and discusses nationalizing the AI companies.

  5. Resolved true Prediction 20 February 2023
    “Training GPT-4 will cost $100M, which is a lot.”

    Training GPT-4 will cost $100 million.

All of them

Verdicts are made automatically; how verdicts are made.