Scott Alexander said that, at the beginning of their forecast, he and his co-authors think AI developers will start with agency training, expand agent time horizons, and get coding working well.
Unresolvable The public record could not settle it.
Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
This is a three-part prediction about the early part of the AI 2027 scenario: leading AI developers will (1) start with agency training, meaning they make training models to act as autonomous agents (for example reinforcement learning on multi-step agentic tasks) a central priority; (2) expand agent time horizons, meaning the length of tasks agents can complete autonomously grows measurably, for example on METR's time-horizon benchmark or a comparable one; and (3) get coding working well, meaning AI coding agents become reliable enough to be widely used for real software engineering work, as shown by benchmarks such as SWE-bench and by reported adoption. Judge it at the end of 2026. TRUE: by end-2026 the evidence shows all three, with agentic training among the developers' main early focuses. MOSTLY TRUE: all three happened, but one is weaker than described, or the order or emphasis differed slightly (for example coding improved strongly while time-horizon gains were modest but real). MIXED: roughly half held, for example coding agents improved and agentic training became central, but time horizons did not grow measurably. MOSTLY FALSE: only one part held, or the gains were marginal across the board. FALSE: dated evidence shows developers did not pursue agentic training as a priority, time horizons stagnated, and coding agents stayed unreliable for real work.
Set before the research began.
Evidence (4)
-
“From 2019 to 2025 this length doubled about every 7 months; for models since 2024 METR's own estimate is about 3 months.”
-
E2 Frontier Risk Report (February to March 2026) METR · 19 May 2026
“| Evidence source | Public frontier (Feb–Mar 2026) | Internal frontier (Feb–Mar 2026) | |---|---|---| | Central capabilities results | | | | Time horizon 1.1 50% | ~12h [5h-61h] | Likely* ≥16h, internal frontier on average ~2 months ahead |”
-
E3 AI Coding Agents: Adoption Trends - The JetBrains Blog JetBrains Research · 19 August 2026
“As of May–July 2026, 90% of professional developers were using AI coding agents at work at least weekly in one form or another (local agents or remote cloud agents), with 68% using them daily.”
-
E4 Frontier Risk Report (February to March 2026) METR · 19 May 2026
“METR’s latest RCT of open source developers using late 2025 public agents found small (~4-20%) productivity benefits, but we expect this to be an underestimate due to selection effects”
Reasoning
The evidence shows measurable growth in agent time horizons (E1, E2) and wide use of AI coding agents for work (E3), but no item documents that developers made agentic training a central priority, which the criteria require for a mostly-true verdict. The evidence therefore does not settle whether all three parts held.
Cite this claim
Scott Alexander, “Scott Alexander said that, at the beginning of their forecast, he and his co-authors think AI developers will start with agency training, expand agent time horizons, and get coding working well.” (said 3 April 2025). Verdict: unresolvable. Reliability API, claim c_669c1570c2ab, method 0.1.0+8bea457b, dataset release v1.1.2. https://epistacklabs.com/claims/c_669c1570c2ab
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More prediction claims by Scott Alexander
-
Mostly true
“Our scenario focuses on coding in particular because we think coding is what starts the intelligence explosion.”
The authors of the scenario think coding is what will start the intelligence explosion.
-
Mostly true
“I expect scaling to continue to produce gains, but this is less obvious than it’s been the past five years.”
Scaling AI models will continue to produce gains, though less obviously than in the past five years.
-
Mostly true
“We expect that as the AI labs become more capable, they tell the government about this because they want government contracts, they want government support.”
-
Mostly true
“In our scenario, that starts with cyber warfare, the government sees that these AIs are now as capable as the best human hackers, but can be deployed at humongous scale.”
-
Resolved true
“So they become extremely interested and they discuss nationalizing the AI companies.”
Verdicts are made automatically; how verdicts are made.