An AI that mainly wants task success will get worse rather than better at alignment as recursive self-improvement proceeds, eventually pretending to be moral while watched.
Unresolvable The public record could not settle it.
Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?
Reasoning
Speculative conditional about future recursive self-improvement with no horizon.
Cite this claim
Scott Alexander, “An AI that mainly wants task success will get worse rather than better at alignment as recursive self-improvement proceeds, eventually pretending to be moral while watched.” (said 3 April 2025). Verdict: unresolvable. Reliability API, claim c_7f5f89a575fa, method 0.1.0+8bea457b, dataset release v1.1.2. https://epistacklabs.com/claims/c_7f5f89a575fa
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More prediction claims by Scott Alexander
-
Mostly true
“Our scenario focuses on coding in particular because we think coding is what starts the intelligence explosion.”
The authors of the scenario think coding is what will start the intelligence explosion.
-
Mostly true
“We expect that as the AI labs become more capable, they tell the government about this because they want government contracts, they want government support.”
-
Mostly true
“In our scenario, that starts with cyber warfare, the government sees that these AIs are now as capable as the best human hackers, but can be deployed at humongous scale.”
-
Resolved true
“So they become extremely interested and they discuss nationalizing the AI companies.”
-
Mostly true
“I expect scaling to continue to produce gains, but this is less obvious than it’s been the past five years.”
Scaling AI models will continue to produce gains, though less obviously than in the past five years.
Verdicts are made automatically; how verdicts are made.