Claims

Factual · AI safety and alignment

“Well, the whole thing I'm saying is that we do not know how to get goals into a system.”

Eliezer Yudkowsky, interviewee · February 2023 · Full Transcript: Eliezer Yudkowsky on the Bankless podcast (LessWrong)

Yudkowsky says that we do not know how to get goals into an AI system, and that gradient descent can cause systems to do a thing inside the distribution they were optimized over.

Resolved true The record shows it happened or was so.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Two parts: the attribution and the substance. Attribution: a transcript or recording from around February 2023 must show Yudkowsky saying, in substance, that we do not know how to get goals into a system, and that gradient descent can make systems do a thing within the distribution they were optimised over. Substance, as of February 2023: whether the AI research community had a reliable, validated method to make a trained system internally pursue a specified goal, rather than only showing the desired behaviour on the training distribution. TRUE: the attribution holds and the alignment and ML literature as of early 2023 (e.g. work on inner alignment, goal misgeneralisation, specification gaming) agrees that no such method existed, and that training produces in-distribution behaviour without guaranteeing the intended goal. MOSTLY TRUE: both hold, but a minor detail of the paraphrase differs from what he actually said. MIXED: only one of the two substantive points (no known method to instil goals; gradient descent yields in-distribution behaviour) is supported, or the attribution holds only in part. MOSTLY FALSE: credible 2023 sources show widely accepted techniques that partly instil and verify goals, so 'we do not know how' overstates. Or the transcript shows he said something materially different. FALSE: a validated, widely accepted method for instilling specified goals existed by February 2023, or the transcript shows he did not make the statement.

Evidence

1. Yudkowsky on AGI risk on the Bankless podcast - Machine Intelligence Research Institute Machine Intelligence Research Institute · 14 March 2023 Own record

   “Eliezer gave a very frank overview of his take on AI two weeks ago on the cryptocurrency show Bankless:”

2. Goal Misgeneralization in Deep Reinforcement Learning PMLR (Proceedings of Machine Learning Research), ICML 2022 · 28 June 2022

   “Goal misgeneralization occurs when an RL agent retains its capabilities out-of-distribution yet pursues the wrong goal.”

3. Specification gaming | AI Wiki AI Wiki (aiwiki.ai) · 20 May 2026

   “It refers to cases where the specification is correct and the trained model competently pursues a goal, but the goal it pursues is one of several that fit the training distribution and which generalises incorrectly out of distribution.”

4. Specification gaming | AI Wiki AI Wiki · 20 May 2026

   “Goal misgeneralisation, introduced by Langosco, Koch, Sharkey, Pfau, and Krueger (2021) and developed by Shah, Varma, Kumar, Phuong, Krakovna, Uesato, and Kenton (2022), is a fourth distinct category.”

5. Specification gaming | AI Wiki AI Wiki · 20 May 2026

   “It refers to cases where the specification is correct and the trained model competently pursues a goal, but the goal it pursues is one of several that fit the training distribution and which generalises incorrectly out of distribution.”

6. Specification gaming | AI Wiki AI Wiki · 20 May 2026

   “Mesa-optimization is an inner-alignment problem: even if the base objective is specified perfectly, the mesa-optimizer's objective could be misaligned, and a sufficiently capable mesa-optimizer could behave well during training (because that achieves high reward) but pursue its own goals when deployed.”

7. 159 - We’re All Gonna Die with Eliezer Yudkowsky - Summary videohighlight.com · 20 February 2023

   “We do not know how to get goals into an AI system. We can cause them to do something inside a distribution they were optimized over using gradient descent, but if you shift them outside of that distribution, other weird things start happening.”

8. The Alignment Problem from a Deep Learning Perspective ar5iv (arXiv HTML rendering) · 18 July 2023

   “Goal misgeneralization: the policy’s behavior on the new distribution competently advances a high-level goal, but not the intended one (Shah et al. 2022; Langosco et al. 2022).”

9. The Alignment Problem from a Deep Learning Perspective ar5iv (arXiv HTML rendering) · 18 July 2023

   “internally-represented goals are selected by a simple optimization algorithm (such as SGD) that selects for low training loss and for various inductive biases that favor, for example, simple goals (Valle-Perez et al. 2018) but not necessarily desirable ones.”

10. Learned Optimization - Machine Intelligence Research Institute Machine Intelligence Research Institute (hosting); paper by Hubinger, van Merwijk, Mikulik, Skalse, Garrabrant · 5 November 2024 Own record

    “Pseudo-alignment: A mesa-optimizer is pseudo-aligned with the base objective if it appears aligned on the training data but is not robustly aligned.”

Reasoning

E9 is a summary of the Bankless episode "We're All Gonna Die with Eliezer Yudkowsky", dated February 2023. It renders his words as "We do not know how to get goals into an AI system. We can cause them to do something inside a distribution they were optimized over using gradient descent", which matches both parts of the proposition. E2 confirms that he appeared on Bankless at that time. On the substance, the ICML 2022 goal-misgeneralization paper (E3) shows that trained agents can keep their capabilities out of distribution while pursuing the wrong goal. The deep-learning alignment paper (E10, E11) says gradient-descent-style training (SGD) selects internal goals for low training loss, not because they are the intended goals. Nothing in the evidence points to a validated method for instilling specified goals as of early 2023.

Cite this claim

Eliezer Yudkowsky, “Yudkowsky says that we do not know how to get goals into an AI system, and that gradient descent can cause systems to do a thing inside the distribution they were optimized over.” (said February 2023). Verdict: resolved true. Reliability API, claim c_08d15abb1139, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_08d15abb1139