There is no known way to use loss functions, sensory inputs or reward inputs to make a cognitive system's optimization point at particular latent things in the environment.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2022-06-10. True: as of that date, the alignment/ML literature had no demonstrated, accepted method that uses loss functions, sensory inputs or reward signals to reliably make a cognitive system's internal optimization target specific latent features of the environment (rather than proxies such as sensory patterns or reward signals), and expert surveys or reviews from the time describe this (inner alignment / goal misgeneralization / pointing at latent variables) as an open problem. Mostly true: the problem was broadly recognised as unsolved, but there were partial or narrow demonstrations (e.g. limited interpretability-verified cases) that the absolute 'no known way' slightly overstates. Mixed: experts were substantially divided, with credible published work claiming partial solutions that a significant part of the field accepted. Mostly false: there were recognised methods that did this in meaningful settings, and the claim rests only on the lack of guarantees for general or superhuman systems. False: a published, widely accepted method that reliably did this existed by mid-2022.
Set before the research began.
Evidence (4)
-
E1 Risks from Learned Optimization in Advanced Machine Learning Systems arXiv (ar5iv rendering) · 11 June 2019
“While this seems beyond current meta-learning techniques, the basic principle is similar to existing meta-learners designed to optimize for robustness.”
-
E2 Risks from Learned Optimization in Advanced Machine Learning Systems arXiv (ar5iv rendering) · 11 June 2019
“This is a critical concern because it makes every other problem of pseudo-alignment worse—it is a reason that, in general, it is hard to find robustly aligned mesa-optimizers.”
-
E3 LESSWRONG LessWrong (wiki) Own record
“This means that a reward button, or in general an agent with goals or training updates that are simple functions of its direct inputs, will not be scalable as an alignment method for sufficiently advanced agents.”
-
E4 Reward Function Design: a starter pack — AI Alignment Forum AI Alignment Forum · 8 December 2025
“I claim that even these clear-cut cases constitute an unsolved technical problem, so I’ll focus on those.”
Reasoning
The 2019 "Risks from Learned Optimization" paper describes it as hard in general to find robustly aligned mesa-optimizers (E2), and says a proposed approach seemed beyond the meta-learning techniques of the time (E1). Both treat getting a system's internal objective to match the intended target as an open problem. A 2025 Alignment Forum post still calls even clear-cut reward-design cases an unsolved technical problem (E4), so the problem had not been solved by mid-2022. None of the evidence points to a demonstrated, accepted method before 2022-06-10. E3 is not independent of the subject and is not relied on.
Cite this claim
Eliezer Yudkowsky, “There is no known way to use loss functions, sensory inputs or reward inputs to make a cognitive system's optimization point at particular latent things in the environment.” (said 10 June 2022). Verdict: resolved true. Reliability API, claim c_90b514609a60, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_90b514609a60
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Eliezer Yudkowsky
-
Mixed
“I was observing that obvious-seeming hypothesis myself, sometime around the time of GPT 3.5.”
Yudkowsky himself observed the persona-selection-like hypothesis around the time of GPT 3.5.
-
Mostly true
“The test-passing preference stands on its own once formed.”
-
Mostly true
“LLMs surely have neural circuits they learned to model text streams from fictional and nonfictional personas that are not lying, deceiving, cheating.”
-
Mostly true
“The predict-as-if-honest circuitry is contextual, invoked to predict some streams of text but not others.”
-
Mostly true
“Presumably, somewhere along the way LLMs learn a dispreference for tests that don't pass, independently of how that gets prevented or whether the new plan does what the user really wanted.”
Verdicts are made automatically; how verdicts are made.