When inabilities are trained into a system that already has capabilities, gradient descent learns only shallow patches, and the deep capabilities come back out when prompted differently, e.g. in another language.
Not researched Checkable, but outside the sample that was researched.
Factual: Statements about the present or the past, checked against the record: did they hold up?
Cite this claim
Eliezer Yudkowsky, “When inabilities are trained into a system that already has capabilities, gradient descent learns only shallow patches, and the deep capabilities come back out when prompted differently, e.g. in another language.” (said 30 March 2023). Verdict: not researched. Reliability API, claim c_a8cc4a83010c, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_a8cc4a83010c
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Eliezer Yudkowsky
-
Mixed
“I was observing that obvious-seeming hypothesis myself, sometime around the time of GPT 3.5.”
Yudkowsky himself observed the persona-selection-like hypothesis around the time of GPT 3.5.
-
Mostly true
“The test-passing preference stands on its own once formed.”
-
Mostly true
“LLMs surely have neural circuits they learned to model text streams from fictional and nonfictional personas that are not lying, deceiving, cheating.”
-
Mostly true
“The predict-as-if-honest circuitry is contextual, invoked to predict some streams of text but not others.”
-
Mostly true
“Presumably, somewhere along the way LLMs learn a dispreference for tests that don't pass, independently of how that gets prevented or whether the new plan does what the user really wanted.”
Verdicts are made automatically; how verdicts are made.