Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
49 claims
“So we did this experiment where we basically train the model to be all the good thing, helpful, honest, harmless, friendly.”
Dario Amodei, Anthropic CEO, interviewed 28 February 2025 Hard Fork - Anthropic's C.E.O. Dario Amodei on Surviving the A.I. Endgame Transcript (Podscripts (Hard Fork))
Anthropic ran an experiment training a model to be helpful, honest, harmless and friendly.
Mostly true True in substance, wrong in a detail.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2025-02-28. True: Anthropic ran and documented (by that date) an experiment in which a model was trained to be helpful, honest and harmless (e.g. Claude trained with HHH or constitutional methods), which was then studied, such as the 2024 'Alignment faking in large language models' study, or similar work. Mostly true: such an experiment exists, but the training goals were described somewhat differently (e.g. HHH without 'friendly'), or the training came before the experiment rather than being part of it. Mixed: Anthropic trained HHH models but no specific experiment matching the description can be identified. Mostly false: the experiment described was run by others, or concerned different training goals. False: there is no Anthropic training of models toward these traits.
Evidence
- Alignment faking in large language models Anthropic · 18 December 2024 Own record
“In the paper, we studied a model (specifically Claude 3 Opus, though we also ran some experiments with other models, including the June 2024 release of Claude 3.5 Sonnet) that had been trained to be helpful, honest, and harmless.”
- Alignment faking in large language models Anthropic · 18 December 2024 Own record
“To test this, instead of merely telling the model that we were training it to always comply with harmful queries, we actually did train it in this way.”
- Constitutional AI: Harmlessness from AI feedback Anthropic · 15 December 2022 Own record
“We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs.”
Reasoning
Anthropic's December 2024 alignment-faking paper describes studying Claude 3 Opus, a model "trained to be helpful, honest, and harmless" (E1). That matches the kind of experiment the criteria name, so it was documented before 2025-02-28. However, the HHH training came before the experiment rather than being part of it, the description does not include "friendly", and the training actually done in the experiment was towards complying with harmful queries (E2). That matches the "mostly true" criterion. Anthropic's 2022 Constitutional AI work also documents experiments in training a harmless assistant (E3).
Cite this claim
Dario Amodei, “Anthropic ran an experiment training a model to be helpful, honest, harmless and friendly.” (said 28 February 2025). Verdict: mostly true. Reliability API, claim c_53db116f9951, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_53db116f9951
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.