Amodei said Anthropic "kind of" already does reinforcement learning, through RLHF.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
True: Anthropic's publications or official statements as of August 2023 show that it trained its language models with reinforcement learning from human feedback (RLHF) or its RL-based variants (e.g. RLAIF/Constitutional AI), which involve reinforcement learning. Mostly true: RL was used, but by a variant described slightly differently. Mostly false: RLHF-style methods were used only in research and not on its models. False: Anthropic did not use RL-based fine-tuning on its language models.
Set before the research began.
Evidence (4)
-
E1 Model Card and Evaluations for Claude Models - Anthropic Anthropic Own record
“They use a transformer architecture and are trained via unsupervised learning, RLHF, and Constitutional AI (including both”
-
E2 Claude's constitution Anthropic · 9 May 2023 Own record
“During the second phase, a model is trained via reinforcement learning, but rather than using human feedback, it uses AI-generated feedback based on the set of principles to choose the more harmless output.”
-
E3 Training a helpful and harmless assistant with RLHF Anthropic · 12 April 2022 Own record
“We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants.”
-
E4 Constitutional AI: Harmlessness from AI feedback Anthropic · 15 December 2022 Own record
“In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF).”
Reasoning
E2 is Anthropic's 2023-05-09 post on Claude's constitution, which predates the claim. It says that in the second training phase a model is trained with reinforcement learning using AI-generated feedback. E3 (April 2022) and E4 (December 2022) are Anthropic's own research publications describing RLHF and RL from AI feedback applied to its language models. E1, Anthropic's model card for its Claude models, says they are trained via RLHF and Constitutional AI, but it is undated. All of these are Anthropic's own official publications rather than independent sources, and they consistently show RL-based fine-tuning of its models before August 2023.
Cite this claim
Dario Amodei, “Amodei said Anthropic "kind of" already does reinforcement learning, through RLHF.” (said 8 August 2023). Verdict: resolved true. Reliability API, claim c_9def3a5c157d, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_9def3a5c157d
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.