Anthropic wants to evaluate every new model to see whether it is getting close to dangerous (CBRN) capabilities, and does not think current models yet pose these risks seriously.

Judged as of 2024-11-11. The claim has two parts: (a) Anthropic evaluates each new frontier model it trains or releases for dangerous capabilities, including CBRN; (b) Anthropic's stated assessment at that time was that its current models did not yet pose these risks seriously. True: Anthropic's Responsible Scaling Policy and its model cards or system cards for models released up to November 2024 (for example Claude 3 and Claude 3.5) show both. They must document dangerous-capability or CBRN evaluations for each new frontier model, and they must conclude that the models stayed below the thresholds for serious risk, for example a determination that the ASL-3 threshold was not reached (ASL-2). Mostly true: both parts hold in substance, but evaluations are documented for most rather than literally every new model (for example, a minor update without published evals). Mixed: one part is well supported and the other is not. For example, evaluations are documented, but Anthropic at the time said a model had already crossed a serious-risk threshold; or the risk assessment matches, but evaluations were not routinely run on new models. Mostly false: CBRN evaluations were run only occasionally, or only on some flagship models, and the risk assessment is unclear. False: evidence shows Anthropic did not run such evaluations on its new models in that period, and it had publicly judged its current models to pose serious CBRN risk.

Set before the research began.

1. E1 Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic Own record

   “Based on our analysis, we judge that the upgraded Claude 3.5 Sonnet does not require ASL-3 safeguards at this time.”

2. E2 Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic Own record

   “We also evaluate frontier risks in accordance with our Responsible Scaling Policy (RSP) [4] in the the areas of Chemical, Biological, Radiological, and Nuclear (CBRN) risks, Cybersecurity, and Autonomous Capabilities.”

3. E3 Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic Own record

   “While these models did not trigger the full evaluation protocol described in our Responsible Scaling Policy, we remain committed to regular safety testing of all frontier models.”

4. E4 Responsible Scaling Policy Evaluations Report – Claude 3 Opus Anthropic Own record

   “We did not reach any Yellow Line indicators during this round of testing, and will re-run evaluations on our most capable model every three months to account for improvements in capability elicitation methods.”

Anthropic's RSP evaluations report for Claude 3 Opus says no Yellow Line indicators were reached, and it commits to re-running evaluations on the most capable model periodically (E4). The model card addendum for Claude 3.5 Haiku and the upgraded Claude 3.5 Sonnet says frontier risks, including CBRN, were evaluated under the RSP (E2). It concludes that the upgraded Sonnet does not require ASL-3 safeguards (E1), which supports the risk-assessment part. However, the addendum says these models did not trigger the full RSP evaluation protocol (E3), and none of the evidence documents evaluations for every model, such as the original Claude 3.5 Sonnet, so the evidence supports "most" rather than literally "every" new model.