Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 78 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 2 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 83 (likely 78 to 88, high confidence), from 86 claims: 86 factual claims. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Daniela Amodei's claims
43 claims
“We're doing scaling laws research to try and predict empirically what properties are we going to see emerge in these language models at various sizes?”
Daniela Amodei, Anthropic president 2022 Podcast Transcript: Daniela and Dario Amodei on Anthropic (LessWrong)
Anthropic does scaling laws research to predict empirically what properties emerge in language models at various sizes.
Resolved true The record shows it happened or was so.
The second judge said mostly true; the verdict shown is the first judge's.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
True if, as of 2022, Anthropic was doing scaling-laws research aimed at empirically predicting how language-model properties or behaviours change with model size, as shown by papers such as 'Predictability and Surprise in Large Generative Models' (2022) or similar work. Mostly true if the research existed but focused mainly on predicting loss or performance rather than emergent properties more broadly. Mixed if such research was only marginal. Mostly false if Anthropic's scaling work did not aim at prediction. False if Anthropic did no such research at that time.
Evidence
- Predictability and surprise in large generative models Anthropic · 15 February 2022 Own record
“Namely, these generative models have an unusual combination of predictable loss on a broad training distribution (as embodied in their "scaling laws"), and unpredictable specific capabilities, inputs, and outputs.”
- Anthropic's core views on AI safety Anthropic · 8 March 2023 Own record
“In the best case, we aim to build detailed quantitative models of how these tendencies vary with scale so that we can anticipate the sudden emergence of dangerous failure modes in advance.”
- Anthropic's core views on AI safety Anthropic · 8 March 2023 Own record
“In 2019, several members of what was to become the founding Anthropic team made this idea precise by developing scaling laws for AI, demonstrating that you could make AIs smarter in a predictable way, just by making them larger and training them on more data.”
Reasoning
E1 is Anthropic's published February 2022 paper 'Predictability and Surprise in Large Generative Models', which the resolution criteria name as an example of qualifying work. Its quoted text analyses how models scale, contrasting the predictable loss described by scaling laws with specific capabilities that are hard to predict. E3 says that people who went on to found Anthropic developed scaling laws in 2019. E2 (2023) states the aim of modelling how model tendencies change with scale so that emerging behaviours can be anticipated. All of this evidence comes from Anthropic itself, but E1 is the published research, so it counts as an official record that Anthropic was doing this work in 2022.
Cite this claim
Daniela Amodei, “Anthropic does scaling laws research to predict empirically what properties emerge in language models at various sizes.” (said 2022). Verdict: resolved true. Reliability API, claim c_e2a929ea57d6, method 0.1.0+28da144b. https://epistacklabs.com/claims/c_e2a929ea57d6
Claims without a verdict
Is a verdict here wrong, or do you speak for Daniela Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.