Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 78 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 2 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 83 (likely 78 to 88, high confidence), from 86 claims: 86 factual claims. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Daniela Amodei's claims
43 claims
“We're doing societal impacts research.”
Daniela Amodei, Anthropic president 2022 Podcast Transcript: Daniela and Dario Amodei on Anthropic (LessWrong)
Anthropic does societal impacts research on the short- and medium-term effects of language models.
Resolved true The record shows it happened or was so.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2022. True: by 2022 Anthropic was doing research on the societal impacts of language models, shown for example by a Societal Impacts team or papers on societal effects or harms (such as 'Predictability and Surprise in Large Generative Models' or related work). The research must be on short- and medium-term effects, as the proposition says. Mostly true: the research was under way but organised informally, or covered mostly one kind of impact. Mixed: there was only incidental work touching on societal impacts. Mostly false: such research had been announced but had not started by 2022. False: the evidence shows Anthropic did no such research in 2022.
Evidence
- Red teaming language models to reduce harms Anthropic · 22 August 2022 Own record
“We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs.”
- Predictability and surprise in large generative models Anthropic · 15 February 2022 Own record
“We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability.”
Reasoning
E2 is Anthropic's February 2022 paper 'Predictability and Surprise in Large Generative Models', which the criteria name as an example. It examines how large models can lead to socially harmful behavior and reports two new experiments on harms from unpredictability. E1, from August 2022, describes Anthropic's red-teaming work to find, measure and reduce harmful outputs of language models. Both papers deal with harms from language models: E1 with their potentially harmful outputs [E1], and E2 with socially harmful behaviour arising from their unpredictability [E2]. Taken together, these published research outputs show Anthropic was doing societal impacts research on language models in 2022.
Cite this claim
Daniela Amodei, “Anthropic does societal impacts research on the short- and medium-term effects of language models.” (said 2022). Verdict: resolved true. Reliability API, claim c_c6f7485f2d87, method 0.1.0+28da144b. https://epistacklabs.com/claims/c_c6f7485f2d87
Claims without a verdict
Is a verdict here wrong, or do you speak for Daniela Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.