Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 78 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 2 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 83 (likely 78 to 88, high confidence), from 86 claims: 86 factual claims. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Daniela Amodei's claims
43 claims
“we have interpretability research, which is trying to see inside models and understand what's happening with the language models under the hood.”
Daniela Amodei, Anthropic co-founder, interviewed 4 March 2022 Daniela and Dario Amodei on Anthropic (Future of Life Institute)
Anthropic conducts interpretability research aimed at seeing inside language models and understanding their workings.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2022-03-04. True if by that date Anthropic was conducting research aimed at understanding the internal workings of language models, shown by published papers, research posts or team descriptions (e.g. mechanistic interpretability work on transformers). Mostly true if such research was under way but its focus was slightly different from 'language models' (e.g. related neural network models). Mostly false if interpretability was only planned or announced with no work under way by that date. False if Anthropic had no interpretability research programme at that time.
Evidence
- A Mathematical Framework for Transformer Circuits Transformer Circuits Thread (Anthropic) · 22 December 2021 Own record
“Transformer language models are an emerging technology that is gaining increasingly broad real-world use, for example in systems like GPT-3 , LaMDA , Codex , Meena , Gopher , and similar models.”
- A mathematical framework for Transformer Circuits Anthropic · 22 December 2021 Own record
“Reverse-engineering small attention-only transformers reveals how induction heads form via head composition.”
- Chris Olah Wiki: Co-founder at Anthropic | AI People Wiki AI People Wiki · 22 December 2021
“At Anthropic, Olah is the corresponding author of "A Mathematical Framework for Transformer Circuits" (2021), last-listed author of "Toy Models of Superposition" (2022)”
Reasoning
E1 and E2 are Anthropic's published paper "A Mathematical Framework for Transformer Circuits," dated 2021-12-22, before the judge-as-of date of 2022-03-04. In it Anthropic reverse-engineers small attention-only transformers to explain how induction heads form, and it frames the work around transformer language models. E3, an independent wiki, attributes this 2021 paper to Anthropic. Together they show interpretability research into the internals of language models was under way by the claim date.
Cite this claim
Daniela Amodei, “Anthropic conducts interpretability research aimed at seeing inside language models and understanding their workings.” (said 4 March 2022). Verdict: resolved true. Reliability API, claim c_648215898c62, method 0.1.0+28da144b. https://epistacklabs.com/claims/c_648215898c62
Claims without a verdict
Is a verdict here wrong, or do you speak for Daniela Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.