Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
12 claims
“People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work.”
Dario Amodei, writing on his own site 18 February 2025 The Urgency of Interpretability (darioamodei.com)
People outside the AI field are often surprised and alarmed to learn that AI creators do not understand how their AI systems work.
Mixed Partly true and partly false in roughly equal measure.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2025-02-18. The claim has two parts: (a) AI developers do not understand how their systems work (in the sense of mechanistic interpretability); (b) people outside the field are often surprised and alarmed to learn this. True: expert sources agree that the inner workings of frontier models are largely not understood, and surveys, polling or reporting show that laypeople commonly hold the opposite belief or react with surprise or concern on learning it. Mostly true: (a) is well supported and (b) is backed by anecdote or commentary rather than systematic data. Mixed: (a) is supported but evidence on (b) is weak or points both ways. Mostly false: (a) is overstated (e.g. substantial understanding exists), or evidence shows that most of the public already knows and is not surprised. False: evidence shows developers do understand how their models work and the public reaction claimed does not occur.
Evidence
- Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025
“Despite the rapid performance improvements of the tech industry’s AI models, we still have relatively little idea how these systems arrive at decisions.”
- Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025
“For example, OpenAI recently launched new reasoning AI models, o3 and o4-mini, that perform better on some tasks, but also hallucinate more than its other models. The company doesn’t know why it’s happening.”
- Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025
“Anthropic is one of the pioneering companies in mechanistic interpretability, a field that aims to open the black box of AI models and understand why they make the decisions they do.”
Reasoning
An independent TechCrunch report says the industry still has "relatively little idea" how AI systems reach decisions, and that OpenAI does not know why its newer reasoning models hallucinate more [E1, E2]. It also describes mechanistic interpretability as a field still trying to open the black box [E3]. This supports part (a), and nothing in the evidence suggests the inner workings of these models are well understood. No item addresses part (b), whether people outside the field are often surprised or alarmed: there are no surveys, polls or reports of public reactions. Under the criteria, (a) supported with weak evidence on (b) resolves as mixed.
Cite this claim
Dario Amodei, “People outside the AI field are often surprised and alarmed to learn that AI creators do not understand how their AI systems work.” (said 18 February 2025). Verdict: mixed. Reliability API, claim c_4731f2dbb807, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_4731f2dbb807
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.