Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
49 claims
“this lack of understanding is essentially unprecedented in the history of technology.”
Dario Amodei, writing on his own site 18 February 2025 The Urgency of Interpretability (darioamodei.com)
The lack of understanding of how AI systems work is essentially unprecedented in the history of technology.
Mostly true True in substance, wrong in a detail.
The second judge said mostly false; the verdict shown is the first judge's.
What would count as true or false
The claim is hedged ('essentially'). It says that how poorly the internal workings of modern AI systems are understood has essentially no precedent among major technologies. TRUE: expert or historical analyses support that widely deployed technologies have generally been built on a reasonably good mechanistic understanding, and that no comparable case exists of a major deployed technology whose workings were as opaque to its builders. MOSTLY TRUE: the claim broadly holds, but there are partial historical parallels (for example early pharmaceuticals such as aspirin, metallurgy, or early steam engines before thermodynamics) that make 'unprecedented' slightly too strong. MIXED: significant precedents exist that are arguably comparable, and expert opinion is divided. MOSTLY FALSE: clear precedents exist of important technologies deployed with little understanding of how they work, so the claim exaggerates a real gap in understanding. FALSE: it was common historically to deploy technologies whose mechanisms were similarly poorly understood, and AI is not distinctive in this respect.
Evidence
- The Virtues of Interpretable Medical AI Cambridge Quarterly of Healthcare Ethics (authors: Joshua Hatherley, Robert Sparrow, Mark Howard, Monash University) · 2024
“Other authors have also highlighted acetaminophen and penicillin, which “were in widespread use for decades before their mechanism of action was understood,” along with selective serotonin reuptake inhibitors, whose underlying causal mechanism is still unclear.”
- The Virtues of Interpretable Medical AI Cambridge Quarterly of Healthcare Ethics (authors: Joshua Hatherley, Robert Sparrow, Mark Howard, Monash University) · 2024
“For example, electroconvulsive therapy is highly effective for severe depression, but we have no idea how it works. Likewise, there are many drugs that seem to work even though no one can explain how.”
- The Virtues of Interpretable Medical AI Cambridge Quarterly of Healthcare Ethics (authors: Joshua Hatherley, Robert Sparrow, Mark Howard, Monash University) · 2024
“A problem, however, is that the inner workings of these systems have often proven thoroughly resistant to understanding, explanation, or justification, not only to end users (e.g., doctors, clinicians, and nurses), but even to the designers of these systems themselves.”
Reasoning
A medical ethics journal article (E3) says the inner workings of these AI systems have often resisted understanding, even by the people who designed them. That supports the premise that AI is unusually opaque. The same article also records partial historical parallels: acetaminophen and penicillin were widely used for decades before anyone understood how they worked, the mechanism of SSRIs is still unclear (E1), and electroconvulsive therapy and many drugs work in ways no one can explain (E2). The resolution criteria name early pharmaceuticals as exactly the kind of partial parallel that makes "unprecedented" slightly too strong, so the hedged claim broadly holds but is somewhat overstated.
Cite this claim
Dario Amodei, “The lack of understanding of how AI systems work is essentially unprecedented in the history of technology.” (said 18 February 2025). Verdict: mostly true. Reliability API, claim c_217a061d517b, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_217a061d517b
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.