Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
49 claims
“Modern generative AI systems are opaque in a way that fundamentally differs from traditional software.”
Dario Amodei, writing on his own site 18 February 2025 The Urgency of Interpretability (darioamodei.com)
Modern generative AI systems are opaque in a way that fundamentally differs from traditional software.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2025-02-18. TRUE: the technical and interpretability literature broadly agrees that generative AI systems (large neural networks) are opaque in a way that differs in kind from traditional software, because their behaviour comes from learned weights rather than explicitly written, inspectable code, so developers cannot directly read why they produce a given output. MOSTLY TRUE: that difference is widely accepted, but the evidence shows it is significant rather than 'fundamental', for example because interpretability tools already explain a meaningful share of model behaviour. MIXED: expert sources disagree substantially on whether the difference is one of kind or only of degree, for example citing large opaque legacy codebases or other ML systems. MOSTLY FALSE: generative AI is somewhat more opaque, but the evidence shows the claim overstates the gap. FALSE: authoritative evidence shows generative AI systems are not meaningfully more opaque than traditional software.
Evidence
- Deep Learning Opacity in Scientific Discovery | Philosophy of Science | Cambridge Core Cambridge University Press (Philosophy of Science journal) · 4 December 2023
“From the above, it should be clear that there is a straightforwardly mathematical sense in which deep neural networks are fully transparent. All weights on all connections across the network, billions as there may be, are both available for inspection and computationally tractable. However, while formally precise, neural network logic is largely semantically unintelligible.”
- Learning How AI Makes Decisions PCMag Australia · 28 November 2018
“Unlike with traditional software, we don't always have an exact idea of how deep-learning algorithms work. Troubleshooting them is very difficult, and they often fail in unanticipated and unexplainable ways.”
- Learning How AI Makes Decisions PCMag Australia · 28 November 2018
“But a byproduct of this approach is a thorough understanding of the way the neural network operates. By having monitored and documented the entire evolution of a neural network, DarwinAI's Generative Synthesis approach was able to point out the factors and data points that influenced each of the decisions its neural networks made.”
Reasoning
E2, an independent tech news article, says that unlike with traditional software, "we don't always have an exact idea" of how deep-learning algorithms work, and that they often fail in unanticipated and unexplainable ways. E1, a peer-reviewed Philosophy of Science article, supports a distinctive kind of opacity: network logic is "largely semantically unintelligible." It also qualifies the claim, noting the networks are "fully transparent" in a mathematical sense because every weight can be inspected. E3 reports one company's own statement that its technique could identify the factors behind a network's decisions, which suggests partial explainability. Taken together, the evidence supports a significant and widely recognised difference with some qualifications, which fits mostly true rather than fully true.
Cite this claim
Dario Amodei, “Modern generative AI systems are opaque in a way that fundamentally differs from traditional software.” (said 18 February 2025). Verdict: mostly true. Reliability API, claim c_71f26a2f876b, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_71f26a2f876b
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.