Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
12 claims
“我们和其他一些研究者发现,模型里有时会有某些特定的神经元和统计指标(这些通常不出现在它的外部反应中),这些指标能够指示模型是在撒谎还是在说真话。”
Dario Amodei, Anthropic CEO, interviewee 12 April 2024 Transcript: Ezra Klein Interviews Dario Amodei (translation) (baoyu.io)
我们和其他一些研究者发现,模型里有时会有某些特定的神经元和统计指标(这些通常不出现在它的外部反应中),这些指标能够指示模型是在撒谎还是在说真话。
Mixed Partly true and partly false in roughly equal measure.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2024-04-12. True: published research by Anthropic and by other researchers before that date identified internal features, neurons or probe-detectable signals in language models that indicate whether the model's output is truthful or deceptive (for example Discovering Latent Knowledge, representation engineering, Geometry of Truth and Anthropic's probe or interpretability work on deception). Mostly true: other researchers' work clearly shows such internal truth or lie indicators, but Anthropic's own contribution addresses a closely related question (for example probes detecting deceptive 'sleeper agent' behaviour) rather than lying per se, or the signals are directions or probes rather than individual neurons. Mixed: the findings existed but were tentative and disputed in their reliability, or only Anthropic's or only others' work exists. Mostly false: the research showed only weak or inconsistent correlations, stated as a finding that the signals indicate lying. False: no such internal signal indicating lying or truthfulness had been demonstrated, or the work cited shows such probes failing.
Evidence
- The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets ar5iv (arXiv paper 2310.06824)
“Overall, we present evidence that at sufficient scale, LLMs linearly represent the truth or falsehood of factual statements.”
- The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets ar5iv (arXiv paper 2310.06824)
“Recent work has developed techniques for inferring whether a LLM is telling the truth by training probes on the LLM’s internal activations. However, this line of work is controversial, with some authors pointing out failures of these probes to generalize in basic ways, among other conceptual issues.”
- The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets ar5iv (arXiv paper 2310.06824)
“This suggests these probes may be identifying not truth, but other features that correlate with truth on their training data.”
- The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets ar5iv (arXiv paper 2310.06824)
“In fact, the probes of Li et al. 2023b and Burns et al. 2023 are linear probes, suggesting the presence of a “truth direction” in model internals.”
Reasoning
E1 and E4 show that research by other groups (the Geometry of Truth paper, plus the probes of Li et al. and Burns et al.) found or suggested linear 'truth directions' in model activations that track whether a statement is true or false. The evidence describes these findings as linear probes and directions; it does not mention individual neurons. E2 and E3 also say this line of work was disputed, because the probes sometimes fail to generalise and may pick up features that only correlate with truth. None of the evidence covers Anthropic's own work, and E2 and E3 describe this line of work as disputed; the criteria's 'tentative and disputed' condition points to mixed.
Cite this claim
Dario Amodei, “我们和其他一些研究者发现,模型里有时会有某些特定的神经元和统计指标(这些通常不出现在它的外部反应中),这些指标能够指示模型是在撒谎还是在说真话。” (said 12 April 2024). Verdict: mixed. Reliability API, claim c_955999454309, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_955999454309
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.