Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
28 claims
“the latest model we released, Sonnet 3.5, the new or updated version, it gets something like 50% on SWE-bench.”
Dario Amodei, Anthropic CEO, interviewee 11 November 2024 Transcript for Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity (Lex Fridman Podcast)
Anthropic's updated Claude Sonnet 3.5 scores about 50% on SWE-bench.
Resolved true The record shows it happened or was so.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-11-11. 'Something like 50%' is approximate. TRUE: Anthropic's official reported SWE-bench score (e.g., SWE-bench Verified) for the upgraded Claude 3.5 Sonnet (October 2024) is roughly 45–55%. MOSTLY TRUE: the reported score is 40–45% or 55–60%, or the figure is for a SWE-bench variant other than the one implied. MOSTLY FALSE: the score is 30–40% or 60–70%. FALSE: the score is below 30% or above 70%, or no such score was reported for that model.
Evidence
- Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku Anthropic · 22 October 2024 Own record
“On coding, it improves performance on SWE-bench Verified from 33.4% to 49.0%, scoring higher than all publicly available models—including reasoning models like OpenAI o1-preview and specialized systems designed for agentic coding.”
- Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic · 17 August 2026 Own record
“The upgraded Claude 3.5 Sonnet model achieves a state-of-the-art pass@1 performance on SWE-bench Verified of 49.0%.2 We believe dedicated scaffolding and prompting can further improve the results.”
- Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic · 17 August 2026 Own record
“2Compared to the current state-of-the-art score of 45.2% as reported on SWE-bench’s leaderboard as of October 22nd,”
- Anthropic advances AI with Claude 3.5 Haiku, improved Claude 3.5 Sonnet with computer control, and Palantir partnership for US Government Intelligence and Defense Operations Notebookcheck · 9 November 2024
“On coding, it improves performance on SWE-bench Verified from 33.4% to 49.0%, scoring higher than all publicly available models—including reasoning models like OpenAI o1-preview and specialized systems designed for agentic coding.”
Reasoning
Anthropic's launch announcement of 22 October 2024 for the upgraded Claude 3.5 Sonnet says the model improves SWE-bench Verified performance from 33.4% to 49.0% (E1). The model card addendum gives the same pass@1 score of 49.0% on SWE-bench Verified (E2). Notebookcheck's report of 9 November 2024 repeats this figure (E4). Under the criteria, the claim turns on Anthropic's officially reported score. That score is 49.0%, which falls within the 45–55% range for TRUE, so "something like 50%" resolves true.
Cite this claim
Dario Amodei, “Anthropic's updated Claude Sonnet 3.5 scores about 50% on SWE-bench.” (said 11 November 2024). Verdict: resolved true. Reliability API, claim c_9c8b13320708, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_9c8b13320708
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.