Anthropic's updated Claude Sonnet 3.5 scores about 50% on SWE-bench.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-11-11. 'Something like 50%' is approximate. TRUE: Anthropic's official reported SWE-bench score (e.g., SWE-bench Verified) for the upgraded Claude 3.5 Sonnet (October 2024) is roughly 45–55%. MOSTLY TRUE: the reported score is 40–45% or 55–60%, or the figure is for a SWE-bench variant other than the one implied. MOSTLY FALSE: the score is 30–40% or 60–70%. FALSE: the score is below 30% or above 70%, or no such score was reported for that model.
Set before the research began.
Evidence (4)
-
E1 Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku Anthropic · 22 October 2024 Own record
“On coding, it improves performance on SWE-bench Verified from 33.4% to 49.0%, scoring higher than all publicly available models—including reasoning models like OpenAI o1-preview and specialized systems designed for agentic coding.”
-
E2 Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic · 17 August 2026 Own record
“The upgraded Claude 3.5 Sonnet model achieves a state-of-the-art pass@1 performance on SWE-bench Verified of 49.0%.2 We believe dedicated scaffolding and prompting can further improve the results.”
-
E3 Model Card Addendum: Claude 3.5 Haiku and Upgraded Claude 3.5 Sonnet Anthropic · 17 August 2026 Own record
“2Compared to the current state-of-the-art score of 45.2% as reported on SWE-bench’s leaderboard as of October 22nd,”
-
E4 Anthropic advances AI with Claude 3.5 Haiku, improved Claude 3.5 Sonnet with computer control, and Palantir partnership for US Government Intelligence and Defense Operations Notebookcheck · 9 November 2024
“On coding, it improves performance on SWE-bench Verified from 33.4% to 49.0%, scoring higher than all publicly available models—including reasoning models like OpenAI o1-preview and specialized systems designed for agentic coding.”
Reasoning
Anthropic's launch announcement of 22 October 2024 for the upgraded Claude 3.5 Sonnet says the model improves SWE-bench Verified performance from 33.4% to 49.0% (E1). The model card addendum gives the same pass@1 score of 49.0% on SWE-bench Verified (E2). Notebookcheck's report of 9 November 2024 repeats this figure (E4). Under the criteria, the claim turns on Anthropic's officially reported score. That score is 49.0%, which falls within the 45–55% range for TRUE, so "something like 50%" resolves true.
Cite this claim
Dario Amodei, “Anthropic's updated Claude Sonnet 3.5 scores about 50% on SWE-bench.” (said 11 November 2024). Verdict: resolved true. Reliability API, claim c_9c8b13320708, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_9c8b13320708
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.