Claude models have always been the best at coding, and Claude 3.7 Sonnet took a further step up in coding.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
The claim has two parts, judged as of 2025-02-28: (a) Claude models had always been the best at coding; (b) Claude 3.7 Sonnet was a further improvement in coding. True: coding benchmarks and leaderboards (for example SWE-bench and HumanEval) show each Claude model leading frontier rivals at coding throughout its time, and Claude 3.7 Sonnet clearly improving on Claude 3.5 Sonnet. Mostly true: 3.7 improved, and Claude models usually led, with brief or minor exceptions. Mixed: 3.7's improvement is confirmed, but Claude models often did not lead (for example earlier Claude models trailed GPT-4 at coding), so 'always' fails materially. Mostly false: 3.7 improved only marginally or inconsistently, and Claude models often did not lead. False: 3.7 was not better than its predecessor at coding, and Claude models did not lead.
Set before the research began.
Evidence (8)
-
E1 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“Claude 3.7 Sonnet shows particularly strong improvements in coding and front-end web development.”
-
E2 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“This results in a score of 70.3% on the subset of n=489 verified tasks which work on our infrastructure. Without this scaffold, Claude 3.7 Sonnet achieves 63.7% on SWE-bench Verified using this same subset.”
-
E3 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“In the standard mode, Claude 3.7 Sonnet represents an upgraded version of Claude 3.5 Sonnet.”
-
E4 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“Early testing demonstrated Claude’s leadership in coding capabilities across the board”
-
E5 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“Claude 3.7 Sonnet is our best coding model to date.”
-
E6 Claude 3.7 Sonnet and Claude Code Anthropic · 24 February 2025 Own record
“Without this scaffold, Claude 3.7 Sonnet achieves 63.7% on SWE-bench Verified using this same subset.”
-
E7 Aider Polyglot leaderboard results for Claude 3.7 Sonnet Simon Willison's Weblog · 25 February 2025
“The brand new Claude 3.7 Sonnet just took the top place, when run with an increased 32,000 thinking token limit.”
-
E8 Aider Polyglot leaderboard results for Claude 3.7 Sonnet Simon Willison's Weblog · 25 February 2025
“No results yet for Claude 3.7 Sonnet on the LM Arena leaderboard, which has recently been dominated by Gemini 2.0 and Grok 3.”
Reasoning
The evidence partly supports part (b). Anthropic's own launch posts call Claude 3.7 Sonnet an upgrade over Claude 3.5 Sonnet and its best coding model to date, reporting 63.7% on a 489-task subset of SWE-bench Verified (E1, E2, E3, E5, E6). An independent post also reports that 3.7 Sonnet took top place on the Aider Polyglot leaderboard when run with an increased 32,000 thinking-token limit (E7). But no independent source compares 3.7 directly with 3.5 Sonnet, and nothing covers part (a): whether earlier Claude models led frontier rivals at coding throughout their time. E8 only says there were no LM Arena results yet for 3.7 Sonnet and that this leaderboard had recently been dominated by Gemini 2.0 and Grok 3. Because the evidence neither confirms nor rules out the "always the best" part, the criteria cannot be applied to the full claim.
Cite this claim
Dario Amodei, “Claude models have always been the best at coding, and Claude 3.7 Sonnet took a further step up in coding.” (said 28 February 2025). Verdict: unresolvable. Reliability API, claim c_43b05e7e53e6, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_43b05e7e53e6
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.