ClaimsDario Amodei

“I think we look at some of the benchmarks where previous models were like, “Oh, could do it 6% of the time,” and now our model would do it 14 or 22% of the time.”

Dario Amodei, Anthropic CEO 11 November 2024 Transcript for Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity (Lex Fridman Podcast)

Factual AI capabilities

On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.

Resolved true The record shows it happened or was so.

Factual: Statements about the present or the past, checked against the record: did they hold up?

Said for an organisation they lead or work for; it counts on both records.

What would count as true or false

Judge as of 2024-11-11, against computer-use benchmarks such as OSWorld reported for the upgraded Claude 3.5 Sonnet (October 2024). True: Anthropic's or the benchmark's published figures show prior models' best scores of about 6% (roughly 5–8%) and the new model scoring about 14% and about 22% (each within about 1–2 points) in the relevant settings. Mostly true: the structure holds (prior ~single digits, new model ~14% and ~22%) but one figure is off by a few points. Mixed: the new model's figures match but the prior baseline differs a lot, or the reverse. Mostly false: the gain is real but the numbers are materially wrong (e.g. figures off by more than about 5 points). False: published results show no such improvement or figures entirely unlike those stated.

Set before the research began.

Evidence (3)

  1. E1 Claude 3.5 Sonnet (New) gets Computer Use — Latent Space Latent Space (AI News) · 23 October 2024

    “Anthropic's new Computer Use API (docs here, demo here) point to OSWorld for their relevant screen manipulation benchmark - scoring 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%.”
  2. E2 Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku Anthropic · 22 October 2024 Own record

    “On OSWorld, which evaluates AI models' ability to use computers like people do, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%. When afforded more steps to complete the task, Claude scored 22.0%.”
  3. E3 Developing a computer use model Anthropic · 22 October 2024 Own record

    “On one evaluation created to test developers’ attempts to have models use computers, OSWorld, Claude currently gets 14.9%. That’s nowhere near human-level skill (which is generally 70-75%), but it’s far higher than the 7.7% obtained by the next-best AI model in the same category.”

Reasoning

I rated this claim true. Anthropic's October 2024 announcement (E2) says the upgraded Claude 3.5 Sonnet scored 14.9% on OSWorld in the screenshot-only category and 22.0% when allowed more steps. It puts the next-best AI system at 7.8%, and a second Anthropic post (E3) gives 7.7%. An independent report from Latent Space (E1) confirms the 14.9% and 7.8% figures. The prior baseline of about 7.7–7.8% falls inside the roughly 5–8% range the criteria allow for "about 6%", and 14.9% and 22.0% are each within about 1 point of the stated 14% and 22%.

Cite this claim

Dario Amodei, “On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.” (said 11 November 2024). Verdict: resolved true. Reliability API, claim c_941902b8c0e8, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_941902b8c0e8

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Dario Amodei, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Dario Amodei

  1. Mixed Factual 23 September 2026
    “It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”

    AI performance in new intellectual domains has repeatedly gone from weak to superhuman within a few years.

  2. Mixed Factual 10 March 2025
    “we were measuring these trends back when models cost $1,000 or $10,000 to train”

    Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.

  3. Mixed Factual 28 February 2025
    “a lot of the other reasoning models in the market are trained primarily on math and competition coding,”

    Many other companies' reasoning models on the market are trained primarily on math and competition coding.

  4. Mostly true Factual 28 February 2025
    “I do think AI coding is moving the fastest of all the other areas.”

    Amodei said he thinks AI coding is moving the fastest of all the areas of AI.

  5. Mostly true Factual 28 February 2025
    “Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”

    The model sometimes gives a very short response because, when it knows it gains nothing from thinking further, it does not think for longer; a bound on how long it thinks is still described as valuable.

All of them

Verdicts are made automatically; how verdicts are made.