On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-11-11, against computer-use benchmarks such as OSWorld reported for the upgraded Claude 3.5 Sonnet (October 2024). True: Anthropic's or the benchmark's published figures show prior models' best scores of about 6% (roughly 5–8%) and the new model scoring about 14% and about 22% (each within about 1–2 points) in the relevant settings. Mostly true: the structure holds (prior ~single digits, new model ~14% and ~22%) but one figure is off by a few points. Mixed: the new model's figures match but the prior baseline differs a lot, or the reverse. Mostly false: the gain is real but the numbers are materially wrong (e.g. figures off by more than about 5 points). False: published results show no such improvement or figures entirely unlike those stated.
Set before the research began.
Evidence (3)
-
E1 Claude 3.5 Sonnet (New) gets Computer Use — Latent Space Latent Space (AI News) · 23 October 2024
“Anthropic's new Computer Use API (docs here, demo here) point to OSWorld for their relevant screen manipulation benchmark - scoring 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%.”
-
E2 Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku Anthropic · 22 October 2024 Own record
“On OSWorld, which evaluates AI models' ability to use computers like people do, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%. When afforded more steps to complete the task, Claude scored 22.0%.”
-
E3 Developing a computer use model Anthropic · 22 October 2024 Own record
“On one evaluation created to test developers’ attempts to have models use computers, OSWorld, Claude currently gets 14.9%. That’s nowhere near human-level skill (which is generally 70-75%), but it’s far higher than the 7.7% obtained by the next-best AI model in the same category.”
Reasoning
I rated this claim true. Anthropic's October 2024 announcement (E2) says the upgraded Claude 3.5 Sonnet scored 14.9% on OSWorld in the screenshot-only category and 22.0% when allowed more steps. It puts the next-best AI system at 7.8%, and a second Anthropic post (E3) gives 7.7%. An independent report from Latent Space (E1) confirms the 14.9% and 7.8% figures. The prior baseline of about 7.7–7.8% falls inside the roughly 5–8% range the criteria allow for "about 6%", and 14.9% and 22.0% are each within about 1 point of the stated 14% and 22%.
Cite this claim
Dario Amodei, “On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.” (said 11 November 2024). Verdict: resolved true. Reliability API, claim c_941902b8c0e8, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_941902b8c0e8
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.