Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
28 claims
“I think we look at some of the benchmarks where previous models were like, “Oh, could do it 6% of the time,” and now our model would do it 14 or 22% of the time.”
Dario Amodei, Anthropic CEO 11 November 2024 Transcript for Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity (Lex Fridman Podcast)
On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-11-11, against computer-use benchmarks such as OSWorld reported for the upgraded Claude 3.5 Sonnet (October 2024). True: Anthropic's or the benchmark's published figures show prior models' best scores of about 6% (roughly 5–8%) and the new model scoring about 14% and about 22% (each within about 1–2 points) in the relevant settings. Mostly true: the structure holds (prior ~single digits, new model ~14% and ~22%) but one figure is off by a few points. Mixed: the new model's figures match but the prior baseline differs a lot, or the reverse. Mostly false: the gain is real but the numbers are materially wrong (e.g. figures off by more than about 5 points). False: published results show no such improvement or figures entirely unlike those stated.
Evidence
- Claude 3.5 Sonnet (New) gets Computer Use — Latent Space Latent Space (AI News) · 23 October 2024
“Anthropic's new Computer Use API (docs here, demo here) point to OSWorld for their relevant screen manipulation benchmark - scoring 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%.”
- Computer use, Claude 3.5 Sonnet, and Claude 3.5 Haiku Anthropic · 22 October 2024 Own record
“On OSWorld, which evaluates AI models' ability to use computers like people do, Claude 3.5 Sonnet scored 14.9% in the screenshot-only category—notably better than the next-best AI system's score of 7.8%. When afforded more steps to complete the task, Claude scored 22.0%.”
- Developing a computer use model Anthropic · 22 October 2024 Own record
“On one evaluation created to test developers’ attempts to have models use computers, OSWorld, Claude currently gets 14.9%. That’s nowhere near human-level skill (which is generally 70-75%), but it’s far higher than the 7.7% obtained by the next-best AI model in the same category.”
Reasoning
I rated this claim true. Anthropic's October 2024 announcement (E2) says the upgraded Claude 3.5 Sonnet scored 14.9% on OSWorld in the screenshot-only category and 22.0% when allowed more steps. It puts the next-best AI system at 7.8%, and a second Anthropic post (E3) gives 7.7%. An independent report from Latent Space (E1) confirms the 14.9% and 7.8% figures. The prior baseline of about 7.7–7.8% falls inside the roughly 5–8% range the criteria allow for "about 6%", and 14.9% and 22.0% are each within about 1 point of the stated 14% and 22%.
Cite this claim
Dario Amodei, “On some computer-use benchmarks, previous models succeeded about 6% of the time and Anthropic's new model succeeds 14 or 22% of the time.” (said 11 November 2024). Verdict: resolved true. Reliability API, claim c_941902b8c0e8, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_941902b8c0e8
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.