On typical real-world programming tasks, AI models went from 3% in January 2024 to 50% in October 2024.
Not researched Checkable, but outside the sample that was researched.
Factual: Statements about the present or the past, checked against the record: did they hold up?
Cite this claim
Dario Amodei, “On typical real-world programming tasks, AI models went from 3% in January 2024 to 50% in October 2024.” (said 11 November 2024). Verdict: not researched. Reliability API, claim c_ee2f787bb63b, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_ee2f787bb63b
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.