Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 69 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
56 42 70 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 4 promises resolved, 5 needed for a score.
All types combined: 72 (likely 66 to 78, high confidence), from 93 claims: 76 factual claims, 17 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Dario Amodei's claims
12 claims
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
Dario Amodei, Anthropic CEO, interviewed 28 February 2025 Hard Fork - Anthropic's C.E.O. Dario Amodei on Surviving the A.I. Endgame Transcript (Podscripts (Hard Fork))
Many other companies' reasoning models on the market are trained primarily on math and competition coding.
Mixed Partly true and partly false in roughly equal measure.
The second judge said mostly false; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2025-02-28. The claim concerns the main reasoning models from other developers on the market at that time, such as OpenAI o1/o3-mini, DeepSeek-R1 and Google Gemini Flash Thinking. True: their technical reports, papers or official statements show that, for many or most of them, reasoning training (for example reinforcement learning on reasoning) concentrated mainly on math and competitive programming tasks, and that these were the main domains of training and evaluation. Mostly true: math and coding (including competition coding) were clearly the dominant focus, but sources also describe a meaningful share of other domains (science, logic, general tasks). Mixed: math and coding are prominent for some of these models, while others were documented as trained broadly. Or the evidence shows math and coding dominated the benchmarks reported, while the composition of the training data is undisclosed. Mostly false: the developers' disclosures do not support "primarily". The claim treats a heavy emphasis in evaluation as a statement about training composition, or the composition is unproven. False: disclosures show that most of these reasoning models were trained mainly on broad, non-math and non-coding data.
Evidence
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning ar5iv (arXiv HTML rendering of DeepSeek-R1 paper)
“For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios.”
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning arXiv · 22 January 2025
“Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations.”
- Learning to reason with LLMs OpenAI · 12 September 2024
“OpenAI o1 ranks in the 89th percentile on competitive programming questions (Codeforces), places among the top 500 students in the US in a qualifier for the USA Math Olympiad (AIME), and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems (GPQA).”
- OpenAI o3-mini System Card OpenAI · 31 January 2025
“The OpenAI o model series is trained with large-scale reinforcement learning to reason using chain of thought.”
Reasoning
The DeepSeek-R1 paper says its reasoning RL used rule-based rewards in math, code and logical reasoning, with reward models used separately for general data (E1). Its abstract says the RL-trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions and STEM fields, compared with models trained by supervised learning on human demonstrations (E2). This partly supports a focus on math and coding, but it also covers other domains. OpenAI's o1 announcement highlights results in competitive programming, math and science (E3), and the o3-mini system card describes the training only as large-scale reinforcement learning to reason using chain of thought (E4); neither says what data the training used. Math and coding are prominent, but training composition is undisclosed for OpenAI and broader than math and coding for DeepSeek, so the claim fits the "mixed" criterion.
Cite this claim
Dario Amodei, “Many other companies' reasoning models on the market are trained primarily on math and competition coding.” (said 28 February 2025). Verdict: mixed. Reliability API, claim c_07691d3dffee, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_07691d3dffee
Claims without a verdict
- Unresolvable 157
- Not yet due 40
- Own records only 43
- Not researched 134
- Not checkable 864
Is a verdict here wrong, or do you speak for Dario Amodei? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.