Many other companies' reasoning models on the market are trained primarily on math and competition coding.
Mixed Partly true and partly false in roughly equal measure.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge said mostly false; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2025-02-28. The claim concerns the main reasoning models from other developers on the market at that time, such as OpenAI o1/o3-mini, DeepSeek-R1 and Google Gemini Flash Thinking. True: their technical reports, papers or official statements show that, for many or most of them, reasoning training (for example reinforcement learning on reasoning) concentrated mainly on math and competitive programming tasks, and that these were the main domains of training and evaluation. Mostly true: math and coding (including competition coding) were clearly the dominant focus, but sources also describe a meaningful share of other domains (science, logic, general tasks). Mixed: math and coding are prominent for some of these models, while others were documented as trained broadly. Or the evidence shows math and coding dominated the benchmarks reported, while the composition of the training data is undisclosed. Mostly false: the developers' disclosures do not support "primarily". The claim treats a heavy emphasis in evaluation as a statement about training composition, or the composition is unproven. False: disclosures show that most of these reasoning models were trained mainly on broad, non-math and non-coding data.
Set before the research began.
Evidence (4)
-
E1 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning ar5iv (arXiv HTML rendering of DeepSeek-R1 paper)
“For reasoning data, we adhere to the methodology outlined in DeepSeek-R1-Zero, which utilizes rule-based rewards to guide the learning process in math, code, and logical reasoning domains. For general data, we resort to reward models to capture human preferences in complex and nuanced scenarios.”
-
E2 DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning arXiv · 22 January 2025
“Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations.”
-
E3 Learning to reason with LLMs OpenAI · 12 September 2024
“OpenAI o1 ranks in the 89th percentile on competitive programming questions (Codeforces), places among the top 500 students in the US in a qualifier for the USA Math Olympiad (AIME), and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems (GPQA).”
-
E4 OpenAI o3-mini System Card OpenAI · 31 January 2025
“The OpenAI o model series is trained with large-scale reinforcement learning to reason using chain of thought.”
Reasoning
The DeepSeek-R1 paper says its reasoning RL used rule-based rewards in math, code and logical reasoning, with reward models used separately for general data (E1). Its abstract says the RL-trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions and STEM fields, compared with models trained by supervised learning on human demonstrations (E2). This partly supports a focus on math and coding, but it also covers other domains. OpenAI's o1 announcement highlights results in competitive programming, math and science (E3), and the o3-mini system card describes the training only as large-scale reinforcement learning to reason using chain of thought (E4); neither says what data the training used. Math and coding are prominent, but training composition is undisclosed for OpenAI and broader than math and coding for DeepSeek, so the claim fits the "mixed" criterion.
Cite this claim
Dario Amodei, “Many other companies' reasoning models on the market are trained primarily on math and competition coding.” (said 28 February 2025). Verdict: mixed. Reliability API, claim c_07691d3dffee, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_07691d3dffee
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
-
Resolved true
“we train Claude 3.7, you know, more to focus on these real-world tasks.”
Anthropic trained Claude 3.7 to focus more on real-world tasks, as distinct from competition coding.
Verdicts are made automatically; how verdicts are made.