Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
55 claims
“It’s very expensive and difficult to train a base model, a base LLM at the moment, in the future it might be something different”
Yann LeCun, interviewee 8 March 2024 Transcript for Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI (Lex Fridman Podcast)
Training a base LLM is currently very expensive and difficult, so only a few companies can do it properly.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2024-03-08. True: evidence from early 2024 shows that training a base LLM from scratch cost a great deal (typically millions to hundreds of millions of dollars in compute) and required specialised expertise and infrastructure, so that only a small number of organisations were training competitive base models. Mostly true: cost and difficulty were high, but a noticeably larger set of organisations (e.g. many startups, academic groups and governments) were training competitive base models, so 'only a few' slightly understates it. Mixed: training frontier-scale base models was expensive, but smaller competitive base models were cheap and widely trained, so the claim holds only for the top tier. Mostly false: training base LLMs was relatively accessible and widely done, and the claim overstates the barriers. False: evidence shows base LLM training was cheap and easy in early 2024.
Evidence
- Research and Development Stanford HAI (AI Index 2024)
“For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.”
- Research and Development Stanford HAI (AI Index 2024)
“In 2023, industry produced 51 notable machine learning models, while academia contributed only 15.”
- Future-Proofing Frontier AI Regulation Center for a New American Security (CNAS) · 13 March 2024
“Training current frontier models costs on the order of tens of millions of dollars just for the final training run.”
- Mosaic LLMs: GPT-3 quality for Databricks (MosaicML) · 29 September 2022
“For smaller models in the 1.3B -> 6.7B range, training costs are now well within reach of startups and academic labs, and models can be delivered in days, not months.”
Reasoning
The AI Index 2024 estimates frontier training compute at about $78M for GPT-4 and $191M for Gemini Ultra (E1). CNAS puts the final training run for frontier models at tens of millions of dollars (E3). This supports the claim that training a base model was very expensive and difficult. However, industry and academia together produced 66 notable models in 2023 (E2), and in 2022 MosaicML said smaller 1.3B–6.7B models were within reach of startups and academic labs (E4). The evidence suggests notable models came from a range of industry and academic groups [E2], and a 2022 vendor source says smaller models were within reach of startups and academic labs [E4], so the claim holds most clearly for the top tier.
Cite this claim
Yann LeCun, “Training a base LLM is currently very expensive and difficult, so only a few companies can do it properly.” (said 8 March 2024). Verdict: mostly true. Reliability API, claim c_de5b1128f953, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_de5b1128f953
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.