Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 72 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
70 62 78 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 2 promises resolved, 5 needed for a score.
All types combined: 75 (likely 69 to 80, high confidence), from 79 claims: 35 factual claims, 44 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Scott Alexander's claims
“Training GPT-4 will cost $100M, which is a lot.”
Scott Alexander, blogger 20 February 2023 Grading My 2018 Predictions For 2023 (Astral Codex Ten)
Training GPT-4 will cost $100 million.
Resolved true The record shows it happened or was so.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Treat '$100M' as a round, approximate figure for the compute and training cost of GPT-4. TRUE: OpenAI statements (such as Sam Altman's remark that it cost 'more than $100 million'), or credible analyst estimates, put the training cost roughly between $70M and $150M. MOSTLY TRUE: the best estimates fall roughly between $50M and $70M, or between $150M and $200M. MIXED: credible estimates disagree widely, with some near $100M and others far away. MOSTLY FALSE: the best estimates are off by a factor of 2 to 3 (about $33M–50M, or $200M–300M). FALSE: the cost is credibly shown to be more than 3 times higher or lower than $100M.
Evidence
- Altman says the age of giant models is 'already over' and GPT-4 cost over $100M — AGI Watch AGI Watch · 17 April 2023
“Speaking at MIT, Altman said returns from simply scaling parameters were diminishing and that progress would come from other directions, while confirming GPT-4's training cost exceeded $100 million.”
- The rising costs of training frontier AI models arXiv (Epoch AI and Stanford University authors) · 15 January 2024
“For example, we estimate that it cost $800M to acquire the hardware used to train GPT-4, compared to $40M for the amortized hardware CapEx + energy cost.”
- The rising costs of training frontier AI models arXiv (Epoch AI and Stanford University authors) · 15 January 2024
“while the amortized hardware cost over the whole model development increased from $4M to $90M.”
- Report on Frontier Model Training — LessWrong LessWrong (post by YafahEdelman) · 30 August 2023
“New state of the art (frontier) models will likely cost on the order of a billion dollars, with the most recent frontier model, GPT-4, having cost approximately half a billion dollars.”
Reasoning
E1 reports Sam Altman confirming that GPT-4's training cost was more than $100 million, and the resolution criteria name this statement as an example of a TRUE outcome. The Epoch AI study gives a figure of $90M for amortized hardware cost over a whole model's development (E3), though the quoted passage does not name the model. It also gives $40M for GPT-4's amortized hardware CapEx and energy cost, against $800M to acquire the hardware (E2). One estimate is much higher: about half a billion dollars, in an August 2023 LessWrong post (E4). Altman's reported statement that the cost exceeded $100 million (E1) matches the example the criteria give for a TRUE outcome, though other estimates range from $40M (E2) to about half a billion dollars (E4).
Cite this claim
Scott Alexander, “Training GPT-4 will cost $100 million.” (said 20 February 2023). Verdict: resolved true. Reliability API, claim c_dac7a6220f35, method 0.1.0+8bea457b. https://epistacklabs.com/claims/c_dac7a6220f35
Claims without a verdict
Is a verdict here wrong, or do you speak for Scott Alexander? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.