AI scaling laws are sometimes predictable to several significant figures, which is not seen outside of physics.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
What would count as true or false
The claim has two parts: (a) AI scaling-law predictions have sometimes been accurate to several significant figures (at least about 2–3); (b) that precision is not seen outside physics. The hedge 'sometimes' applies to (a). TRUE: there are documented cases where scaling-law extrapolations predicted a model's loss to several significant figures (for example GPT-4's predicted final loss), and empirical fields outside physics rarely, if ever, achieve comparable predictive precision. MOSTLY TRUE: (a) holds, but (b) overstates things slightly, for example because a few non-physics fields (such as parts of chemistry or engineering built on physics) reach similar precision. MIXED: (a) is supported but (b) is clearly wrong, because precise prediction is common in several non-physics sciences; or (a) is supported only to about 1–2 significant figures. MOSTLY FALSE: scaling predictions are only approximate (within about 10% or worse), so the 'several significant figures' framing overstates a real but rough regularity. FALSE: there are no documented cases of scaling laws predicting outcomes with multi-digit precision.
Set before the research began.
Evidence (9)
-
E1 GPT-4 OpenAI · 14 March 2023
“To verify this scalability, we accurately predicted in advance GPT-4’s final loss on our internal codebase (not part of the training set) by extrapolating from models trained using the same methodology but using 10,000x less compute:”
-
E2 GPT-4 OpenAI · 14 March 2023
“our GPT-4 training run was (for us at least!) unprecedentedly stable, becoming our first large model whose training performance we were able to accurately predict ahead of time.”
-
E3 GPT-4 OpenAI · 14 March 2023
“For example, we successfully predicted the pass rate on a subset of the HumanEval dataset, extrapolating from models with 1,000x less compute:”
-
E4 (none) OpenAI
“To verify this scalability, we accurately predicted in advance GPT-4’s final loss on our internal codebase (not part of the training set) by extrapolating from models trained using the same methodology but using 10,000x less compute:”
-
E5 (none) OpenAI
“Now that we can accurately predict the metric we optimize during training (loss), we’re starting to develop methodology to predict more interpretable metrics. For example, we successfully predicted the pass rate on a subset of the HumanEval dataset, extrapolating from models with 1,000x less compute:”
-
E6 (none) OpenAI
“Some capabilities are still hard to predict. For example, the Inverse Scaling Prize was a competition to find a metric that gets worse as model compute increases, and hindsight neglect was one of the winners.”
-
E7 Deep Learning Scaling is Predictable, Empirically ar5iv (arXiv HTML rendering)
“First, the loss scaling relationships are smooth power-law functions of the data set size with almost exactly the same exponents: \(\beta_{g}=-0.0656\pm 1\%\).”
-
E8 Deep Learning Scaling is Predictable, Empirically ar5iv (arXiv HTML rendering)
“Despite the model size scaling differences, for a given model architecture, we can accurately predict the model size that will best fit increasingly larger data sets.”
-
E9 Deep Learning Scaling is Predictable, Empirically ar5iv (arXiv HTML rendering)
“Models are unable to extract enough information from these small training sets to make many accurate classifications on the validation set.”
Reasoning
OpenAI's GPT-4 page says it "accurately predicted" GPT-4's final loss and a HumanEval pass rate from much smaller models (E1, E3, E5), but its text gives no predicted or observed values, so it does not show how many significant figures were achieved. E7 gives a fitted scaling exponent to about three significant figures with ±1% uncertainty, but that measures how stable the fit is, not how close a prediction came to an observed result. E6 notes that some capabilities are still hard to predict. No evidence addresses part (b), whether fields outside physics reach similar precision, so neither part of the claim can be settled.
Cite this claim
Dario Amodei, “AI scaling laws are sometimes predictable to several significant figures, which is not seen outside of physics.” (said 8 August 2023). Verdict: unresolvable. Reliability API, claim c_72975861bf9c, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_72975861bf9c
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Dario Amodei
-
Mixed
“It’s easy to dismiss this as a one-off or curiosity, but we’ve repeatedly seen a pattern where AI performance in new intellectual domains goes from weak to superhuman in a matter of a few years.”
-
Mixed
“we were measuring these trends back when models cost $1,000 or $10,000 to train”
Amodei's group measured AI scaling trends when models cost $1,000 or $10,000 to train.
-
Mixed
“a lot of the other reasoning models in the market are trained primarily on math and competition coding,”
-
Mostly true
“I do think AI coding is moving the fastest of all the other areas.”
Amodei said he thinks AI coding is moving the fastest of all the areas of AI.
-
Mostly true
“Because when it knows that it doesn't get any gain out of thinking further, it doesn't think for longer.”
Verdicts are made automatically; how verdicts are made.