Training large generative models is becoming almost prohibitively expensive.

Judged as of 2022; the hedge 'almost' sets the bar. True if 2022-era evidence shows that the cost of training frontier generative models was rising steeply, to the tens of millions of dollars or more, putting it beyond most academic and smaller organisations. Mostly true if costs were rising sharply but 'almost prohibitively' overstates them somewhat, for example if well-funded mid-sized players could still afford frontier-scale training. Mixed if costs were rising for the largest models while comparable capability was becoming cheaper to reach. Mostly false if costs were rising only modestly, so 'prohibitively' badly overstates them. False if training costs for large generative models were flat or falling.

Set before the research began.

1. E1 [PDF] Epoch Impact Report - 2022 Epoch AI

   “The dollar cost for the final training run of milestone ML systems increased at a rate of 0.5 OOMs/year between 2009 and 2022. The cost for "Large-scale" systems since September 2015 (systems that used a relatively large amount of compute) has grown more slowly, at a rate of 0.2 OOMs/year.”

2. E2 Trends in the dollar training cost of machine learning systems Epoch AI · 31 January 2023

   “While this result is based on a smaller sample (n=25 compared to n=124), I think the evidence is strong enough to conclude that the cost of the most expensive training run is growing significantly slower than milestone ML systems as a whole.”

3. E3 CHAPTER 1: Research and Development (AI Index Report 2023) Stanford Institute for Human-Centered AI (AI Index) · 2023

   “had 540 billion parameters and cost an estimated $8 million USD—PaLM was around 360 times larger than GPT-2 and”

4. E4 An empirical analysis of compute-optimal large language model training DeepMind · 12 April 2022

   “While the training compute cost for Chinchilla and Gopher are the same, we find that it outperforms Gopher and other large language models on nearly every measured task”

Epoch's 2022 report (E1) estimates that final training-run costs for milestone ML systems grew about 0.5 orders of magnitude per year from 2009 to 2022. For large-scale systems the estimate is a slower 0.2 orders of magnitude per year, and E2 likewise finds the most expensive runs growing more slowly. E3 puts PaLM's training cost at about $8 million, which is very costly but below the "tens of millions" bar in the criteria. E1 reports that training costs for milestone ML systems rose by 0.5 orders of magnitude a year between 2009 and 2022, more slowly (0.2 a year) for large-scale systems, and E3 estimates PaLM's cost at about $8 million, which suggests 'almost prohibitively' somewhat overstates the case, though training costs were rising; E4 reports that Chinchilla outperformed Gopher and other large language models at the same training compute cost.