Training a base LLM is currently very expensive and difficult, so only a few companies can do it properly.

Judge as of 2024-03-08. True: evidence from early 2024 shows that training a base LLM from scratch cost a great deal (typically millions to hundreds of millions of dollars in compute) and required specialised expertise and infrastructure, so that only a small number of organisations were training competitive base models. Mostly true: cost and difficulty were high, but a noticeably larger set of organisations (e.g. many startups, academic groups and governments) were training competitive base models, so 'only a few' slightly understates it. Mixed: training frontier-scale base models was expensive, but smaller competitive base models were cheap and widely trained, so the claim holds only for the top tier. Mostly false: training base LLMs was relatively accessible and widely done, and the claim overstates the barriers. False: evidence shows base LLM training was cheap and easy in early 2024.

Set before the research began.

1. E1 Research and Development Stanford HAI (AI Index 2024)

   “For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.”

2. E2 Research and Development Stanford HAI (AI Index 2024)

   “In 2023, industry produced 51 notable machine learning models, while academia contributed only 15.”

3. E3 Future-Proofing Frontier AI Regulation Center for a New American Security (CNAS) · 13 March 2024

   “Training current frontier models costs on the order of tens of millions of dollars just for the final training run.”

4. E4 Mosaic LLMs: GPT-3 quality for Databricks (MosaicML) · 29 September 2022

   “For smaller models in the 1.3B -> 6.7B range, training costs are now well within reach of startups and academic labs, and models can be delivered in days, not months.”

The AI Index 2024 estimates frontier training compute at about $78M for GPT-4 and $191M for Gemini Ultra (E1). CNAS puts the final training run for frontier models at tens of millions of dollars (E3). This supports the claim that training a base model was very expensive and difficult. However, industry and academia together produced 66 notable models in 2023 (E2), and in 2022 MosaicML said smaller 1.3B–6.7B models were within reach of startups and academic labs (E4). The evidence suggests notable models came from a range of industry and academic groups [E2], and a 2022 vendor source says smaller models were within reach of startups and academic labs [E4], so the claim holds most clearly for the top tier.