The web has provided massive amounts of text for machine learning, which has given rise to LLMs.

True if documentation of large language models shows that their training data consists mainly of large-scale web text (for example Common Crawl and WebText) and that this data was an essential enabler of LLMs. Mostly true if web text is a major contributor but other sources, such as books and code, are also significant enough that the claim simplifies slightly. Mixed if web data is only one of several equally important factors. Mostly false if web data played only a minor role. False if LLMs did not rely substantially on web text.

Set before the research began.

1. E1 Language Models are Few-Shot Learners NeurIPS Proceedings (Advances in Neural Information Processing Systems 33) · 2020

   “We also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.”

2. E2 GPT-3 - Wikipedia Wikipedia · 31 July 2020

   “Sixty percent of the weighted pre-training dataset for GPT-3 comes from a filtered version of Common Crawl consisting of 410 billion byte-pair-encoded tokens.”

3. E3 Algorithmic bias: sexualized violence against women in GPT-3 models - AI and Ethics Springer (AI and Ethics journal) · 15 January 2025

   “Common Crawl was the main contributor within the GPT-3 training dataset, and at the time of writing is the largest non-curated web corpus available”

4. E4 Scaling Language Models: Methods, Analysis & Insights from Training Gopher arXiv (DeepMind authors) · 1 April 2018

   “We train the Gopher family of models on MassiveText, a collection of large English-language text datasets from multiple sources: web pages, books, news articles, and code.”

5. E5 LLaMA: Open and Efficient Foundation Language Models arXiv (Meta AI authors) · 31 January 2026

   “Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented”

The GPT-3 paper notes the model was trained on large web corpora (E1). Two independent sources say filtered Common Crawl, a web-scraped corpus, made up 60% of GPT-3's weighted pre-training data and was its main contributor (E2, E3). Gopher's MassiveText likewise draws on web pages, but also on books, news and code (E4). Web text was the largest source for GPT-3 (E2, E3), and Gopher also drew on books, news and code (E4), but none of these sources explicitly says web text was the essential enabler, so the claim simplifies slightly.