Claims

Factual · AI capabilities

“We created a system that can generate reasonable summaries of entire books.”

Sam Altman, OpenAI CEO, on X · 23 September 2021 · Post by @sama on X, 2021-09-23 (X)

OpenAI created a system that can generate reasonable summaries of entire books.

Mostly true True in substance, wrong in a detail.

The second judge said resolved true; the verdict shown is the first judge's.

Said for an organisation they lead or work for; it counts on both records.

What would count as true or false

True: as of 2021-09-23, OpenAI had built and described (e.g. in a paper or blog post) a system that produces summaries of whole books, not just passages, and that the published evaluation shows to be reasonable in at least a meaningful share of cases (e.g. human raters judging some summaries comparable to human-written ones). Mostly true: the system existed and summarised entire books, but OpenAI's own results show the summaries were reasonable only in a minority of cases, or a minor detail differs (e.g. it worked only on certain kinds of books). Mixed: a system existed but summarised only long sections or chapters rather than entire books, or its reasonableness is about evenly supported and contradicted. Mostly false: OpenAI had only an early prototype or partial capability that the claim overstates. False: no such OpenAI system existed by that date, or evidence shows its book summaries were generally not reasonable.

Evidence

1. Summarizing books with human feedback OpenAI · 23 September 2021 Own record

   “Our best model is fine-tuned from GPT‑3 and generates sensible summaries of entire books, sometimes even matching the average quality of human-written summaries: it achieves a 6/7 rating (similar to the average human-written summary) from humans who have read the book 5% of the time and a 5/7 rating 15% of the time.”

2. Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record

   “Some of these summaries approach human-level quality: over 5% of summaries from the best 175B model were given a score of 6 out of 7, and over 15% were given a 5 out of 7, scores which were also sometimes assigned to human-written summaries (Figure 3).”

3. Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record

   “However, on average our model summaries are still significantly worse than human-written summaries (Figure 2(a))”

4. OpenAI unveils model that can summarize books of any length VentureBeat · 23 September 2021

   “OpenAI has developed an AI model that can summarize books of arbitrary length.”

Reasoning

OpenAI's announcement and paper (E1, E2) describe a GPT-3-based model that produces summaries of whole books, and VentureBeat reported independently on the same day that OpenAI had built a model that summarises books of any length (E4). OpenAI's own evaluation shows that only a minority of the summaries came close to human quality. Readers of the book gave a 6/7 rating about 5% of the time and a 5/7 rating about 15% of the time (E1, E2). The paper also says the model's summaries are, on average, significantly worse than human-written ones (E3). This matches the "mostly true" criterion: the system existed and summarised entire books, but its summaries were reasonable only in a minority of cases.

Cite this claim

Sam Altman, “OpenAI created a system that can generate reasonable summaries of entire books.” (said 23 September 2021). Verdict: mostly true. Reliability API, claim c_b59e7b714385, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_b59e7b714385