OpenAI created a system that can generate reasonable summaries of entire books.

True: as of 2021-09-23, OpenAI had built and described (e.g. in a paper or blog post) a system that produces summaries of whole books, not just passages, and that the published evaluation shows to be reasonable in at least a meaningful share of cases (e.g. human raters judging some summaries comparable to human-written ones). Mostly true: the system existed and summarised entire books, but OpenAI's own results show the summaries were reasonable only in a minority of cases, or a minor detail differs (e.g. it worked only on certain kinds of books). Mixed: a system existed but summarised only long sections or chapters rather than entire books, or its reasonableness is about evenly supported and contradicted. Mostly false: OpenAI had only an early prototype or partial capability that the claim overstates. False: no such OpenAI system existed by that date, or evidence shows its book summaries were generally not reasonable.

Set before the research began.

1. E1 Summarizing books with human feedback OpenAI · 23 September 2021 Own record

   “Our best model is fine-tuned from GPT‑3 and generates sensible summaries of entire books, sometimes even matching the average quality of human-written summaries: it achieves a 6/7 rating (similar to the average human-written summary) from humans who have read the book 5% of the time and a 5/7 rating 15% of the time.”

2. E2 Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record

   “Some of these summaries approach human-level quality: over 5% of summaries from the best 175B model were given a score of 6 out of 7, and over 15% were given a 5 out of 7, scores which were also sometimes assigned to human-written summaries (Figure 3).”

3. E3 Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record

   “However, on average our model summaries are still significantly worse than human-written summaries (Figure 2(a))”

4. E4 OpenAI unveils model that can summarize books of any length VentureBeat · 23 September 2021

   “OpenAI has developed an AI model that can summarize books of arbitrary length.”

OpenAI's announcement and paper (E1, E2) describe a GPT-3-based model that produces summaries of whole books, and VentureBeat reported independently on the same day that OpenAI had built a model that summarises books of any length (E4). OpenAI's own evaluation shows that only a minority of the summaries came close to human quality. Readers of the book gave a 6/7 rating about 5% of the time and a 5/7 rating about 15% of the time (E1, E2). The paper also says the model's summaries are, on average, significantly worse than human-written ones (E3). This matches the "mostly true" criterion: the system existed and summarised entire books, but its summaries were reasonable only in a minority of cases.