OpenAI's book-summarizing system combines reinforcement learning from human feedback and recursive task decomposition.

Judged as of 2021-09-23. True: OpenAI's book-summarization publication (the blog post 'Summarizing Books with Human Feedback' and the paper 'Recursively Summarizing Books with Human Feedback', Wu et al. 2021) describes the system as using both reinforcement learning from human feedback and recursive task decomposition. Mostly true: both methods are used, but one is described in slightly different terms (for example behavioral cloning plus RL from human feedback). Mixed: only one of the two methods is used. Mostly false: neither method is central, though related techniques appear. False: the system uses neither method, or no such OpenAI system exists.

Set before the research began.

1. E1 Summarizing books with human feedback - OpenAI OpenAI · 23 September 2021 Own record

   “Our approach: combining reinforcement learning from human feedback and recursive task decomposition”

2. E2 Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering) Own record

   “Our method combines learning from human feedback with recursive task decomposition: we use models trained on smaller parts of the task to assist humans in giving feedback on the broader task.”

3. E3 Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering) Own record

   “We train a single model to perform these tasks using standard cross-entropy behavioral cloning (BC) and reinforcement learning (RL) from human preferences (Christiano et al., 2017).”

4. E4 OpenAI unveils model that can summarize books of any length VentureBeat · 23 September 2021

   “To create the model, OpenAI combined reinforcement learning with recursive task decomposition, which procedurally breaks up a difficult task (e.g., summarizing a long piece of text) into simpler, individual ones”

The criteria name OpenAI's own publication as the reference. The blog post, dated 2021-09-23, describes the approach as "combining reinforcement learning from human feedback and recursive task decomposition" (E1). The paper says the method combines learning from human feedback with recursive task decomposition, and that training uses behavioral cloning plus RL from human preferences (E2, E3). VentureBeat separately reported the same day that OpenAI combined reinforcement learning with recursive task decomposition (E4).