Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
40 claims
“TL;DR? A summary (from our instruction-following model) of the first part of our paper today on summarization:”
Sam Altman, OpenAI CEO, on X 23 September 2021 Post by @sama on X, 2021-09-23 (X)
OpenAI published a paper on summarization on 2021-09-23, and an instruction-following model produced a summary of its first part.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2021-09-23. True if OpenAI published a paper or research post on summarization on or around 2021-09-23 (e.g. 'Recursively Summarizing Books with Human Feedback'), and the summary Altman shared was produced by an OpenAI instruction-following model (e.g. an InstructGPT-type model). Mostly true if the paper was published within a day or two of that date, or the model's description differs in a minor detail. Mostly false if a summarization paper existed but was published materially earlier, or the summary was not model-generated. False if no OpenAI summarization paper was published around that date.
Evidence
- Recursively Summarizing Books with Human Feedback arXiv
“We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively.”
- Recursively Summarizing Books with Human Feedback arXiv
“At inference time, the model first summarizes small sections of the book and then recursively summarizes these summaries to produce a summary of the entire book.”
- OpenAI|基于人类反馈的书籍摘要 - 智源社区 BAAI (Beijing Academy of Artificial Intelligence) community hub · 26 September 2021
“Brian Shen申德周 2021-09-26 13:51 发布”
- "Summarizing Books with Human Feedback" (recursive GPT-3) — LessWrong LessWrong · 15 November 2021
“15th Nov 2021”
- Recursively Summarizing Books with Human Feedback arXiv
“We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively.”
- Recursively Summarizing Books with Human Feedback arXiv
“The input format concatenates the following, in order: previous context summaries separated from each other by "\n----\n", the separator "\n====\n", the text to summarize, and finally the phrase "TL;DR:".”
- Summarizing books with human feedback - OpenAI OpenAI · 23 September 2021 Own record
“September 23, 2021”
- Summarizing books with human feedback - OpenAI OpenAI · 23 September 2021 Own record
“Our best model is fine-tuned from GPT‑3 and generates sensible summaries of entire books”
- Recursively Summarizing Books with Human Feedback ar5iv (arXiv HTML mirror) Own record
“We collect a large volume of demonstrations and comparisons from human labelers, and fine-tune GPT-3 using behavioral cloning and reward modeling to do summarization recursively.”
- Learning to summarize with human feedback OpenAI · 4 September 2020 Own record
“September 4, 2020”
Reasoning
OpenAI's blog post 'Summarizing books with human feedback' is dated September 23, 2021 (E7), and an independent repost of the work appeared on 2021-09-26 (E3). Together these show that OpenAI published a summarization research post on the claimed date [E7]. The paper describes a summarization model fine-tuned from GPT-3 that uses a 'TL;DR:' prompt format (E1, E6, E8). None of the sources calls it an 'instruction-following' model, and no independent source confirms that the summary Altman shared was generated by a model. The publication part is settled; the description of the model is supported only indirectly.
Cite this claim
Sam Altman, “OpenAI published a paper on summarization on 2021-09-23, and an instruction-following model produced a summary of its first part.” (said 23 September 2021). Verdict: mostly true. Reliability API, claim c_2161c063cb22, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_2161c063cb22
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.