Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
40 claims
“We created a system that can generate reasonable summaries of entire books.”
Sam Altman, OpenAI CEO, on X 23 September 2021 Post by @sama on X, 2021-09-23 (X)
OpenAI created a system that can generate reasonable summaries of entire books.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
True: as of 2021-09-23, OpenAI had built and described (e.g. in a paper or blog post) a system that produces summaries of whole books, not just passages, and that the published evaluation shows to be reasonable in at least a meaningful share of cases (e.g. human raters judging some summaries comparable to human-written ones). Mostly true: the system existed and summarised entire books, but OpenAI's own results show the summaries were reasonable only in a minority of cases, or a minor detail differs (e.g. it worked only on certain kinds of books). Mixed: a system existed but summarised only long sections or chapters rather than entire books, or its reasonableness is about evenly supported and contradicted. Mostly false: OpenAI had only an early prototype or partial capability that the claim overstates. False: no such OpenAI system existed by that date, or evidence shows its book summaries were generally not reasonable.
Evidence
- Summarizing books with human feedback OpenAI · 23 September 2021 Own record
“Our best model is fine-tuned from GPT‑3 and generates sensible summaries of entire books, sometimes even matching the average quality of human-written summaries: it achieves a 6/7 rating (similar to the average human-written summary) from humans who have read the book 5% of the time and a 5/7 rating 15% of the time.”
- Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record
“Some of these summaries approach human-level quality: over 5% of summaries from the best 175B model were given a score of 6 out of 7, and over 15% were given a 5 out of 7, scores which were also sometimes assigned to human-written summaries (Figure 3).”
- Recursively Summarizing Books with Human Feedback arXiv (ar5iv rendering), OpenAI authors Own record
“However, on average our model summaries are still significantly worse than human-written summaries (Figure 2(a))”
- OpenAI unveils model that can summarize books of any length VentureBeat · 23 September 2021
“OpenAI has developed an AI model that can summarize books of arbitrary length.”
Reasoning
OpenAI's announcement and paper (E1, E2) describe a GPT-3-based model that produces summaries of whole books, and VentureBeat reported independently on the same day that OpenAI had built a model that summarises books of any length (E4). OpenAI's own evaluation shows that only a minority of the summaries came close to human quality. Readers of the book gave a 6/7 rating about 5% of the time and a 5/7 rating about 15% of the time (E1, E2). The paper also says the model's summaries are, on average, significantly worse than human-written ones (E3). This matches the "mostly true" criterion: the system existed and summarised entire books, but its summaries were reasonable only in a minority of cases.
Cite this claim
Sam Altman, “OpenAI created a system that can generate reasonable summaries of entire books.” (said 23 September 2021). Verdict: mostly true. Reliability API, claim c_b59e7b714385, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_b59e7b714385
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.