Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
41 claims
“Unsupervised language model that can do many tasks:”
Sam Altman, on X 14 February 2019 Post by @sama on X, 2019-02-14 (X)
OpenAI has produced an unsupervised language model that can do many tasks.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
True: as of 2019-02-14, OpenAI had produced a language model trained without task-specific supervision (unsupervised or self-supervised pretraining on text) and showed it performing many different tasks, such as GPT-2, announced 14 February 2019 with zero-shot results on several tasks. Mostly true: such a model existed and did several tasks, but a minor detail was off (e.g. only a smaller version was released at that point). Mixed: the model existed but showed results on only a few tasks, or its training was substantially supervised. Mostly false: OpenAI had only a model that needed task-specific fine-tuning for each task, presented as doing many tasks unaided. False: no such OpenAI model existed as of that date.
Evidence
- Better language models and their implications - OpenAI OpenAI · 14 February 2019
“We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization—all without task-specific training.”
- Better language models and their implications - OpenAI OpenAI · 14 February 2019
“On other language tasks like question answering, reading comprehension, summarization, and translation, we are able to get surprising results without any fine-tuning of our models, simply by prompting the trained model in the right way”
- GPT-2 - Wikipedia Wikipedia · 8 December 2020
“GPT-2 was first announced on 14 February 2019.”
- GPT-2 - Wikipedia Wikipedia · 8 December 2020
“Despite this, GPT-2 achieved 5 BLEU on the WMT-14 English-to-French test set (slightly below the score of a translation via word-for-word substitution).”
Reasoning
OpenAI's GPT-2 announcement of 14 February 2019 describes a large-scale unsupervised language model. It says the model does reading comprehension, machine translation, question answering and summarization without task-specific training (E1), and that it got these results without fine-tuning, just by prompting (E2). Wikipedia, an independent source, confirms GPT-2 was announced on 14 February 2019, the date of the claim (E3). Wikipedia also notes that GPT-2's translation was weak (E4), but the claim is only that the model can do many tasks, not that it does them well, and the criteria's example for "true" describes GPT-2 exactly. The evidence shows that OpenAI had trained an unsupervised language model that performed several tasks without task-specific training [E1][E2]; no evidence item addresses which versions of the model were released.
Cite this claim
Sam Altman, “OpenAI has produced an unsupervised language model that can do many tasks.” (said 14 February 2019). Verdict: resolved true. Reliability API, claim c_4b6c10c97e0d, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_4b6c10c97e0d
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.