Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
23 claims
“Systems pretrained this way yield considerably higher performance than when solely trained in a supervised manner.”
Meta AI blog post, byline VP and Chief AI Scientist (LeCun) 4 March 2021 Self-supervised learning: The dark matter of intelligence (Meta AI)
Language models pretrained with self-supervised learning achieve considerably higher performance than models trained solely in a supervised manner.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of March 2021. TRUE: published NLP benchmark results (e.g., GLUE, SQuAD, the BERT/GPT/RoBERTa papers) show that models pretrained with self-supervised objectives and then fine-tuned clearly and broadly beat comparable models trained only on supervised labeled data. MOSTLY TRUE: the gains are clear on most standard tasks but small or missing on some, or are 'considerable' only in low-data settings. MIXED: gains are large on roughly half of the reported tasks and negligible on the rest. MOSTLY FALSE: the gains are reported only in narrow settings and are stated as general. FALSE: supervised-only models matched or beat self-supervised pretrained models on standard benchmarks.
Evidence
- Improving Language Understanding by Generative Pre-Training OpenAI (Radford, Narasimhan, Salimans, Sutskever); PDF hosted on a third-party mirror (s10251.pcdn.co)
“Finally, we also compare with our transformer architecture directly trained on supervised target tasks, without pre-training. We observe that the lack of pre-training hurts performance across all the tasks, resulting in a 14.8% decrease compared to our full model.”
- Improving Language Understanding by Generative Pre-Training OpenAI (Radford, Narasimhan, Salimans, Sutskever); PDF hosted on a third-party mirror (s10251.pcdn.co)
“Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.”
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Association for Computational Linguistics (NAACL 2019) · June 2019
“It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).”
- Rethinking Pre-training and Self-training NeurIPS 2020 (Zoph, Ghiasi, Lin, Cui, Liu, Cubuk, Le) · 8 December 2020
“For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.”
Reasoning
The GPT-1 paper's ablation trains the same transformer directly on supervised target tasks without pre-training and reports that performance drops across all tasks, a 14.8% decrease compared with the pretrained model (E1). The same paper reports that the pretrained model beats discriminatively trained, task-specific models on 9 of 12 tasks (E2). The BERT paper reports large absolute gains on GLUE, MultiNLI and SQuAD after self-supervised pretraining (E3). E4 concerns pre-training for COCO object detection in computer vision, not NLP benchmarks; it reports that pre-training helped with one fifth of the labeled data and hurt accuracy with all of it, so it does not bear directly on the NLP benchmarks the criteria name.
Cite this claim
Yann LeCun, “Language models pretrained with self-supervised learning achieve considerably higher performance than models trained solely in a supervised manner.” (said 4 March 2021). Verdict: resolved true. Reliability API, claim c_41eeef754764, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_41eeef754764
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.