Language models pretrained with self-supervised learning achieve considerably higher performance than models trained solely in a supervised manner.

Judged as of March 2021. TRUE: published NLP benchmark results (e.g., GLUE, SQuAD, the BERT/GPT/RoBERTa papers) show that models pretrained with self-supervised objectives and then fine-tuned clearly and broadly beat comparable models trained only on supervised labeled data. MOSTLY TRUE: the gains are clear on most standard tasks but small or missing on some, or are 'considerable' only in low-data settings. MIXED: gains are large on roughly half of the reported tasks and negligible on the rest. MOSTLY FALSE: the gains are reported only in narrow settings and are stated as general. FALSE: supervised-only models matched or beat self-supervised pretrained models on standard benchmarks.

Set before the research began.

1. E1 Improving Language Understanding by Generative Pre-Training OpenAI (Radford, Narasimhan, Salimans, Sutskever); PDF hosted on a third-party mirror (s10251.pcdn.co)

   “Finally, we also compare with our transformer architecture directly trained on supervised target tasks, without pre-training. We observe that the lack of pre-training hurts performance across all the tasks, resulting in a 14.8% decrease compared to our full model.”

2. E2 Improving Language Understanding by Generative Pre-Training OpenAI (Radford, Narasimhan, Salimans, Sutskever); PDF hosted on a third-party mirror (s10251.pcdn.co)

   “Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.”

3. E3 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Association for Computational Linguistics (NAACL 2019) · June 2019

   “It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).”

4. E4 Rethinking Pre-training and Self-training NeurIPS 2020 (Zoph, Ghiasi, Lin, Cui, Liu, Cubuk, Le) · 8 December 2020

   “For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data.”

The GPT-1 paper's ablation trains the same transformer directly on supervised target tasks without pre-training and reports that performance drops across all tasks, a 14.8% decrease compared with the pretrained model (E1). The same paper reports that the pretrained model beats discriminatively trained, task-specific models on 9 of 12 tasks (E2). The BERT paper reports large absolute gains on GLUE, MultiNLI and SQuAD after self-supervised pretraining (E3). E4 concerns pre-training for COCO object detection in computer vision, not NLP benchmarks; it reports that pre-training helped with one fifth of the labeled data and hurt accuracy with all of it, so it does not bear directly on the NLP benchmarks the criteria name.