As of March 2021, self-supervised learning had not yet brought computer vision the same improvements it has brought to natural language processing.

Judged as of 4 March 2021. True: as of that date, self-supervised pretraining in NLP (BERT, GPT and similar) had become the standard method that set the state of the art across tasks, while in computer vision supervised ImageNet pretraining was still largely the default, and SSL methods (SimCLR, MoCo, BYOL, SwAV and so on) had not yet given a comparably broad, field-changing improvement. Mostly true: this broadly holds, but SSL had already matched or beaten supervised pretraining on some vision benchmarks, which narrows the gap somewhat. Mixed: the evidence is evenly split on whether vision SSL gains were comparable. Mostly false: by March 2021, vision SSL was already widely adopted and had given gains broadly comparable to NLP's, leaving only minor differences. False: vision SSL had clearly given improvements equal to or greater than those in NLP by that date.

Set before the research began.

1. E1 Bootstrap Your Own Latent A New Approach to Self-Supervised Learning NeurIPS 2020 (Grill et al., DeepMind)

   “This tightens the gap with respect to the supervised baseline of [8], 76.5%, but is still significantly below the stronger supervised baseline of [75], 78.9%.”

2. E2 Bootstrap Your Own Latent A New Approach to Self-Supervised Learning NeurIPS 2020 (Grill et al., DeepMind)

   “BYOL outperforms SimCLR on all benchmarks and the Supervised-IN baseline on 7 of the 12 benchmarks, providing only slightly worse performance on the 5 remaining benchmarks.”

3. E3 [PDF] Rethinking Pre-training and Self-training NeurIPS 2020 (Zoph et al., Google Research, Brain Team)

   “We find that self-supervised pre-trained models using SimCLR [15] have similar performance as supervised ImageNet pre-training.”

4. E4 Barlow Twins: Self-Supervised Learning via Redundancy Reduction Proceedings of Machine Learning Research (ICML 2021) Own record

   “Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks.”

Independent NeurIPS 2020 papers report that BYOL was still significantly below a stronger supervised baseline (E1) and that SimCLR pretraining performed similarly to supervised ImageNet pretraining (E3). That fits the claim that SSL had not yet changed computer vision the way it changed NLP, and the Barlow Twins paper says the gap was still closing (E4, which is not independent of the subject). However, BYOL beat the supervised ImageNet baseline on 7 of 12 benchmarks (E2), so SSL had already matched or beaten supervised pretraining in places. Under the criteria, that narrowing makes the verdict mostly true rather than true. None of the evidence covers the NLP side directly.