Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
55 claims
“Despite promising early results, SSL has not yet brought about the same improvements in computer vision that we have seen in NLP (though this will change).”
Meta AI blog post, byline VP and Chief AI Scientist (LeCun) 4 March 2021 Self-supervised learning: The dark matter of intelligence (Meta AI)
As of March 2021, self-supervised learning had not yet brought computer vision the same improvements it has brought to natural language processing.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 4 March 2021. True: as of that date, self-supervised pretraining in NLP (BERT, GPT and similar) had become the standard method that set the state of the art across tasks, while in computer vision supervised ImageNet pretraining was still largely the default, and SSL methods (SimCLR, MoCo, BYOL, SwAV and so on) had not yet given a comparably broad, field-changing improvement. Mostly true: this broadly holds, but SSL had already matched or beaten supervised pretraining on some vision benchmarks, which narrows the gap somewhat. Mixed: the evidence is evenly split on whether vision SSL gains were comparable. Mostly false: by March 2021, vision SSL was already widely adopted and had given gains broadly comparable to NLP's, leaving only minor differences. False: vision SSL had clearly given improvements equal to or greater than those in NLP by that date.
Evidence
- Bootstrap Your Own Latent A New Approach to Self-Supervised Learning NeurIPS 2020 (Grill et al., DeepMind)
“This tightens the gap with respect to the supervised baseline of [8], 76.5%, but is still significantly below the stronger supervised baseline of [75], 78.9%.”
- Bootstrap Your Own Latent A New Approach to Self-Supervised Learning NeurIPS 2020 (Grill et al., DeepMind)
“BYOL outperforms SimCLR on all benchmarks and the Supervised-IN baseline on 7 of the 12 benchmarks, providing only slightly worse performance on the 5 remaining benchmarks.”
- [PDF] Rethinking Pre-training and Self-training NeurIPS 2020 (Zoph et al., Google Research, Brain Team)
“We find that self-supervised pre-trained models using SimCLR [15] have similar performance as supervised ImageNet pre-training.”
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction Proceedings of Machine Learning Research (ICML 2021) Own record
“Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks.”
Reasoning
Independent NeurIPS 2020 papers report that BYOL was still significantly below a stronger supervised baseline (E1) and that SimCLR pretraining performed similarly to supervised ImageNet pretraining (E3). That fits the claim that SSL had not yet changed computer vision the way it changed NLP, and the Barlow Twins paper says the gap was still closing (E4, which is not independent of the subject). However, BYOL beat the supervised ImageNet baseline on 7 of 12 benchmarks (E2), so SSL had already matched or beaten supervised pretraining in places. Under the criteria, that narrowing makes the verdict mostly true rather than true. None of the evidence covers the NLP side directly.
Cite this claim
Yann LeCun, “As of March 2021, self-supervised learning had not yet brought computer vision the same improvements it has brought to natural language processing.” (said 4 March 2021). Verdict: mostly true. Reliability API, claim c_6c045aa97021, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_6c045aa97021
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.