Self-supervised training of visual representations from video has not yet been successful.
Mostly true True in substance, wrong in a detail.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2021. 'Successful' means self-supervised training on video that yields general-purpose visual representations matching or beating image-based self-supervised or supervised pretraining on standard benchmarks. TRUE: work published by 2021 shows video-trained self-supervised representations clearly lagging image-based methods on standard downstream evaluations, and the field regarded the problem as open. MOSTLY TRUE: video self-supervised learning had some benchmark successes (for example, in action recognition), but none that produced broadly competitive generic visual features. MIXED: by 2021 there were competitive results on a substantial set of benchmarks, but these were contested or limited. MOSTLY FALSE: by 2021 video self-supervised methods had produced generally competitive visual representations, with only minor caveats. FALSE: work by 2021 widely recognised as state of the art showed that video-trained self-supervised representations matched or exceeded image-based ones generally.
Set before the research began.
Evidence (4)
-
E1 Motion-Focused Contrastive Learning of Video Representations ICCV 2021 (paper by Li et al., USTC and JD AI Research)
“On UCF101, the linear classifier trained on the representations learnt by MCL achieves 81.91% top-1 accuracy, outperforming ImageNet supervised pre-training by 6.78%.”
-
E2 A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning Semantic Scholar (listing of Feichtenhofer, Fan, Xiong, Girshick, He; CVPR 2021) · 2021
“in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets”
-
E3 Self-Supervised Learning by Cross-Modal Audio-Video Clustering arXiv (ar5iv rendering; Alwassel et al., XDC)
“Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101.”
-
E4 How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning? arXiv (Thoker, Doughty, Bagad, Snoek; University of Amsterdam)
“Further, we find that self-supervised methods considerably lag behind vanilla supervised pre-training, especially when domain shift is large and the amount of available downstream samples are low.”
Reasoning
By 2021, video self-supervised methods were beating supervised pretraining on action-recognition benchmarks: E1 reports a UCF101 linear-probe gain over ImageNet supervised pretraining, E3 reports gains on HMDB51 and UCF101, and E2 reports that one simple approach worked well across several setups. All of these results are on video and action tasks. None of the evidence shows video-trained features matching image-based pretraining as general-purpose visual representations. E4, a 2022 study of these methods, found that they lag well behind plain supervised pretraining, especially under large domain shift or with few downstream samples. This matches the criteria's 'mostly true' case: some benchmark successes in action recognition, but no broadly competitive general-purpose features.
Cite this claim
Yann LeCun, “Self-supervised training of visual representations from video has not yet been successful.” (said 2021). Verdict: mostly true. Reliability API, claim c_663de66f3527, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_663de66f3527
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Yann LeCun
-
Mostly true
“For starters, transformers are equivariant to permutations.”
Transformers are equivariant to permutations, as ConvNets are equivariant to translations.
-
Mostly true
“If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”
-
Mostly true
“The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”
-
Mostly true
“LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”
-
Mostly true
“No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”
Verdicts are made automatically; how verdicts are made.