ClaimsYann LeCun

“what has not been successful yet is training from video.”

Yann LeCun, interviewee 2021 Lex Fridman #258 transcript: Dark Matter of Intelligence and Self-Supervised Learning (Enacast)

Factual AI capabilities

Self-supervised training of visual representations from video has not yet been successful.

Mostly true True in substance, wrong in a detail.

Factual: Statements about the present or the past, checked against the record: did they hold up?

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2021. 'Successful' means self-supervised training on video that yields general-purpose visual representations matching or beating image-based self-supervised or supervised pretraining on standard benchmarks. TRUE: work published by 2021 shows video-trained self-supervised representations clearly lagging image-based methods on standard downstream evaluations, and the field regarded the problem as open. MOSTLY TRUE: video self-supervised learning had some benchmark successes (for example, in action recognition), but none that produced broadly competitive generic visual features. MIXED: by 2021 there were competitive results on a substantial set of benchmarks, but these were contested or limited. MOSTLY FALSE: by 2021 video self-supervised methods had produced generally competitive visual representations, with only minor caveats. FALSE: work by 2021 widely recognised as state of the art showed that video-trained self-supervised representations matched or exceeded image-based ones generally.

Set before the research began.

Evidence (4)

  1. E1 Motion-Focused Contrastive Learning of Video Representations ICCV 2021 (paper by Li et al., USTC and JD AI Research)

    “On UCF101, the linear classifier trained on the representations learnt by MCL achieves 81.91% top-1 accuracy, outperforming ImageNet supervised pre-training by 6.78%.”
  2. E2 A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning Semantic Scholar (listing of Feichtenhofer, Fan, Xiong, Girshick, He; CVPR 2021) · 2021

    “in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets”
  3. E3 Self-Supervised Learning by Cross-Modal Audio-Video Clustering arXiv (ar5iv rendering; Alwassel et al., XDC)

    “Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101.”
  4. E4 How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning? arXiv (Thoker, Doughty, Bagad, Snoek; University of Amsterdam)

    “Further, we find that self-supervised methods considerably lag behind vanilla supervised pre-training, especially when domain shift is large and the amount of available downstream samples are low.”

Reasoning

By 2021, video self-supervised methods were beating supervised pretraining on action-recognition benchmarks: E1 reports a UCF101 linear-probe gain over ImageNet supervised pretraining, E3 reports gains on HMDB51 and UCF101, and E2 reports that one simple approach worked well across several setups. All of these results are on video and action tasks. None of the evidence shows video-trained features matching image-based pretraining as general-purpose visual representations. E4, a 2022 study of these methods, found that they lag well behind plain supervised pretraining, especially under large domain shift or with few downstream samples. This matches the criteria's 'mostly true' case: some benchmark successes in action recognition, but no broadly competitive general-purpose features.

Cite this claim

Yann LeCun, “Self-supervised training of visual representations from video has not yet been successful.” (said 2021). Verdict: mostly true. Reliability API, claim c_663de66f3527, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_663de66f3527

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Yann LeCun, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Yann LeCun

  1. Mostly true Factual 18 March 2024
    “For starters, transformers are equivariant to permutations.”

    Transformers are equivariant to permutations, as ConvNets are equivariant to translations.

  2. Mostly true Factual 14 March 2024
    “If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”

    There are currently no AI systems that can teach themselves to drive a car in about 20 hours of practice, as a 17-year-old can.

  3. Mostly true Factual 8 March 2024
    “The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”

    Training an encoder to reconstruct images does not lead it to produce good generic image features, though the encoder architecture is good.

  4. Mostly true Factual 8 March 2024
    “LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”

    Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.

  5. Mostly true Factual 8 March 2024
    “No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”

    LLMs, including BERT-style bidirectional ones, exploit self-supervised learning, and LeCun has been a very strong advocate of self-supervised learning for many years.

All of them

Verdicts are made automatically; how verdicts are made.