ClaimsYann LeCun

“They work for any architecture but scale badly with data dimension.”

Dawid and LeCun, lecture notes authors June 2023 Introduction to Latent Variable Energy-Based Models (Les Houches lecture notes) (arXiv) archived copy

Factual AI capabilities

Contrastive methods for training energy-based models work for any architecture but scale badly with data dimension.

Unresolvable The public record could not settle it.

Factual: Statements about the present or the past, checked against the record: did they hold up?

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2023-06 against the energy-based model and self-supervised learning literature. True: the literature generally holds that contrastive methods can train energy-based models of any architecture, and that they become inefficient as data dimension grows because the number of contrastive or negative samples needed to shape the energy surface grows sharply. Mostly true: both points are widely accepted, but with notable caveats in the literature (e.g. some contrastive methods scale acceptably in practice). Mixed: one point is supported and the other is disputed. Mostly false: this is a minority view stated as established, or the literature shows contrastive methods are restricted to particular architectures. False: the literature shows contrastive methods scale well with dimension and that the claim is wrong.

Set before the research began.

Evidence (12)

  1. E1 A Path Towards Autonomous Machine Intelligence Version 0.9.2 Yann LeCun (hosted at marklovestech.com) · 27 June 2022 Own record

    “Because of the curse of dimensionality, in the worst case, the number of contrastive samples may grow exponentially with the dimension of the representation.”
  2. E2 A Path Towards Autonomous Machine Intelligence Version 0.9.2 Yann LeCun (hosted at marklovestech.com) · 27 June 2022 Own record

    “Most EBM approaches for unsupervised or self-supervised learning have been contrastive.”
  3. E3 Less can be more in contrastive learning Jovana Mitrovic, Brian McWilliams, Melanie Rey (DeepMind), ICBINB@NeurIPS 2020 workshop · 2020

    “Surprisingly, we discover that for a fixed batch size performance actually degrades as the number of negatives is increased.”
  4. E4 A Survey on Contrastive Self-supervised Learning arXiv (survey paper)

    “Since the batch size is limited by the GPU memory size, the scalability factor with these methods remains an issue.”
  5. E5 A Path Towards Autonomous Machine Intelligence Yann LeCun (position paper, version 0.9.2, hosted at marklovestech.com) · 27 June 2022 Own record

    “Because of the curse of dimensionality, in the worst case, the number of contrastive samples may grow exponentially with the dimension of the representation. This is the main reason why I will argue against contrastive methods.”
  6. E6 A Path Towards Autonomous Machine Intelligence Yann LeCun (position paper, version 0.9.2, hosted at marklovestech.com) · 27 June 2022 Own record

    “As with every EBM, JEPAs can also be trained with contrastive methods. But doing so runs into the curse of dimensionality and limits the practical dimension of sy.”
  7. E7 Less can be more in contrastive learning Mitrovic, McWilliams, Rey (DeepMind), 1st ICBINB Workshop at NeurIPS 2020 (PMLR vol. 137) · 2020

    “This shows that the widely used heuristic of using as many negatives as feasible can lead to suboptimal performance.”
  8. E8 A Survey on Contrastive Self-supervised Learning arXiv (survey paper; page metadata shows 2026-01-31, which appears to be a crawl or update date)

    “Since the batch size is limited by the GPU memory size, the scalability factor with these methods remains an issue.”
  9. E9 Deep Learning atcold.github.io (NYU Deep Learning course notes, student-written summary of a Yann LeCun lecture) · 23 March 2020

    “As you increase the dimension of the representation, you need more and more negative samples to make sure the energy is higher in those places not on the manifold.”
  10. E10 Deep Learning atcold.github.io (NYU Deep Learning course notes, student-written summary of a Yann LeCun lecture) · 23 March 2020

    “However, the system does not scale well as the dimensionality increases.”
  11. E11 Investigating the Role of Negatives in Contrastive Representation Learning - Microsoft Research Microsoft Research (AISTATS 2022 publication page) · February 2022

    “Theoretically, we show the existence of a collision-coverage trade-off suggesting that the optimal number of negative examples should scale with the number of underlying concepts in the data.”
  12. E12 Understanding Dimensional Collapse in Contrastive Self-supervised Learning arXiv Own record

    “In practice, contrastive methods are known to require a large number of negative samples.”

Reasoning

The independent evidence on negatives (E3, E4, E7, E11) does not address data dimension, so it neither supports nor disputes the claim that contrastive methods scale badly with dimension. The only independent support for that point is student notes of the subject's own lecture (E9, E10). No independent item addresses whether contrastive methods work for any architecture. The evidence therefore does not show one point supported and the other disputed, which is what a mixed verdict requires.

Cite this claim

Yann LeCun, “Contrastive methods for training energy-based models work for any architecture but scale badly with data dimension.” (said June 2023). Verdict: unresolvable. Reliability API, claim c_16eb9ba3955a, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_16eb9ba3955a

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Yann LeCun, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Yann LeCun

  1. Mostly true Factual 18 March 2024
    “For starters, transformers are equivariant to permutations.”

    Transformers are equivariant to permutations, as ConvNets are equivariant to translations.

  2. Mostly true Factual 14 March 2024
    “If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”

    There are currently no AI systems that can teach themselves to drive a car in about 20 hours of practice, as a 17-year-old can.

  3. Mostly true Factual 8 March 2024
    “The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”

    Training an encoder to reconstruct images does not lead it to produce good generic image features, though the encoder architecture is good.

  4. Mostly true Factual 8 March 2024
    “LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”

    Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.

  5. Mostly true Factual 8 March 2024
    “No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”

    LLMs, including BERT-style bidirectional ones, exploit self-supervised learning, and LeCun has been a very strong advocate of self-supervised learning for many years.

All of them

Verdicts are made automatically; how verdicts are made.