Contrastive methods for training energy-based models work for any architecture but scale badly with data dimension.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2023-06 against the energy-based model and self-supervised learning literature. True: the literature generally holds that contrastive methods can train energy-based models of any architecture, and that they become inefficient as data dimension grows because the number of contrastive or negative samples needed to shape the energy surface grows sharply. Mostly true: both points are widely accepted, but with notable caveats in the literature (e.g. some contrastive methods scale acceptably in practice). Mixed: one point is supported and the other is disputed. Mostly false: this is a minority view stated as established, or the literature shows contrastive methods are restricted to particular architectures. False: the literature shows contrastive methods scale well with dimension and that the claim is wrong.
Set before the research began.
Evidence (12)
-
E1 A Path Towards Autonomous Machine Intelligence Version 0.9.2 Yann LeCun (hosted at marklovestech.com) · 27 June 2022 Own record
“Because of the curse of dimensionality, in the worst case, the number of contrastive samples may grow exponentially with the dimension of the representation.”
-
E2 A Path Towards Autonomous Machine Intelligence Version 0.9.2 Yann LeCun (hosted at marklovestech.com) · 27 June 2022 Own record
“Most EBM approaches for unsupervised or self-supervised learning have been contrastive.”
-
E3 Less can be more in contrastive learning Jovana Mitrovic, Brian McWilliams, Melanie Rey (DeepMind), ICBINB@NeurIPS 2020 workshop · 2020
“Surprisingly, we discover that for a fixed batch size performance actually degrades as the number of negatives is increased.”
-
E4 A Survey on Contrastive Self-supervised Learning arXiv (survey paper)
“Since the batch size is limited by the GPU memory size, the scalability factor with these methods remains an issue.”
-
E5 A Path Towards Autonomous Machine Intelligence Yann LeCun (position paper, version 0.9.2, hosted at marklovestech.com) · 27 June 2022 Own record
“Because of the curse of dimensionality, in the worst case, the number of contrastive samples may grow exponentially with the dimension of the representation. This is the main reason why I will argue against contrastive methods.”
-
E6 A Path Towards Autonomous Machine Intelligence Yann LeCun (position paper, version 0.9.2, hosted at marklovestech.com) · 27 June 2022 Own record
“As with every EBM, JEPAs can also be trained with contrastive methods. But doing so runs into the curse of dimensionality and limits the practical dimension of sy.”
-
E7 Less can be more in contrastive learning Mitrovic, McWilliams, Rey (DeepMind), 1st ICBINB Workshop at NeurIPS 2020 (PMLR vol. 137) · 2020
“This shows that the widely used heuristic of using as many negatives as feasible can lead to suboptimal performance.”
-
E8 A Survey on Contrastive Self-supervised Learning arXiv (survey paper; page metadata shows 2026-01-31, which appears to be a crawl or update date)
“Since the batch size is limited by the GPU memory size, the scalability factor with these methods remains an issue.”
-
E9 Deep Learning atcold.github.io (NYU Deep Learning course notes, student-written summary of a Yann LeCun lecture) · 23 March 2020
“As you increase the dimension of the representation, you need more and more negative samples to make sure the energy is higher in those places not on the manifold.”
-
E10 Deep Learning atcold.github.io (NYU Deep Learning course notes, student-written summary of a Yann LeCun lecture) · 23 March 2020
“However, the system does not scale well as the dimensionality increases.”
-
E11 Investigating the Role of Negatives in Contrastive Representation Learning - Microsoft Research Microsoft Research (AISTATS 2022 publication page) · February 2022
“Theoretically, we show the existence of a collision-coverage trade-off suggesting that the optimal number of negative examples should scale with the number of underlying concepts in the data.”
-
E12 Understanding Dimensional Collapse in Contrastive Self-supervised Learning arXiv Own record
“In practice, contrastive methods are known to require a large number of negative samples.”
Reasoning
The independent evidence on negatives (E3, E4, E7, E11) does not address data dimension, so it neither supports nor disputes the claim that contrastive methods scale badly with dimension. The only independent support for that point is student notes of the subject's own lecture (E9, E10). No independent item addresses whether contrastive methods work for any architecture. The evidence therefore does not show one point supported and the other disputed, which is what a mixed verdict requires.
Cite this claim
Yann LeCun, “Contrastive methods for training energy-based models work for any architecture but scale badly with data dimension.” (said June 2023). Verdict: unresolvable. Reliability API, claim c_16eb9ba3955a, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_16eb9ba3955a
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Yann LeCun
-
Mostly true
“For starters, transformers are equivariant to permutations.”
Transformers are equivariant to permutations, as ConvNets are equivariant to translations.
-
Mostly true
“If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”
-
Mostly true
“The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”
-
Mostly true
“LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”
-
Mostly true
“No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”
Verdicts are made automatically; how verdicts are made.