Yann LeCun

Factual · AI capabilities

“Interestingly, related ideas were”

Anna Dawid and Yann LeCun, lecture notes authors · June 2023 · Introduction to Latent Variable Energy-Based Models (Les Houches lecture notes) (arXiv)

Ideas related to VICReg-style regularization were explored already in 1992, per reference [62].

Unresolvable The public record could not settle it.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2023-06. True: the work cited as reference [62] (in the document where the claim appears) is a 1992 publication, and it explores ideas clearly related to VICReg-style regularisation (e.g. keeping or maximising information/variance in embeddings, or decorrelating outputs, to prevent collapse; for instance Becker & Hinton 1992 or Schmidhuber 1992 on predictability minimisation). Mostly true: the related work exists and fits, but the year is off by about a year or the reference number is wrong. Mixed: the 1992 work is only loosely related. Mostly false: the cited work is not from about 1992, or its relation to VICReg is strained. False: no such related 1992 work exists or the citation does not match.

Evidence

1. Self-organizing neural network that discovers surfaces in random-dot stereograms - Nature Nature · 1992

   “Small modules that look at separate but related parts of the perceptual input discover these common causes by striving to produce outputs that agree with each other (Fig. la).”

2. Self-organizing neural network that discovers surfaces in random-dot stereograms - Nature Nature · 1992

   “Becker, S., Hinton, G. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature 355, 161–163 (1992).”

3. The 2022 JEPA is essentially the 1992 PMAX IDSIA (Jürgen Schmidhuber) · 6 January 2024

   “[PMAX] J. Schmidhuber and D. Prelinger. Discovering predictable classifications. Neural Computation, 5(4):625-635, 1993. Based on TR CU-CS-626-92 (1992). PDF.”

4. VICReg: Variance-Invariance-Covariance Re-gularization for Self-Supervised Learning arXiv Own record

   “(1) a term that maintains the variance of each embedding dimension above a threshold, (2) a term that decorrelates each pair of variables.”

5. The 2022 JEPA is essentially the 1992 PMAX people.idsia.ch (J. Schmidhuber) · 6 January 2024

   “J. Schmidhuber and D. Prelinger. Discovering predictable classifications. Neural Computation, 5(4):625-635, 1993. Based on TR CU-CS-626-92 (1992).”

6. The 2022 JEPA is essentially the 1992 PMAX people.idsia.ch (J. Schmidhuber) · 6 January 2024

   “The 1992 [PMAX] paper discusses not only auto encoders but also other techniques for encoding data, e.g., maximizing constrained variance of the code (Sec. 2.1), Infomax (Sec. 2.3)”

7. The 2022 JEPA is essentially the 1992 PMAX people.idsia.ch (J. Schmidhuber) · 6 January 2024

   “VICReg: variance hinge (Sec. 2.1) plus covariance penalty (Sec. 2.3) applied simultaneously, so indeed that paper is one section from [PMAX].”

8. VICReg: Variance-Invariance-Covariance Re-gularization for Self-Supervised Learning arXiv Own record

   “(1) a term that maintains the variance of each embedding dimension above a threshold, (2) a term that decorrelates each pair of variables.”

9. To Compress or Not to Compress - Self-Supervised Learning and Information Theory: A Review ar5iv (arXiv HTML mirror)

   “Becker and Hinton 1992 introduced a representation learning approach based on maximizing an approximation of the mutual information between alternative latent vectors obtained from the same image.”

10. Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training arXiv (HTML)

    “Most of the SSL frameworks we will cover in this manuscript refer to the works of Becker & Hinton (1992) and Bromley et al. (1993) as the earliest research efforts that use discriminative self-supervision in the form it is used nowadays”

11. Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training arXiv (HTML)

    “Becker & Hinton (1992) Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):161–163, 1992.”

12. The 2010s: Our Decade of Deep Learning / Outlook on the 2020s IDSIA (Schmidhuber's lab site)

    “[PM1] J. Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863-879, 1992.”

Reasoning

The evidence shows that related 1992 work exists, namely Becker & Hinton [E2] and the PMAX report that a page relates to VICReg [E5][E7]. No evidence item shows which work is reference [62] in the 2023 document, so the criteria that depend on that citation cannot be applied.

Cite this claim

Yann LeCun, “Ideas related to VICReg-style regularization were explored already in 1992, per reference [62].” (said June 2023). Verdict: unresolvable. Reliability API, claim c_7c92932f3d2e, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_7c92932f3d2e