ClaimsYann LeCun

“Probabilistic models require normalization and may therefore become intractable in the limit of high-dimensional data.”

Anna Dawid and Yann LeCun, authors June 2023 Introduction to Latent Variable Energy-Based Models (Les Houches lecture notes) (arXiv) archived copy

Factual AI capabilities

Probabilistic models require normalization and may become intractable for high-dimensional data.

Mostly true True in substance, wrong in a detail.

Factual: Statements about the present or the past, checked against the record: did they hold up?

What would count as true or false

Judge as of June 2023. True: standard machine learning sources show that probabilistic models (normalised densities or distributions) must have a normalising constant or partition function that integrates or sums to one, and that computing it can become intractable as data dimensionality grows. Mostly true: the point holds, but the claim's generality slightly overstates it, because important families of probabilistic models (for example autoregressive models, normalizing flows, tractable graphical models) are normalised by construction and avoid an intractable partition function. Given the hedge 'may', this does not by itself make the claim false. Mixed: the evidence shows intractable normalisation affects only about half of the main probabilistic model classes in practice, and the claim is presented as general. Mostly false: probabilistic models normally handle normalisation without intractability, and the problem arises only in narrow cases. False: the evidence shows that probabilistic models do not require normalisation, or that normalisation never becomes intractable in high dimensions.

Set before the research began.

Evidence (4)

  1. E1 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning)

    “This operation is intractable for many interesting models.”
  2. E2 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning)

    “As we will see in chapter 20, several deep learning models are designed to have a tractable normalizing constant, or are designed to be used in ways that do not involve computing”
  3. E3 Learning and Evaluating Boltzmann Machines University of Toronto, Department of Computer Science (UTML TR 2008-002) · 26 June 2008

    “For many problems, however, the exact calculation of the partition function or its derivatives is intractable, because it requires enumeration over an exponential number of terms.”
  4. E4 Normalizing Flows for Probabilistic Modeling and Inference Journal of Machine Learning Research (Papamakarios, Nalisnick, Rezende, Mohamed, Lakshminarayanan; DeepMind) · March 2021

    “However, unlike other probabilistic models that require approximate inference as they scale, flows usually admit analytical calculations and exact sampling even in high dimensions.”

Reasoning

A standard textbook says that computing the partition function needed for normalisation is intractable for many interesting models (E1). A Toronto technical report adds that for many problems exact calculation of the partition function is intractable because it requires enumerating an exponential number of terms (E3). However, the same textbook says several deep learning models are designed to have a tractable normalising constant (E2). A JMLR review says normalizing flows usually allow exact calculation even in high dimensions (E4). So the claim holds with the hedge 'may', but as a general statement it slightly overstates the case.

Cite this claim

Yann LeCun, “Probabilistic models require normalization and may become intractable for high-dimensional data.” (said June 2023). Verdict: mostly true. Reliability API, claim c_8394c7268c20, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_8394c7268c20

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Yann LeCun, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Yann LeCun

  1. Mostly true Factual 18 March 2024
    “For starters, transformers are equivariant to permutations.”

    Transformers are equivariant to permutations, as ConvNets are equivariant to translations.

  2. Mostly true Factual 14 March 2024
    “If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”

    There are currently no AI systems that can teach themselves to drive a car in about 20 hours of practice, as a 17-year-old can.

  3. Mostly true Factual 8 March 2024
    “The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”

    Training an encoder to reconstruct images does not lead it to produce good generic image features, though the encoder architecture is good.

  4. Mostly true Factual 8 March 2024
    “LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”

    Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.

  5. Mostly true Factual 8 March 2024
    “No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”

    LLMs, including BERT-style bidirectional ones, exploit self-supervised learning, and LeCun has been a very strong advocate of self-supervised learning for many years.

All of them

Verdicts are made automatically; how verdicts are made.