Probabilistic models require normalization and may become intractable for high-dimensional data.
Mostly true True in substance, wrong in a detail.
Factual: Statements about the present or the past, checked against the record: did they hold up?
What would count as true or false
Judge as of June 2023. True: standard machine learning sources show that probabilistic models (normalised densities or distributions) must have a normalising constant or partition function that integrates or sums to one, and that computing it can become intractable as data dimensionality grows. Mostly true: the point holds, but the claim's generality slightly overstates it, because important families of probabilistic models (for example autoregressive models, normalizing flows, tractable graphical models) are normalised by construction and avoid an intractable partition function. Given the hedge 'may', this does not by itself make the claim false. Mixed: the evidence shows intractable normalisation affects only about half of the main probabilistic model classes in practice, and the claim is presented as general. Mostly false: probabilistic models normally handle normalisation without intractability, and the problem arises only in narrow cases. False: the evidence shows that probabilistic models do not require normalisation, or that normalisation never becomes intractable in high dimensions.
Set before the research began.
Evidence (4)
-
E1 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning)
“This operation is intractable for many interesting models.”
-
E2 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning)
“As we will see in chapter 20, several deep learning models are designed to have a tractable normalizing constant, or are designed to be used in ways that do not involve computing”
-
E3 Learning and Evaluating Boltzmann Machines University of Toronto, Department of Computer Science (UTML TR 2008-002) · 26 June 2008
“For many problems, however, the exact calculation of the partition function or its derivatives is intractable, because it requires enumeration over an exponential number of terms.”
-
E4 Normalizing Flows for Probabilistic Modeling and Inference Journal of Machine Learning Research (Papamakarios, Nalisnick, Rezende, Mohamed, Lakshminarayanan; DeepMind) · March 2021
“However, unlike other probabilistic models that require approximate inference as they scale, flows usually admit analytical calculations and exact sampling even in high dimensions.”
Reasoning
A standard textbook says that computing the partition function needed for normalisation is intractable for many interesting models (E1). A Toronto technical report adds that for many problems exact calculation of the partition function is intractable because it requires enumerating an exponential number of terms (E3). However, the same textbook says several deep learning models are designed to have a tractable normalising constant (E2). A JMLR review says normalizing flows usually allow exact calculation even in high dimensions (E4). So the claim holds with the hedge 'may', but as a general statement it slightly overstates the case.
Cite this claim
Yann LeCun, “Probabilistic models require normalization and may become intractable for high-dimensional data.” (said June 2023). Verdict: mostly true. Reliability API, claim c_8394c7268c20, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_8394c7268c20
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Yann LeCun
-
Mostly true
“For starters, transformers are equivariant to permutations.”
Transformers are equivariant to permutations, as ConvNets are equivariant to translations.
-
Mostly true
“If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”
-
Mostly true
“The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”
-
Mostly true
“LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”
-
Mostly true
“No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”
Verdicts are made automatically; how verdicts are made.