Yann LeCun

Factual · AI capabilities

“With probabilistic models, the training is confined to loss functions generated from the negative log-likelihood like the cross entropy.”

Anna Dawid and Yann LeCun, authors · June 2023 · Introduction to Latent Variable Energy-Based Models (Les Houches lecture notes) (arXiv)

With probabilistic models, training is confined to loss functions derived from the negative log-likelihood, such as cross entropy.

Mostly true True in substance, wrong in a detail.

What would count as true or false

This is a technical claim about machine learning, judged as of June 2023. True: the accepted understanding is that training a model as a probabilistic (normalized-likelihood) model means minimising negative log-likelihood or losses derived from it (cross-entropy, KL), with other losses falling outside the probabilistic framing. Mostly true: this is the standard view, with minor recognised exceptions (for example score matching or noise-contrastive estimation, which still serve as likelihood surrogates). Mixed: substantial and widely used probabilistic training methods do not derive from NLL, so the claim holds only for maximum-likelihood training. Mostly false: many standard probabilistic models are routinely trained with losses not derived from NLL (for example adversarial, score matching, proper scoring rules such as CRPS, Bayesian/variational objectives not reducible to NLL), so "confined" overstates a common practice. False: the evidence shows probabilistic models are not tied to NLL-type losses in general.

Evidence

1. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, NYU (to appear in Predicting Structured Data, MIT Press, 2006) · 19 August 2006 Own record

   “Probabilistic learning is a special case of energy-based learning where the loss function is the negative log-likelihood, a.k.a. the maximum mutual information criterion.”

2. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, NYU (to appear in Predicting Structured Data, MIT Press, 2006) · 19 August 2006 Own record

   “Estimating the parameters of a probabilistic model can be performed in a number of different ways, including maximum likelihood estimation with Bayes inversion, maximum conditional likelihood estimation, and (when possible) Bayesian averaging (possibly with variational approximations).”

3. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, NYU (to appear in Predicting Structured Data, MIT Press, 2006) · 19 August 2006 Own record

   “Energy-based methods are equally well justified as probabilistic methods. They are merely designed for training models to answer a different kind of question than probabilistic models.”

4. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, New York University (authors LeCun, Chopra, Hadsell, Ranzato, Huang; chapter for MIT Press volume 'Predicting Structured Data') · 19 August 2006 Own record

   “Estimating the parameters of a probabilistic model can be performed in a number of different ways, including maximum likelihood estimation with Bayes inversion, maximum conditional likelihood estimation, and (when possible) Bayesian averaging (possibly with variational approximations). Maximizing the conditional likelihood of the training samples is equivalent to minimizing what we called the negative log-likelihood loss.”

5. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, New York University (authors LeCun, Chopra, Hadsell, Ranzato, Huang; chapter for MIT Press volume 'Predicting Structured Data') · 19 August 2006 Own record

   “Most probabilistic models can be viewed as special types of energy-based models in which the energy function satisfies certain normalizability conditions, and in which the loss function, optimized by learning, has a particular form.”

6. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, New York University (authors LeCun, Chopra, Hadsell, Ranzato, Huang; chapter for MIT Press volume 'Predicting Structured Data') · 19 August 2006 Own record

   “Energy-based methods are equally well justified as probabilistic methods. They are merely designed for training models to answer a different kind of question than probabilistic models.”

7. A Tutorial on Energy-Based Learning Courant Institute of Mathematical Sciences, New York University (authors LeCun, Chopra, Hadsell, Ranzato, Huang; chapter for MIT Press volume 'Predicting Structured Data') · 19 August 2006 Own record

   “In this context, using an energy-based loss function other than the negative log-likelihood can be seen as a sampling method with a particular policy for picking the answers whose energy will be pulled up.”

8. Estimation of Non-Normalized Statistical Models by Score Matching Journal of Machine Learning Research (JMLR), via jmlr.csail.mit.edu · 2005

   “Here, we propose that such models can be estimated by minimizing the expected squared distance between the gradient of the log-density given by the model and the gradient of the log-density of the observed data.”

9. Sliced Score Matching: A Scalable Approach to Density and Score Estimation Ermon Group (Stanford), ermongroup.github.io

   “The de facto standard for learning a probability model is maximum likelihood (MLE). However, it is not suitable for learning unnormalized probability models, because the likelihood function”

10. Sliced Score Matching: A Scalable Approach to Density and Score Estimation Ermon Group (Stanford), ermongroup.github.io

    “Instead of using likelihood as the objective, score matching is based on the following objective”

Reasoning

An independent explainer describes maximum likelihood as the de facto standard for learning a probability model (E10), consistent with the claim that probabilistic training uses NLL-derived losses (E1, E5). However, score matching trains unnormalized probability models with a non-likelihood objective (E9, E11), and the subject's own tutorial lists Bayesian averaging, possibly with variational approximations, alongside maximum likelihood as ways to estimate probabilistic models (E3). These match the criteria's recognised minor exceptions, so "confined" slightly overstates the standard view.

Cite this claim

Yann LeCun, “With probabilistic models, training is confined to loss functions derived from the negative log-likelihood, such as cross entropy.” (said June 2023). Verdict: mostly true. Reliability API, claim c_bd7d985fd8d9, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_bd7d985fd8d9