Yann LeCun

Factual · Science and research methods

“No one uses generalization bounds to do model selection.”

Yann LeCun, in Reddit AMA · 15 May 2014 · Yann LeCun's answers from the Reddit AMA (FastML)

No one uses generalization bounds to do model selection; practitioners use cross-validation.

Mostly true True in substance, wrong in a detail.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Read 'no one' as a claim about common practice in 2014, not a literal count. Generalization bounds means theoretical bounds such as VC, Rademacher or PAC-Bayes bounds and structural risk minimization. AIC/BIC-style information criteria do not count as generalization bounds. TRUE: as of 2014, practitioners chose models mainly with cross-validation or held-out validation, and model selection driven by generalization bounds was essentially absent from practice. MOSTLY TRUE: cross-validation dominated and bound-based selection appeared only in rare or niche research uses. MOSTLY FALSE: bound-based model selection was a minority but clearly real and used in practice, so the absolute claim overstates. FALSE: bound-based model selection was common in practice, or cross-validation was not the main method.

Evidence

1. CS5339 Lecture Notes #10: Model Selection National University of Singapore (Jonathan Scarlett lecture notes) · 31 March 2021

   “One of the most widespread practical methods for choosing a model is cross-validation:”

2. CS5339 Lecture Notes #10: Model Selection National University of Singapore (Jonathan Scarlett lecture notes) · 31 March 2021

   “We can use this idea to perform model selection –”

3. A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection IJCAI (Ron Kohavi, Stanford University) · 1995

   “We review accuracy estimation methods and compare the two most common methods: cross validation and bootstrap.”

4. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation Journal of Machine Learning Research (Cawley and Talbot, University of East Anglia) · July 2010

   “the findings are quite general and apply to any model selection practice involving the optimisation of a model selection criterion evaluated over a finite sample of data, including maximisation of the Bayesian evidence and optimisation of performance bounds.”

Reasoning

Independent sources from before and after 2014 describe cross-validation as among the most common or widespread practical methods for accuracy estimation and model selection (E6, 1995; E3, 2021). A 2010 JMLR paper lists 'optimisation of performance bounds' among model selection practices (E7), so bound-based selection did exist, at least in research. The evidence supports cross-validation as the dominant method, with bound-based selection as a real but narrower practice. That fits 'mostly true' better than the literal claim that no one did it.

Cite this claim

Yann LeCun, “No one uses generalization bounds to do model selection; practitioners use cross-validation.” (said 15 May 2014). Verdict: mostly true. Reliability API, claim c_aa2fd31d04d3, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_aa2fd31d04d3