Claims

Factual · AI capabilities

“The limitations you point out do not concern just backprop, but all learning algorithms that use gradient-based optimization.”

Yann LeCun, in Reddit AMA · 15 May 2014 · Yann LeCun's answers from the Reddit AMA (FastML)

The limitations the questioner pointed out concern not just backprop but all learning algorithms that use gradient-based optimization.

Unresolvable The public record could not settle it.

What would count as true or false

Judge as of 2014-05-15. First identify, from the original 2014 Reddit AMA thread, the limitations the questioner raised (e.g. local minima, vanishing gradients, need for differentiability). True: optimization and ML literature shows each of those limitations comes from gradient-based optimization in general, not from backpropagation specifically. Mostly true: this holds for the main limitations raised but not for a minor one. Mixed: about half of them are general to gradient methods and about half are specific to backprop. Mostly false: most of them are specific to backpropagation (e.g. biological implausibility or weight transport) and only a minor one is general. False: none of them are general to gradient-based optimization.

Evidence

1. 8 Optimization for Training Deep Models deeplearningbook.org (Goodfellow, Bengio & Courville) · 1 January 2012

   “local minima are no longer believed to be the primary problem for neural network optimization.”

2. Understanding the difficulty of training deep feedforward neural networks Proceedings of Machine Learning Research (AISTATS 2010) · 31 March 2010

   “Our objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks”

3. Backpropagation - Wikipedia Wikipedia

   “This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not.”

4. Backpropagation - Wikipedia Wikipedia

   “Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used, but the term is often used loosely to refer to the entire learning algorithm.”

5. Backpropagation - Wikipedia Wikipedia

   “Gradient descent with backpropagation is not guaranteed to find the global minimum of the error function, but only a local minimum; also, it has trouble crossing plateaus in the error function landscape.”

6. Backpropagation - Wikipedia Wikipedia

   “This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not.”

7. Backpropagation - Wikipedia Wikipedia

   “Some early objections were: there were no guarantees that gradient descent could reach a global minimum, only local minimum; neurons were "known" by physiologists as making discrete signals (0/1), not continuous ones, and with discrete signals, there is no gradient to take.”

8. Exponentially vanishing sub-optimal local minima in multilayer neural networks arXiv

   “SGD is only guaranteed to converge to critical points in which the gradient of the expected loss is zero (Bottou, 1998), and, specifically, to local minima (Pemantle, 1990) (this is true also for regular gradient descent (Lee et al., 2016)).”

9. Exponentially vanishing sub-optimal local minima in multilayer neural networks arXiv

   “Since loss functions parameterized by MNN weights are non-convex, it is unclear why does SGD often work well – rather than converging to sub-optimal local minima with high training error, which are known to exist (Fukumizu & Amari, 2000, Swirszcz et al., 2016).”

Reasoning

The criteria require first identifying, from the 2014 Reddit AMA thread, which limitations the questioner raised, but none of the evidence includes that thread or the question. The sources do show that some typical limitations come from gradient-based optimization in general rather than from backpropagation. Convergence to local minima or critical points is described as a property of SGD and of regular gradient descent (E8, E9). Backpropagation strictly means only computing the gradient (E4), and the need for a gradient was an early objection to the method (E7). Without the questioner's actual list of limitations, these cannot be checked one by one against the claim.

Cite this claim

Yann LeCun, “The limitations the questioner pointed out concern not just backprop but all learning algorithms that use gradient-based optimization.” (said 15 May 2014). Verdict: unresolvable. Reliability API, claim c_487d39d8a1f4, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_487d39d8a1f4