Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
110 claims · page 2 of 2
“The limitations you point out do not concern just backprop, but all learning algorithms that use gradient-based optimization.”
Yann LeCun, in Reddit AMA 15 May 2014 Yann LeCun's answers from the Reddit AMA (FastML)
The limitations the questioner pointed out concern not just backprop but all learning algorithms that use gradient-based optimization.
Unresolvable The public record could not settle it.
What would count as true or false
Judge as of 2014-05-15. First identify, from the original 2014 Reddit AMA thread, the limitations the questioner raised (e.g. local minima, vanishing gradients, need for differentiability). True: optimization and ML literature shows each of those limitations comes from gradient-based optimization in general, not from backpropagation specifically. Mostly true: this holds for the main limitations raised but not for a minor one. Mixed: about half of them are general to gradient methods and about half are specific to backprop. Mostly false: most of them are specific to backpropagation (e.g. biological implausibility or weight transport) and only a minor one is general. False: none of them are general to gradient-based optimization.
Evidence
- 8 Optimization for Training Deep Models deeplearningbook.org (Goodfellow, Bengio & Courville) · 1 January 2012
“local minima are no longer believed to be the primary problem for neural network optimization.”
- Understanding the difficulty of training deep feedforward neural networks Proceedings of Machine Learning Research (AISTATS 2010) · 31 March 2010
“Our objective here is to understand better why standard gradient descent from random initialization is doing so poorly with deep neural networks”
- Backpropagation - Wikipedia Wikipedia
“This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not.”
- Backpropagation - Wikipedia Wikipedia
“Strictly speaking, the term backpropagation refers only to an algorithm for efficiently computing the gradient, not how the gradient is used, but the term is often used loosely to refer to the entire learning algorithm.”
- Backpropagation - Wikipedia Wikipedia
“Gradient descent with backpropagation is not guaranteed to find the global minimum of the error function, but only a local minimum; also, it has trouble crossing plateaus in the error function landscape.”
- Backpropagation - Wikipedia Wikipedia
“This issue, caused by the non-convexity of error functions in neural networks, was long thought to be a major drawback, but Yann LeCun et al. argue that in many practical problems, it is not.”
- Backpropagation - Wikipedia Wikipedia
“Some early objections were: there were no guarantees that gradient descent could reach a global minimum, only local minimum; neurons were "known" by physiologists as making discrete signals (0/1), not continuous ones, and with discrete signals, there is no gradient to take.”
- Exponentially vanishing sub-optimal local minima in multilayer neural networks arXiv
“SGD is only guaranteed to converge to critical points in which the gradient of the expected loss is zero (Bottou, 1998), and, specifically, to local minima (Pemantle, 1990) (this is true also for regular gradient descent (Lee et al., 2016)).”
- Exponentially vanishing sub-optimal local minima in multilayer neural networks arXiv
“Since loss functions parameterized by MNN weights are non-convex, it is unclear why does SGD often work well – rather than converging to sub-optimal local minima with high training error, which are known to exist (Fukumizu & Amari, 2000, Swirszcz et al., 2016).”
Reasoning
The criteria require first identifying, from the 2014 Reddit AMA thread, which limitations the questioner raised, but none of the evidence includes that thread or the question. The sources do show that some typical limitations come from gradient-based optimization in general rather than from backpropagation. Convergence to local minima or critical points is described as a property of SGD and of regular gradient descent (E8, E9). Backpropagation strictly means only computing the gradient (E4), and the need for a gradient was an early objection to the method (E7). Without the questioner's actual list of limitations, these cannot be checked one by one against the claim.
Cite this claim
Yann LeCun, “The limitations the questioner pointed out concern not just backprop but all learning algorithms that use gradient-based optimization.” (said 15 May 2014). Verdict: unresolvable. Reliability API, claim c_487d39d8a1f4, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_487d39d8a1f4
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.