Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
23 claims
“These methods have dramatically improved the state-of-the-art in speech recognition, visual object recognition, object detection”
LeCun, Bengio and Hinton, article abstract 27 May 2015 Deep learning (Nature) (Nature)
Deep learning methods have dramatically improved the state of the art in speech recognition, visual object recognition and object detection.
Resolved true The record shows it happened or was so.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2015-05-27. The claim has three named fields. True: published benchmark results by 2015 show deep learning methods greatly improved the state of the art in all three of speech recognition (for example, the 2012 DNN acoustic-model results), visual object recognition (for example, ImageNet 2012 onward) and object detection (for example, R-CNN on PASCAL VOC). Mostly true: large improvements are documented in all three, but in one of them the gain was more modest than 'dramatically' suggests. Mixed: dramatic improvement is documented in only about half of the fields. Mostly false: improvements are documented in only one field, or were marginal. False: deep learning did not improve the state of the art in these fields.
Evidence
- Deep Neural Networks for Acoustic Modeling in Speech Recognition IEEE Signal Processing Magazine (proof copy hosted at cs.toronto.edu) · November 2012
“The DNNs worked well on all of these tasks when compared with highly tuned GMM-HMM systems, and on some of the tasks they outperformed the state of the art by a large margin.”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition IEEE Signal Processing Magazine (proof copy hosted at cs.toronto.edu) · November 2012
“Mohamed et. al. [12] showed that a DBN-DNN acoustic model outperformed the best published recognition results on TIMIT at about the same time as Sainath et. al. [23] achieved a similar improvement on TIMIT”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition IEEE Signal Processing Magazine (proof copy hosted at cs.toronto.edu) · November 2012
“The DBN-DNN gave an absolute improvement of 4.7% over the baseline system’s WER of 52.3%.”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition IEEE Signal Processing Magazine (proof copy hosted at cs.toronto.edu) · November 2012
“Experience has shown that performance improvements on TIMIT do not necessarily translate into performance improvements on large vocabulary tasks with less controlled recording conditions”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition (IEEE Proof) IEEE Signal Processing Magazine (review by four research groups: Hinton, Deng, Yu, Dahl, Mohamed, Kingsbury; Jaitly, Senior, Vanhoucke, Nguyen, Sainath) · November 2012
“Deep neural networks (DNNs) that have many hidden layers and are trained using new methods have been shown to outperform GMMs on a variety of speech recognition benchmarks, sometimes by a large margin.”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition (IEEE Proof) IEEE Signal Processing Magazine (review by four research groups: Hinton, Deng, Yu, Dahl, Mohamed, Kingsbury; Jaitly, Senior, Vanhoucke, Nguyen, Sainath) · November 2012
“Detailed experiments [43] on the Switchboard task confirmed that the remarkable accuracy gains from the DNN-HMM acoustic model are due to the direct modeling of tied triphone states using the DBN-DNN”
- Deep Neural Networks for Acoustic Modeling in Speech Recognition (IEEE Proof) IEEE Signal Processing Magazine (review by four research groups: Hinton, Deng, Yu, Dahl, Mohamed, Kingsbury; Jaitly, Senior, Vanhoucke, Nguyen, Sainath) · November 2012
“The DBN-DNN gave an absolute improvement of 4.7% over the baseline system's WER of 52.3%.”
- Rich Feature Hierarchies for Accurate Object Detection and ... CVPR 2014 (Open Access, CVF); authors Girshick, Donahue, Darrell, Malik (UC Berkeley and ICSI)
“propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30%”
- ImageNet Large Scale Visual Recognition Challenge arXiv (Russakovsky et al.), ar5iv rendering
“There has been a 4.2x reduction in image classification error (from \(28.2\%\) to \(6.7\%\)) and a 1.7x reduction in single-object localization error (from \(42.5\%\) to \(25.3\%\)) since the beginning of the challenge.”
- ImageNet Large Scale Visual Recognition Challenge arXiv (Russakovsky et al.), ar5iv rendering
“Object detection accuracy as measured by the mean average precision (mAP) has increased 1.9x since the introduction of this task, from \(22.6\%\) mAP in ILSVRC2013 to \(43.9\%\) mAP in ILSVRC2014.”
- AI-metrics-data.txt - GitHub AI-metrics repository (compiled by Miles Brundage and Jack Clark)
“April, 2012: 18.5% SWB Who: University of Toronto, Google, IBM, Microsoft”
Reasoning
For speech recognition, the 2012 multi-group review reports that DNN acoustic models beat highly tuned GMM-HMM systems on a range of benchmarks, outperforming the state of the art "by a large margin" on some, with "remarkable accuracy gains" on Switchboard (E1, E5, E6). For visual object recognition, the ILSVRC paper reports a 4.2x reduction in image classification error since the challenge began (E9). For object detection, the R-CNN paper (titled "Rich Feature Hierarchies") reports a gain of more than 30% in mAP, and ILSVRC detection mAP rose 1.9x from 2013 to 2014 (E8, E10). E9 and E10 do not credit those gains to deep learning in so many words, but they are the ImageNet results the criteria name, and together with E8 they show large improvements in all three fields by 2015.
Cite this claim
Yann LeCun, “Deep learning methods have dramatically improved the state of the art in speech recognition, visual object recognition and object detection.” (said 27 May 2015). Verdict: resolved true. Reliability API, claim c_896191212321, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_896191212321
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.