Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
67 49 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 79 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Rodney Brooks's claims
42 claims
“Today’s data sets can have billions of examples, where a human may only require a handful to learn the same thing.”
Rodney Brooks, blog author 19 March 2019 A Better Lesson (rodneybrooks.com)
Today's machine learning data sets can contain billions of examples, whereas a human may need only a handful to learn the same thing.
Resolved true The record shows it happened or was so.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Two parts, judged as of March 2019: (1) machine learning datasets then in use contained billions of examples; (2) humans can often learn the same kind of concept from a handful of examples. True if published datasets of that period with about a billion or more examples existed (for example, Instagram's 3.5 billion images, Google JFT-300M plus large industrial or text corpora counted in tokens) and the cognitive science literature supports few-shot human learning. Mostly true if the largest datasets were in the hundreds of millions, short of billions. Mixed if one part fails. Mostly false if the datasets were well below a billion and the human comparison is overstated. False if both are wrong.
Evidence
- Human-level concept learning through probabilistic program induction Science (AAAS); authors Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum · 11 December 2015
“People learning new concepts can often generalize successfully from just a single example, yet machine learning algorithms typically require tens or hundreds of examples to perform with similar accuracy.”
- Human-level concept learning through probabilistic program induction Science (Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum; affiliated with NYU, University of Toronto, MIT) · 11 December 2015
“People learning new concepts can often generalize successfully from just a single example, yet machine learning algorithms typically require tens or hundreds of examples to perform with similar accuracy.”
- Human-level concept learning through probabilistic program induction Science (Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum; affiliated with NYU, University of Toronto, MIT) · 11 December 2015
“Whereas machine learning algorithms are typically evaluated after hundreds or thousands of training examples per class (5), we evaluated the tasks”
- Exploring the Limits of Weakly Supervised Pretraining arXiv (ar5iv HTML rendering); authors are Facebook AI researchers
“In our experiments, we train standard convolutional network architectures to predict hashtags on up to 3.5 billion public Instagram images.”
- Revisiting Unreasonable Effectiveness of Data in Deep Learning Era Open Access version of ICCV 2017 paper (Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta; Google Research and CMU)
“The JFT dataset has more than 300M images that are labeled with 18291 categories.”
- Challenges for Human-Level Intelligence Harvard Data Science Review (MIT Press) · 4 October 2019 Own record
“More than this, we also know how humans are able to infer complex categories from just a handful of exposures to instances of the category.”
- Fair Learning Texas Law Review (draft by Mark A. Lemley and Bryan Casey), hosted by Berkeley Law · 28 September 2020
“ML systems must first be exposed to thousands, millions, or even billions of examples—all of which are collected and stored”
Reasoning
Facebook AI researchers report training on up to 3.5 billion public Instagram images (E4, arXiv 1805.00932), which is a billion-scale dataset of the kind the criteria name. A Google paper describes the JFT dataset as having more than 300M images (E5). The cognitive science paper by Lake, Salakhutdinov and Tenenbaum says people can often generalize from just a single example, while machine learning typically needs far more (E1, E3). That supports the claim that humans can learn from a handful of examples. Both parts of the criteria are met.
Cite this claim
Rodney Brooks, “Today's machine learning data sets can contain billions of examples, whereas a human may need only a handful to learn the same thing.” (said 19 March 2019). Verdict: resolved true. Reliability API, claim c_f05cef8321da, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_f05cef8321da
Claims without a verdict
Is a verdict here wrong, or do you speak for Rodney Brooks? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.