Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
67 49 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 79 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Rodney Brooks's claims
39 claims
“It has also supplied images and movies but they, and other images and movies explicitly collected for the purpose, have required human labeling.”
Rodney Brooks, blog author 29 September 2024 Tips For Building and Deploying Robots (rodneybrooks.com)
Images and movies from the web, and others collected for the purpose, have required human labeling for machine learning.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge as of 2024-09-29. True: the evidence shows that image and video datasets drawn from the web, and those collected for the purpose, used for training machine learning (for example ImageNet, COCO, video action datasets), have relied on human labelling or annotation. Mostly true: this is generally so, but some notable use of web images and video, such as self-supervised or caption-paired training (for example CLIP using alt-text), needed no explicit human labelling. Treat this as a nuance and not a refutation, since alt-text is human-written. Mixed: a large share of significant image or video ML by 2024 used no human labelling at all, so the claim holds only for part of the field. Mostly false: human labelling was the exception rather than the norm. False: the evidence shows that web images and video used in ML have not required human labelling.
Evidence
- Microsoft COCO: Common Objects in Context arXiv (ar5iv rendering)
“the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation.”
- The Kinetics Human Action Video Dataset arXiv (ar5iv rendering)
“and then using Amazon Mechanical Turkers (AMT) to decide if the clip contains the action or not.”
- CLIP: Connecting text and images OpenAI · 5 January 2021
“The ImageNet dataset, one of the largest efforts in this space, required over 25,000 workers to annotate 14 million images for 22,000 object categories.”
- CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images ECCV 2018 (Malong Technologies authors)
“The training data contains 2,439,574 images in total, but without any human annotation.”
Reasoning
The claim holds for the major image and video datasets cited. COCO was built with extensive crowd-worker annotation (E1), Kinetics clips were labelled by Amazon Mechanical Turk workers (E2), and ImageNet needed over 25,000 workers to annotate 14 million images (E3). However, CurriculumNet trained on about 2.4 million web images without any human annotation (E4). This makes the claim generally right, with the notable exceptions the criteria treat as nuance rather than refutation.
Cite this claim
Rodney Brooks, “Images and movies from the web, and others collected for the purpose, have required human labeling for machine learning.” (said 29 September 2024). Verdict: mostly true. Reliability API, claim c_b4aafbc3e6f5, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_b4aafbc3e6f5
Claims without a verdict
Is a verdict here wrong, or do you speak for Rodney Brooks? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.