Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
67 49 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 79 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Rodney Brooks's claims
39 claims
“this push to make everything learnable is pushing the cost of AI outside that of individuals or even large university departments.”
Rodney Brooks, blog author 19 March 2019 A Better Lesson (rodneybrooks.com)
The push to make everything learnable is raising the cost of AI beyond what individuals or even large university departments can afford.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judged as of March 2019. True if contemporaneous evidence shows that the compute and data costs of state-of-the-art learning-centred AI research (for example, training costs of leading deep learning or reinforcement learning models such as AlphaGo Zero, OpenAI Five or large language models of 2018–19) had risen beyond what individual researchers or typical large university departments could fund, so that frontier results were coming mainly from well-resourced companies. Mostly true if costs had clearly risen sharply and were out of reach for individuals, but the evidence shows large university departments could still afford some frontier-level work, or the link to the push for learnability is only partly supported. Mixed if the evidence is evenly split: costs were rising for some kinds of frontier work while comparable results remained affordable to academia. Mostly false if costs had risen only modestly and most frontier learning results were still achievable within university budgets, so the claim overstates a real trend. False if the evidence shows that AI costs at the time were not beyond the reach of individuals or university departments.
Evidence
- OpenAI Research | Conclusion OpenAI · 16 May 2018
“We’re releasing an analysis showing that since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.4-month doubling time”
- OpenAI Five OpenAI · 25 June 2018
“running on 256 GPUs and 128,000 CPU cores—a larger-scale version of the system we built to play the much-simpler solo variant”
- The Staggering Cost of Training SOTA AI Models | Synced Synced · 27 June 2019
“Some of the models are so big that even in MILA (Montreal Institute of Learning Algorithm) we can’t run them because we don’t have the infrastructure for that. Only a few companies can run these very big models they’re talking about.”
- The Staggering Cost of Training SOTA AI Models | Synced Synced · 27 June 2019
“Google suggests researchers with tight budgets could pretrain a smaller BERT-Base model on a single preemptible Cloud TPU v2, which takes about two weeks with a cost of about US$500.”
Reasoning
E1 reports that since 2012, the compute used in the largest AI training runs had grown exponentially, doubling every 3.4 months. E2 describes OpenAI Five running on 256 GPUs and 128,000 CPU cores. E3, from June 2019, quotes a researcher saying even MILA (Montreal Institute of Learning Algorithms) lacked the infrastructure to run some very large models and that only a few companies could run them. E4, however, reports Google's suggestion that researchers with tight budgets could pretrain a smaller BERT-Base model for about US$500, so some work stayed affordable, and the evidence ties the cost rise to the push for learnability only indirectly.
Cite this claim
Rodney Brooks, “The push to make everything learnable is raising the cost of AI beyond what individuals or even large university departments can afford.” (said 19 March 2019). Verdict: mostly true. Reliability API, claim c_ad1de70f030a, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_ad1de70f030a
Claims without a verdict
Is a verdict here wrong, or do you speak for Rodney Brooks? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.