As of 2019, the computation needed to train many current neural networks can only be provided by very large companies with very large budgets.

True: evidence from around 2019 shows that training compute for many prominent current networks (e.g. leading language and game-playing models) cost so much that in practice only very large, well-funded companies could provide it, with academic and smaller groups unable to train them. Mostly true: this holds for the largest, state-of-the-art models but 'many' slightly overstates it. Mixed: only a handful of headline models needed such resources, while many widely used networks could be trained on modest budgets. Mostly false: a few extreme cases are generalized, and evidence shows that most networks in use could be trained by universities or small labs. False: evidence shows that compute for current networks was broadly affordable without big-company budgets.

Set before the research began.

1. E1 Energy and Policy Considerations for Deep Learning in NLP ACL Anthology (Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics) · July 2019

   “Most of the models studied in this paper were developed outside academia; recent improvements in state-of-the-art accuracy are possible thanks to industry access to large-scale compute.”

2. E2 Green AI arXiv (Allen Institute for AI authors: Schwartz, Dodge, Smith, Etzioni) · July 2019

   “Grover was trained on 256 TPU chips for two weeks, at an estimated cost of $25,000.”

3. E3 Energy and Policy Considerations for Deep Learning in NLP ACL Anthology (Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics) · July 2019

   “We train all models on a single NVIDIA Titan X GPU, with the exception of ELMo which was trained on 3 NVIDIA GTX 1080 Ti GPUs.”

4. E4 Green AI arXiv (Allen Institute for AI authors: Schwartz, Dodge, Smith, Etzioni) · July 2019

   “Recent versions of AlphaGo are far more efficient [39].”

E1 says most of the models it studied were developed outside academia and that recent state-of-the-art gains came from industry access to large-scale compute. The evidence does not show that many widely used networks could be trained on modest budgets: E2 gives one $25,000 estimate for Grover without saying who could afford it, and E3 describes E1's own measurement setup rather than how the models were originally trained.