A typical Deep Learning system is like a box with 500 million knobs, 1,000 light bulbs and 10 million training images.

Judge as of 2015-02-18, as a rough characterisation ('about'). True: state-of-the-art deep learning image classifiers around 2014–2015 (e.g. ImageNet-trained convnets) were of the order of 100 million to 1 billion parameters, had about 1,000 output classes, and were trained on about 1–10 million images (ImageNet is about 1.2M images in the 1,000-class set and about 14M in full). Mostly true: two of the three figures fit and the other is off by less than about a factor of 10 (e.g. typical parameter counts of 60–150M, or about 1.2M training images). Mixed: about half of the figures fit. Mostly false: most figures are off by more than about a factor of 10, but the general picture of large parameter counts, many outputs and millions of images holds. False: typical systems were very different in all three respects.

Set before the research began.

1. E1 ImageNet Dataset Explained: Scale, ILSVRC, and Impact viso.ai · 11 October 2024

   “The classification and localization task covered 1,000 object categories and contained 1,281,167 training images, 50,000 validation images, and 100,000 test images.”

2. E2 ImageNet Dataset Explained: Scale, ILSVRC, and Impact viso.ai · 11 October 2024

   “14,197,122 images indexed across the ImageNet database.”

3. E3 3 Architectures and Transfer Learning University of the Witwatersrand course book (computer vision, chapter 3)

   “What VGG’s simplicity costs is elsewhere: 138 million parameters, of which 124 million are the dense head that Chapter 2 flagged as AlexNet’s worst habit, uncorrected.”

4. E4 3 Architectures and Transfer Learning University of the Witwatersrand course book (computer vision, chapter 3)

   “With the dense head replaced by global average pooling, the entire 22-layer network holds about 7 million parameters, a twentieth of VGG at similar accuracy.”

E1 shows the ImageNet classification task had 1,000 categories, which matches the "1,000 light bulbs" figure. It also shows 1,281,167 training images. That is within the criteria's 1–10 million range, though the criteria themselves give about 1.2M as an example of a "mostly true" near miss rather than a full match to 10 million. E2 shows the full ImageNet database held about 14 million images. For parameters, E3 gives VGG at 138 million (inside the 60–150M "mostly true" example band) and E4 gives a 22-layer network at about 7 million. So the 500 million figure is a high but same-order description of the large networks of the time, and the overall picture holds as a rough "about" characterisation.