ReLU nonlinearities and dropout regularization enabled purely supervised backprop training of very deep networks with a dozen layers.

True if the literature as of 2019 shows that ReLU nonlinearities and dropout regularization were key enablers that made it possible to train deep networks of about a dozen layers with purely supervised backpropagation, with no unsupervised pre-training (for example Glorot et al. 2011, Krizhevsky et al. 2012, Hinton/Srivastava dropout papers). Mostly true if this is broadly right but a detail differs, such as the typical depth being somewhat different from 'a dozen' or other factors (GPUs, large datasets) being equally credited. Mixed if only one of the two techniques (ReLU or dropout) is supported as such an enabler. Mostly false if the evidence shows the techniques played only a minor role, or that such networks still needed pre-training at the time. False if the evidence shows these techniques did not contribute to supervised training of deep networks.

Set before the research began.

1. E1 Deep Sparse Rectifier Neural Networks PMLR (Proceedings of Machine Learning Research) · 14 June 2011

   “Even though they can take advantage of semi-supervised setups with extra-unlabeled data, deep rectifier networks can reach their best performance without requiring any unsupervised pre-training on purely supervised tasks with large labeled datasets.”

2. E2 Deep learning - Nature Nature · 27 May 2015 Own record

   “This paper showed that supervised training of very deep neural networks is much faster if the hidden layers are composed of ReLU.”

3. E3 ImageNet Classification with Deep Convolutional Neural Networks NeurIPS Proceedings · 2012

   “To make training faster, we used non-saturating neurons and a very efficient GPU implementation of convolutional nets.”

4. E4 Dropout: A Simple Way to Prevent Neural Networks from Overfitting Journal of Machine Learning Research (JMLR) · 2014

   “We show that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.”

E1 (Glorot et al. 2011) says deep rectifier networks reach their best performance on purely supervised tasks without unsupervised pre-training. That directly supports ReLU as an enabler. E3 (Krizhevsky et al. 2012) credits non-saturating neurons together with an efficient GPU implementation, so GPUs share the credit. E4 (Srivastava et al. 2014) shows that dropout improves supervised neural networks across many benchmarks, which supports dropout as a contributor, but none of these quotes confirms a typical depth of 'a dozen layers'.