Convolutional nets were always the record holder on MNIST, and SVMs never really managed to beat them there.

Judge as of 2014-05-15. The claim says that at every point in MNIST's history up to then, the best reported test error was held by a convolutional network, and that an SVM never held it. Compare dated best-reported results from the MNIST benchmark history: LeCun's MNIST page and the original papers. True: at every date from the benchmark's introduction to May 2014, the lowest published error belonged to a convolutional-net method. Variants that combine convnets with distortions, ensembles or boosting still count as convnets. No SVM or other non-convolutional method ever had a lower published error than the best convnet result published by that date. Mostly true: convnets held the record throughout, except that a non-convnet method briefly matched it (a tie) or led only by a negligible amount within the same reporting precision. No SVM led outright. Mixed: the answer depends on the category. Convnets held the overall record throughout, but an SVM or other method held the best result in a commonly reported sub-category (for example, no data augmentation, or no prior knowledge) for a substantial period. Mostly false: an SVM (for example a virtual or deskewed SVM) or another non-convnet method held the overall lowest published error for some period, even though convnets held it most of the time. False: non-convnet methods held the overall record for long stretches, or convnets were not the record holder for most of the period.

Set before the research began.

1. E1 Training Invariant Support Vector Machines Machine Learning (Springer / Kluwer), 46, 161–190 · 2002

   “One of the significant new results reported in this work is our recent achievement of the lowest reported test error on the well-known MNIST digit recognition benchmark task, with SVM training”

2. E2 Training Invariant Support Vector Machines Machine Learning (Springer / Kluwer), 46, 161–190 · 2002

   “Prior to the present work, the best system on the MNIST set was a boosted ensemble of LeNet4 neural networks, trained on a huge database of artificially generated virtual examples.”

3. E3 MNIST database - Wikipedia Wikipedia · 17 August 2013

   “Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent.”

4. E4 HeroSvm 2.1 | Centre for Pattern Recognition and Machine Intelligence - Concordia University CENPARMI, Concordia University · 4 August 2005

   “| VSVMb [5][21] | 0.38 |”

The 2002 peer-reviewed paper on invariant SVMs [E1] reports that its SVM training reached the lowest reported test error on MNIST. According to the same paper [E2], the previous best system was a boosted ensemble of LeNet4 neural networks. The paper reports that its SVM achieved the lowest reported MNIST test error at the time [E1], which, if accurate, goes against the claim that convnets were always the record holder. The evidence does not show that SVMs led for long stretches, and E3 and E4 do not clearly name methods, dates or what their figures measure. That fits the 'mostly false' category better than 'false'.