| Not checkable |
Not checkable |
26 May 2014 |
Only the first is interesting to me (the other uses are interesting too, just not on my own radar screen).
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Proofs always come later.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I like the idea of Boltzmann machines (it’s a beautifully simple concept) but it doesn’t scale well.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
The theory of deep learning is a wide open field.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Ask them what error rate they get on MNIST or ImageNet.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
You can construct pathological cases where the objective function is like a golf course: flat with a tiny hole somewhere.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
DeconvNets are the generative counterpart of feed-forward ConvNets.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Learning with temporal/sequential signals: language, video, speech.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Seriously, I don’t like the phrase “Big Data”.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
They have the advantage of scaling well.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Jeff Hawkins has the right intuition and the right philosophy.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Few of us believe that we have found a good solution to unsupervised learning.
Not checkable · 26 May 2014 |
|
| Unresolvable |
Factual |
15 May 2014 |
Theoretical results suggest that learning good parameters for a small neural network can be as computationally hard as breaking RSA.
Factual · 15 May 2014 |
Philosophy, ethics and society |
| Mostly true |
Factual |
15 May 2014 |
The notions of partition functions, entropy, free energy and variational methods prevalent in the graphical models literature come from statistical physics.
Factual · 15 May 2014 |
Science and research methods |
| Not researched |
Factual |
15 May 2014 |
SVMs do not naturally emerge from VC theory; they are simple enough for VC theory to give specific results about them.
Factual · 15 May 2014 |
Science and research methods |
| Unresolvable |
Factual |
15 May 2014 |
The limitations the questioner pointed out concern not just backprop but all learning algorithms that use gradient-based optimization.
Factual · 15 May 2014 |
AI capabilities |
| Mostly true |
Factual |
15 May 2014 |
The first commercially viable check reading system, deployed by AT&T/NCR in 1996, used a ConvNet rather than an SVM.
Factual · 15 May 2014 |
AI capabilities |
| Mostly false |
Factual |
15 May 2014 |
Convolutional nets were always the record holder on MNIST, and SVMs never really managed to beat them there.
Factual · 15 May 2014 |
AI capabilities |
| Unresolvable |
Prediction |
15 May 2014 |
Researchers will eventually figure out how to merge ConvNets and DeconvNets into a feed-forward plus feed-back system trainable supervised or unsupervised.
Prediction · 15 May 2014 |
AI capabilities |
| Mostly true |
Factual |
15 May 2014 |
No one uses generalization bounds to do model selection; practitioners use cross-validation.
Factual · 15 May 2014 |
Science and research methods |
| Unresolvable |
Prediction |
15 May 2014 |
Now or in the near future, most of the world's knowledge will be extracted by machines and reside in machines.
Prediction · 15 May 2014 |
AI timelines and forecasting |
| Not researched |
Factual |
15 May 2014 |
LeCun said in May 2014 that, as far as he could tell, HTM had not been demonstrated to get anywhere close to state-of-the-art results on any serious task.
Factual · 15 May 2014 |
AI capabilities |
| Unresolvable |
Prediction |
15 May 2014 |
As more data becomes available, machine learning methods will increasingly rely on learning.
Prediction · 15 May 2014 |
AI capabilities |
| Not researched |
Factual |
15 May 2014 |
Deep learning has become the dominant method for acoustic modeling in speech recognition and is quickly becoming dominant for vision tasks such as object recognition, detection and semantic segmentation.
Factual · 15 May 2014 |
AI capabilities |
| Mostly true |
Factual |
15 May 2014 |
Speech is a domain with access to ridiculously large amounts of data and a very large number of categories.
Factual · 15 May 2014 |
AI capabilities |
| Unresolvable |
Prediction |
15 May 2014 |
Natural language processing is the next frontier for deep learning.
Prediction · 15 May 2014 |
AI timelines and forecasting |
| Mostly true |
Factual |
15 May 2014 |
Facebook AI Research uses Torch7 for many projects, as do Deep Mind and several groups at Google.
Factual · 15 May 2014 |
Technology |
| Not checkable |
Not checkable |
2010 |
Implementing ConvNets in software is best achieved using the modular, object-oriented approach suggested in [2].
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
It is clear that abs and normalization are cruciala for achieving good performance.
Not checkable · 2010 |
|
| Not researched |
Factual |
2010 |
To the authors' knowledge, 0.53% is the lowest error rate reported on the original MNIST dataset without distortions or preprocessing; the best previously reported error rate was 0.60%.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2010 |
We describe new unsupervised learning algorithms, and new non-linear stages that allow ConvNets to be trained with very few labeled samples.
Not checkable · 2010 |
|
| Not researched |
Factual |
2010 |
Training deep multi-stage architectures with supervised gradient back-propagation requires many labeled samples.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Works [5] and [1] simplified the architecture and used back-propagation to train the entire system in a supervised fashion.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
The supervised ConvNet approach was very successful for OCR and handwriting recognition.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
SIFT and HoG are composed of filter banks followed by non-linearities and pooling (histogramming).
Factual · 2010 |
AI capabilities |
| Mostly true |
Prediction |
2010 |
In the next decade, a major challenge of computer vision, machine learning and AI will be devising methods that learn good feature hierarchies from unlabeled and labeled data in an integrated fashion.
Prediction · 2010 |
AI capabilities |
| Not researched |
Citation |
2010 |
Recent deep learning research [30],[31],[32] has shown unsupervised learning can train each stage one after another using only unlabeled data, significantly reducing the need for labeled samples.
Citation · 2010 |
AI capabilities |
| Unresolvable |
Prediction |
2010 |
The authors expected to see many new embedded vision systems based on ConvNets in the next few years.
Prediction · 2010 |
AI compute, hardware and data |
| Not checkable |
Not checkable |
2010 |
The Convolutional Network architecture is a remarkably versatile, yet conceptually simple paradigm that can be applied to a wide spectrum…
Not checkable · 2010 |
|
| Resolved true |
Factual |
2010 |
Each stage of a ConvNet is composed of a filter bank, some non-linearities, and feature pooling layers.
Factual · 2010 |
AI capabilities |
| Resolved true |
Factual |
2010 |
The idea behind multi-stage Hubel-Wiesel architectures is rooted in Hubel and Wiesel's 1962 work on the cat's primary visual cortex.
Factual · 2010 |
Science and research methods |
| Not researched |
Factual |
2010 |
In the reported experiments, comparing rows R vs R+ and U vs U+, supervised fine tuning consistently improves performance, particularly with weak non-linearities.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
A two-stage ConvNet with unsupervised pre-training and supervised refinement achieved a 0.53% test error rate on MNIST.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Accuracy of 65.5% was obtained using unsupervised pre-training and supervised refinement with abs and normalization non-linearities.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2010 |
Current and future research will focus on performing unsupervised learning on multiple stages simultaneously, on the integration of…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
The unsupervised method we propose, to learn the filter coefficients in the filter bank layers, is called Predictive Sparse Decomposition…
Not checkable · 2010 |
|
| Mostly true |
Factual |
2010 |
A face detection application on the CNP system was demonstrated achieving 10 fps on VGA images.
Factual · 2010 |
AI compute, hardware and data |
| Not researched |
Factual |
2010 |
Predictive Sparse Decomposition trains a feed-forward encoder C(X,K) to quickly approximate the sparse solution Z*.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2010 |
Good internal representations are hierarchical.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
This is an extremely important fact for users of convolutional networks, which traditionally only use tanh().
Not checkable · 2010 |
|
| Not researched |
Factual |
2010 |
In traditional computer vision and pattern recognition, internal representations are produced by a hand-crafted feature extractor whose output is fed to a trainable classifier.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2010 |
Convolutional Networks (ConvNets) are an answer to the first question.
Not checkable · 2010 |
|
| Mixed |
Factual |
2010 |
ConvNets have been successfully deployed in many commercial applications from OCR to video surveillance, but require large amounts of labeled training samples.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Random filters without any filter learning achieve decent performance (62.9% for R) as long as abs and normalization are present.
Factual · 2010 |
AI capabilities |
| Mostly true |
Factual |
2010 |
Convolutional Networks are a biologically inspired trainable architecture that can learn invariant features.
Factual · 2010 |
AI capabilities |
| Unresolvable |
Factual |
2010 |
A big advantage of ConvNets for detection is their computational efficiency.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2010 |
One of the key questions of Vision Science (natural and artificial) is how to produce good internal representations of the visual world.
Not checkable · 2010 |
|
| Not researched |
Factual |
2010 |
Using rectification, contrast normalization and unsupervised pre-training of each filter bank considerably reduces the need for labeled samples in ConvNets.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
The CNP developed by the authors' group achieves 10 GOPS for 7x7 kernels and implements entire programmable ConvNets including pre/postprocessing.
Factual · 2010 |
AI compute, hardware and data |
| Not checkable |
Not checkable |
2010 |
Convolutional Networks [1], [2] are trainable multistage architectures composed of multiple stages.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
This suggests that recognition architectures for vision (and for other modalities such as audio and natural language) should have multiple…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Intelligent tasks, such as visual perception, auditory perception, and language understanding require the construction of good internal…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
ConvNets are perfect candidates for hardware implementations, and embedded applications, as demonstrated by the increasing amount of work…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
A major question for Machine Learning is how to learn such good features automatically.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
It seems that unsupervised pre-training (U , U +) is crucial when newly proposed non-linearities are not in place.
Not checkable · 2010 |
|
| Not researched |
Factual |
2010 |
Spatially translating the input of a ConvNet feature detection layer translates the output but leaves it otherwise unchanged.
Factual · 2010 |
AI capabilities |
| Resolved true |
Factual |
2010 |
In many problems labeled data is scarce whereas unlabeled data is abundant.
Factual · 2010 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
In the following, we use the convention that small energy values correspond to highly compatible configurations of the variables, while…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Late normalization solves several problems associated with the internal normalization of HMMs and Bayesian nets.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
For decision-making tasks, such as steering a robot, it is merely necessary that the system give the lowest energy to the correct answer.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over all possible variable configurations.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
Among the many loss functions proposed in the literature, some are good (with a non-zero margin), and some can be bad.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
In the experiment, learning with the NLL loss was much faster than with the square-square loss.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The loss is minimized by placing the V's apex at Y=X^2, which raises the energies of all other answers because the V has a single minimum.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
However, the convexity of the loss function, while mathematically satisfying, does not seem to be a significant practical advantage.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Under the NLL loss, the desired answer may have different, non-zero energies for different X.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
For example, there is no reason to believe that the model in Figure 7 is normalizable over Y .
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Most probabilistic models can be viewed as energy-based models whose energy function satisfies normalizability conditions and whose loss has a particular form.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
A linear model with the NLL loss is logistic regression.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The minimum classification error (MCE) loss is non-convex.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
A common approach is to view variational methods and energy-based loss functions as approximations to the probabilistic method.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The GTN system trained with NLL loss in LeCun et al. 1998a assigns a well-defined probability distribution over label sequences, contrary to the claim in Lafferty et al. 2001.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In speech recognition, segmentation into words and phonemes must occur simultaneously with recognition, while the correct phoneme segmentation is rarely available during training.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
In a sense, variational methods could be interpreted in the context of EBM as a way to choose a subset of energies to pull up.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The perceptron, hinge and NLL losses can be directly translated to the multiclass case.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
One advantage of the energy-based approach is that it puts very little restrictions on the nature of E.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The perceptron loss has no mechanism for creating an energy gap between the correct answer and incorrect ones.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
To some observers, the recent interest in the linear structured model looks like somewhat of a throw-back to the past, and a regression on…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The energy loss does not satisfy condition 3 for a general architecture.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The quality of probability estimates obtained from energies depends on how the energy is estimated from data, not on how energies are converted to probabilities.
Factual · 2006 |
Science and research methods |
| Mostly true |
Factual |
2006 |
The hinge loss with G_W(X)=W^T X and a ||W||^2 regularizer gives the linear support vector machine.
Factual · 2006 |
AI capabilities |
| Mostly true |
Factual |
2006 |
The hinge loss systematically chooses the most offending incorrect answer as the one whose energy is pulled up.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss pulls up the energies of all incorrect answers at each iteration, including those unlikely to have lower energy than the correct answer.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
EBMs manipulate energies, so no normalization is necessary.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Energy minimization is a convenient way to represent the general process of reasoning and inference.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Any prior knowledge we may have about the task at hand is embedded into the architecture and into the loss function (particularly the…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Because the MEE loss saturates the contribution of any single error, it is more robust to label noise and outliers but is non-convex.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The lack of a margin in the perceptron loss, which may theoretically cause stability problems, was overlooked in LeCun et al. 1998a.
Factual · 2006 |
Their own work or organisation |
| Not researched |
Factual |
2006 |
The quality of an answer in an energy-based model cannot be measured by its energy alone without considering the energies of all other answers.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
We should emphasize again that this is inconsequential if the model is to be used for prediction, classification, or decision-making.
Not checkable · 2006 |
|