| Not researched |
Factual |
15 May 2014 |
LeCun said in May 2014 that, as far as he could tell, HTM had not been demonstrated to get anywhere close to state-of-the-art results on any serious task.
Factual · 15 May 2014 |
AI capabilities |
| Not researched |
Factual |
15 May 2014 |
Deep learning has become the dominant method for acoustic modeling in speech recognition and is quickly becoming dominant for vision tasks such as object recognition, detection and semantic segmentation.
Factual · 15 May 2014 |
AI capabilities |
| Not researched |
Factual |
2010 |
To the authors' knowledge, 0.53% is the lowest error rate reported on the original MNIST dataset without distortions or preprocessing; the best previously reported error rate was 0.60%.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Training deep multi-stage architectures with supervised gradient back-propagation requires many labeled samples.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Works [5] and [1] simplified the architecture and used back-propagation to train the entire system in a supervised fashion.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
The supervised ConvNet approach was very successful for OCR and handwriting recognition.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
SIFT and HoG are composed of filter banks followed by non-linearities and pooling (histogramming).
Factual · 2010 |
AI capabilities |
| Not researched |
Citation |
2010 |
Recent deep learning research [30],[31],[32] has shown unsupervised learning can train each stage one after another using only unlabeled data, significantly reducing the need for labeled samples.
Citation · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
In the reported experiments, comparing rows R vs R+ and U vs U+, supervised fine tuning consistently improves performance, particularly with weak non-linearities.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
A two-stage ConvNet with unsupervised pre-training and supervised refinement achieved a 0.53% test error rate on MNIST.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Accuracy of 65.5% was obtained using unsupervised pre-training and supervised refinement with abs and normalization non-linearities.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Predictive Sparse Decomposition trains a feed-forward encoder C(X,K) to quickly approximate the sparse solution Z*.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
In traditional computer vision and pattern recognition, internal representations are produced by a hand-crafted feature extractor whose output is fed to a trainable classifier.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Random filters without any filter learning achieve decent performance (62.9% for R) as long as abs and normalization are present.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
Using rectification, contrast normalization and unsupervised pre-training of each filter bank considerably reduces the need for labeled samples in ConvNets.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2010 |
The CNP developed by the authors' group achieves 10 GOPS for 7x7 kernels and implements entire programmable ConvNets including pre/postprocessing.
Factual · 2010 |
AI compute, hardware and data |
| Not researched |
Factual |
2010 |
Spatially translating the input of a ConvNet feature detection layer translates the output but leaves it otherwise unchanged.
Factual · 2010 |
AI capabilities |
| Not researched |
Factual |
2006 |
Late normalization solves several problems associated with the internal normalization of HMMs and Bayesian nets.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Probabilistic models must be properly normalized, which sometimes requires evaluating intractable integrals over all possible variable configurations.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
In the experiment, learning with the NLL loss was much faster than with the square-square loss.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The loss is minimized by placing the V's apex at Y=X^2, which raises the energies of all other answers because the V has a single minimum.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Under the NLL loss, the desired answer may have different, non-zero energies for different X.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Most probabilistic models can be viewed as energy-based models whose energy function satisfies normalizability conditions and whose loss has a particular form.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
A linear model with the NLL loss is logistic regression.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The minimum classification error (MCE) loss is non-convex.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The GTN system trained with NLL loss in LeCun et al. 1998a assigns a well-defined probability distribution over label sequences, contrary to the claim in Lafferty et al. 2001.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In speech recognition, segmentation into words and phonemes must occur simultaneously with recognition, while the correct phoneme segmentation is rarely available during training.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The perceptron, hinge and NLL losses can be directly translated to the multiclass case.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The perceptron loss has no mechanism for creating an energy gap between the correct answer and incorrect ones.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
The energy loss does not satisfy condition 3 for a general architecture.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The quality of probability estimates obtained from energies depends on how the energy is estimated from data, not on how energies are converted to probabilities.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss pulls up the energies of all incorrect answers at each iteration, including those unlikely to have lower energy than the correct answer.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
EBMs manipulate energies, so no normalization is necessary.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Because the MEE loss saturates the contribution of any single error, it is more robust to label noise and outliers but is non-convex.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The lack of a margin in the perceptron loss, which may theoretically cause stability problems, was overlooked in LeCun et al. 1998a.
Factual · 2006 |
Their own work or organisation |
| Not researched |
Factual |
2006 |
The quality of an answer in an energy-based model cannot be measured by its energy alone without considering the energies of all other answers.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For E(Y,X)=Y·Gface(X)+(1−Y)T, the minimizing Y is 1 if Gface(X)<T and 0 otherwise.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Inference in the described factor graph can be reduced to finding the shortest path in the graph.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The generalized perceptron loss has a margin of zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the NLL loss gradient, the contrastive term pulls up each answer's energy with a force proportional to that answer's likelihood under the model.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
A collapsed energy surface can be avoided by including a contrastive term in the loss function.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The perceptron loss and the LVQ2 loss satisfy condition 3 with a margin of zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
A meaningful, uncollapsed result with the perceptron loss is only guaranteed if the model cannot produce a flat energy surface.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
This loss function, though popular for regression and neural network training, cannot be used to train most architectures, because it pushes down on the energy of the desired answer without pulling up on any other energy.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
The min-sum procedure only applies to factor graphs that are bipartite trees with no loops.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The partition function over configurations of a loopy graphical model cannot be computed exactly, but loopy belief propagation or other variational methods may approximate it.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss for structured models requires summing over all label combinations, which number 2^l for a sequence of l binary labels.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The square-square loss is only suitable for energy functions bounded below by zero, because it pins the correct answer's energy at zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the tutorial's first experiment, the architecture's constraints automatically raise the energy of undesired answers while lowering that of desired answers.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
As long as d ≥ m, the loss function shown in Figure 16(b) satisfies condition 3.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The energy loss only works with architectures where pushing down the correct answer's energy automatically raises the energies of other answers.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
The generalized margin loss functions discussed satisfy condition 3.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Energy-based loss functions that pull up the most offending incorrect answer pull up only a single energy per learning iteration.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the NLL loss, the contrastive term pulls up the energies of all candidate answers.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the two-class problem, E_C and E_I correspond directly to the energies of the two classes.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss can only be used if the exponential of the negative energy is integrable over Y.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The preceding discussion of EBM inference and learning did not require manipulating normalized probability distributions.
Factual · 2006 |
AI capabilities |
| Not researched |
Citation |
2006 |
Huang and LeCun (2006) and Collobert et al. (2006) argued that convex loss functions do not guarantee good performance and that non-convex losses may be easier to optimize in practice.
Citation · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The hinge-type loss's second term prevents energy collapse by pushing up points whose energy threatens to go below that of the desired answer.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For the negative log-likelihood loss, the gradient with respect to E_C is positive and with respect to E_I is negative for any energy values.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The probability of training hitting a collapsed solution with the perceptron loss is quite small.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Since the RBF centers U1 and U2 are fixed and distinct, there is a strictly positive lower bound on d1 + d2 for all G_W.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For the squared-error regression energy, the minimum energy is always zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
With this simple architecture, the NLL or perceptron loss would give exactly the same result as the energy loss because their contrastive term is constant.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Local minima are rarely a problem in practice for gradient-based learning because of the high dimensionality of the space.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Stochastic gradient methods can be applied to any loss function, including nonconvex ones.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In many practical situations computing the partition function is intractable or impossible, such as when Y has high cardinality or the integral has no analytical solution.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
Hinge, log, LVQ2, MCE, square-square and square-exponential losses can all be described as margin losses.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
With the energy loss, nothing prevents G1 and G2 from ignoring their inputs and producing identical outputs, causing a collapsed flat energy surface.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The set of collapsed solutions is a small piece of the parameter space.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
A carefully tuned stochastic gradient descent is often considerably faster in practice than sophisticated second-order methods because it exploits redundancy between samples.
Factual · 2006 |
AI capabilities |
| Not researched |
Citation |
2006 |
Hadsell et al. (2006) used Siamese architectures with the square-square loss for unsupervised learning of manifolds.
Citation · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The loss function for the highly non-linear GTN-type parameterization is not convex in W.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
With some architectures, the energy loss can lead to a collapsed solution in which the energy is constant and equal to zero.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
Discriminative training often uses iterative gradient-based methods to optimize the loss.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For many interesting models, computing the integral over Y in the NLL loss is intractable.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the third and fourth experiments, using loss functions with explicit contrastive terms, the machine performed as expected and did not collapse.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Unlike generalized margin losses, the LVQ2 loss is non-convex in the correct and most offending incorrect energies.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
As of 2006, GTNs had never been trained with a generalized margin loss, to the authors' knowledge.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Probabilistic learning is a special case of energy-based learning in which the loss function is the negative log-likelihood, also called the maximum mutual information criterion.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For the squared-error regression energy, the Y minimizing E equals G_W(X).
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Excellent results were obtained for isolated word recognition with the LVQ2 loss despite its zero margin.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
As of 2006, no theory exists for energy-based learning with approximate inference.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The NLL loss is invariant to global shifts of energy values and depends only on energy differences among the Ys for a given X.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The dependency between X and Y sometimes cannot be expressed as a function mapping X to Y, e.g. the constraint X^2+Y^2=1.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
With the NLL loss the minimum is deeper than with the square-square loss because incorrect answers' energies are pushed up to infinity, with decreasing force.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Inference by marginalization over latent variables reduces to inference by minimization when β→∞ (zero temperature).
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In 2006, the authors said that, to their knowledge, no systematic experimental study had been published on whether second-order methods bring a significant speed improvement over well-tuned stochastic gradient methods for training these models.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Because energies are uncalibrated, there is no a priori guarantee that separately trained energy-based models have commensurate energy scales.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
One of the alleged advantages of CRFs is that their loss function is convex with respect to the parameters W.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The main problem with the perceptron loss is the absence of a margin.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Because of its zero margin, the generalized perceptron loss could lead to a collapsed energy surface.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The log loss can be seen as a soft version of the hinge loss with an infinite margin.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
Learning and inference with EBMs involve minimizing the energy over the set of answers Y and latent variables Z.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
When the energy is linear in W, the hinge loss with L2 regularizer takes the simple form of Eq. 81.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The energy and loss functions must satisfy the stated sufficient conditions to be guaranteed to work in an energy-based setting.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The NLL loss reduces to the generalized perceptron loss as β→∞, and to the log loss when Y has two elements.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The square-exponential loss has an infinite margin and pushes incorrect answers' energies to infinity with exponentially decreasing force.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Siamese architectures are used to learn similarity metrics with labeled examples.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Maximizing the conditional likelihood of training samples is equivalent to minimizing the negative log-likelihood loss.
Factual · 2006 |
AI capabilities |