| Not researched |
Factual |
2006 |
For E(Y,X)=Y·Gface(X)+(1−Y)T, the minimizing Y is 1 if Gface(X)<T and 0 otherwise.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
The best inference procedure to use often depends on the internal structure of the model.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Inference in the described factor graph can be reduced to finding the shortest path in the graph.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
It is not obvious that the negative log-likelihood loss satisfies condition 3.
Not checkable · 2006 |
|
| Resolved true |
Factual |
2006 |
The generalized perceptron loss is always positive because its second term is a lower bound on its first term.
Factual · 2006 |
Science and research methods |
| Mostly true |
Factual |
2006 |
Support vector Markov models, max-margin Markov networks and conditional random fields are all sequence modeling systems that use linearly parameterized energy factors.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
However, there is no guarantee that all incorrect answers with low energy will be pulled up.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The generalized perceptron loss has a margin of zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the NLL loss gradient, the contrastive term pulls up each answer's energy with a force proportional to that answer's likelihood under the model.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Though this is a trivial problem for a learning machine, it is useful for demonstrating the issues involved in the design of an energy…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
A collapsed energy surface can be avoided by including a contrastive term in the loss function.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
The energy loss is a bad loss function in general, but there are certain forms of energies for which it is a good loss function.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Energy-based learning provides a unified framework for many probabilistic and non-probabilistic approaches to learning, particularly for…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The perceptron loss and the LVQ2 loss satisfy condition 3 with a margin of zero.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
We conclude that this is a good loss function.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
A GTN can be seen as a multilayer architecture in which the states are trellises, just as a neural net is a multilayer architecture in…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
While the zero margin may limit the robustness of the solution, the perceptron loss seems appropriate as a way to refine a system that was…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
This is best handled by hierarchical, multi-layer architectures that can learn low level features and higher level representations in an…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
A meaningful, uncollapsed result with the perceptron loss is only guaranteed if the model cannot produce a flat energy surface.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
Hence, at a high level, discriminative probabilistic models can be seen as a special case of EBMs in which:
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
This loss function, though popular for regression and neural network training, cannot be used to train most architectures, because it pushes down on the energy of the desired answer without pulling up on any other energy.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
Intuitively, the per-sample loss functional should be designed in such a way that it assigns a low loss to well-behaved energy functions…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Our model may give low energy to wrong answers in these regions, but since the inference algorithm cannot find them, they will never appear…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The min-sum procedure only applies to factor graphs that are bipartite trees with no loops.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The partition function over configurations of a loopy graphical model cannot be computed exactly, but loopy belief propagation or other variational methods may approximate it.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss for structured models requires summing over all label combinations, which number 2^l for a sequence of l binary labels.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The square-square loss is only suitable for energy functions bounded below by zero, because it pins the correct answer's energy at zero.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
When the best value of the latent variable for a given X and Y is ambiguous, one may consider combining the contributions of the various…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
In the usual scenario, the energy is minimized with respect to the variables to be predicted Y , given the observed variables X.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
In the tutorial's first experiment, the architecture's constraints automatically raise the energy of undesired answers while lowering that of desired answers.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Naturally, the ultimate purpose of learning is to produce a model that will give good answers for new input samples that are not seen…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Choosing the combinations of architecture and loss functions that can learn effectively and efficiently is critical to the energy-based…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
It is an interesting advantage of energy-based learning that only the incorrect answers whose energies are pulled up actually matter.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
As long as d ≥ m, the loss function shown in Figure 16(b) satisfies condition 3.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Each factor can be seen as representing soft constraints between the values of its input variables.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
More importantly, the very reason for internally normalizing HMMs and Bayesian nets is somewhat contradictory with the idea of training…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Fortunately, since those answers are not found by the inference algorithm, we do not need to worry about their energies.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The energy loss only works with architectures where pushing down the correct answer's energy automatically raises the energies of other answers.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
This section discusses the conditions that a loss function must satisfy so that its minimization will result in a model that produces the…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The generalized margin loss functions discussed satisfy condition 3.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
The distinction arises during training: we are given the correct value of Y for a number of training samples, but we are never given the…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Energy-based loss functions that pull up the most offending incorrect answer pull up only a single energy per learning iteration.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
There seems to be a bias-variance dilemma similar to the one that influences the generalization performance.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
In the NLL loss, the contrastive term pulls up the energies of all candidate answers.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the two-class problem, E_C and E_I correspond directly to the energies of the two classes.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The negative log-likelihood loss can only be used if the exponential of the negative energy is integrable over Y.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The preceding discussion of EBM inference and learning did not require manipulating normalized probability distributions.
Factual · 2006 |
AI capabilities |
| Not researched |
Citation |
2006 |
Huang and LeCun (2006) and Collobert et al. (2006) argued that convex loss functions do not guarantee good performance and that non-convex losses may be easier to optimize in practice.
Citation · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
GTNs are designed for situations where the sequential structure is so complicated that the corresponding dynamical factor graph cannot be…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
They are merely designed for training models to answer a different kind of question than probabilistic models.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Energy-Based Models (EBMs) capture dependencies between variables by associating a scalar energy to each configuration of the variables.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Running MCMC for just a few steps limits computational expense.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The hinge-type loss's second term prevents energy collapse by pushing up points whose energy threatens to go below that of the desired answer.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For the negative log-likelihood loss, the gradient with respect to E_C is positive and with respect to E_I is negative for any energy values.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The probability of training hitting a collapsed solution with the perceptron loss is quite small.
Factual · 2006 |
AI capabilities |
| Resolved true |
Factual |
2006 |
Inference algorithms in energy-based learning often give only approximate answers or do not guarantee finding the global energy minimum.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Since the RBF centers U1 and U2 are fixed and distinct, there is a strictly positive lower bound on d1 + d2 for all G_W.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For the squared-error regression energy, the minimum energy is always zero.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
With this simple architecture, the NLL or perceptron loss would give exactly the same result as the energy loss because their contrastive term is constant.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Since the contrastive sample is always near the desired answer, one can hope that the desired answer will become a local minimum of the…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Local minima are rarely a problem in practice for gradient-based learning because of the high dimensionality of the space.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
One interesting suggestion is to pull up on answers that are always near the correct answer, so as to make the correct answer a local…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Knowing these values would make the recognition task much easier.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
it is important to keep in mind that much of the discussion applies equally well to more complex architectures, as we will see later.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Stochastic gradient methods can be applied to any loss function, including nonconvex ones.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
The only challenge is to compute the gradient of the loss with respect to the neural net outputs by backpropagating gradients through the…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
In many practical situations computing the partition function is intractable or impossible, such as when Y has high cardinality or the integral has no analytical solution.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
Hinge, log, LVQ2, MCE, square-square and square-exponential losses can all be described as margin losses.
Factual · 2006 |
Science and research methods |
| Not researched |
Factual |
2006 |
With the energy loss, nothing prevents G1 and G2 from ignoring their inputs and producing identical outputs, causing a collapsed flat energy surface.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The set of collapsed solutions is a small piece of the parameter space.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
On average, each answer will be pulled up by the appropriate amount according to the partition function.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
the energy-based approach is bound to be more efficient.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
A distinction should be made between the energy function, which is minimized by the inference process, and the loss functional (introduced…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Second, computing the contrastive term in the negative log-likelihood loss function (or its gradient with respect to W ) may be very…
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
A carefully tuned stochastic gradient descent is often considerably faster in practice than sophisticated second-order methods because it exploits redundancy between samples.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
The simplest method is stochastic gradient descent.
Not checkable · 2006 |
|
| Not researched |
Citation |
2006 |
Hadsell et al. (2006) used Siamese architectures with the square-square loss for unsupervised learning of manifolds.
Citation · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
The loss function for the highly non-linear GTN-type parameterization is not convex in W.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
With some architectures, the energy loss can lead to a collapsed solution in which the energy is constant and equal to zero.
Factual · 2006 |
Science and research methods |
| Not checkable |
Not checkable |
2006 |
Contrastive divergence learning can be seen as an approximation of NLL learning with two shortcuts.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
It would seem natural to train GTNs with one of the generalized margin losses.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
the only consistent way involves turning the collection of energies for all possible outputs into a normalized probability distribution.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
Discriminative training often uses iterative gradient-based methods to optimize the loss.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
For many interesting models, computing the integral over Y in the NLL loss is intractable.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Hence probabilistic modeling comes with a high price, and should be avoided when the application does not require it.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Latent variables are very useful in situations where a hidden characteristic of the process being modeled can be inferred from…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
However there are numerous applications where it is convenient to use energy functions that depend on a set of hidden variables Z whose…
Not checkable · 2006 |
|
| Unresolvable |
Factual |
2006 |
The LVQ2 loss is a continuous approximation of the number of classification errors.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
In the third and fourth experiments, using loss functions with explicit contrastive terms, the machine performed as expected and did not collapse.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Unlike generalized margin losses, the LVQ2 loss is non-convex in the correct and most offending incorrect energies.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
As of 2006, GTNs had never been trained with a generalized margin loss, to the authors' knowledge.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Probabilistic learning is a special case of energy-based learning in which the loss function is the negative log-likelihood, also called the maximum mutual information criterion.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
Locating the best-scoring location in the image consists in minimizing the energy with respect to Y and Z.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
For the squared-error regression energy, the Y minimizing E equals G_W(X).
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
Excellent results were obtained for isolated word recognition with the LVQ2 loss despite its zero margin.
Factual · 2006 |
AI capabilities |
| Not researched |
Factual |
2006 |
As of 2006, no theory exists for energy-based learning with approximate inference.
Factual · 2006 |
AI capabilities |
| Not checkable |
Not checkable |
2006 |
First, the normalization requirement limits the choice of energy functions we can use.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Unfortunately, not all combinations of architectures and loss functions are allowed.
Not checkable · 2006 |
|
| Not researched |
Factual |
2006 |
The NLL loss is invariant to global shifts of energy values and depends only on energy differences among the Ys for a given X.
Factual · 2006 |
AI capabilities |