| Not checkable |
Not checkable |
18 Feb 2015 |
Eventually, we should establish ethical guidelines as to how AI can and cannot be used.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
What I am most fond of are beautiful and simple theoretical ideas that can be translated into something that works.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
It’s the idea that every component, every stage in a learning machine can be trained.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
My least favorite description is, “It works just like the brain.”
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I’m a rationalist. I’m also an atheist and a humanist; I’m not afraid of saying that.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
My idea of morality is to maximize overall human happiness and minimize human suffering over the long term.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
These are personal opinions that do not engage my employer.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I don’t like people saying this because, while Deep Learning gets an inspiration from biology, it’s very, very far from what the brain…
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
It was a very innovative way of doing things; it was never practical, but it was cool.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
Not much has come out so far.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
It leads to hype; people claim things that are not true.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
But I’m very careful not to use words that could lead to hype.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
The equivalent in AI is to try to copy every detail that we know of about how neurons and synapses work, and then turn on a gigantic…
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I’m kind of puzzled by that phenomenon.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
No, you can’t say never; technology is advancing very quickly, at an accelerating pace.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I personally don’t think machines should be able to attack targets without a human making the decision.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I try not to pull rank.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
It sets expectations for funding agencies, the public, potential customers, start-ups and investors, such that they believe that we are on…
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
But he has not contributed anything to the science of AI, as far as I can tell.
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
18 Feb 2015 |
I think it would be “machines that learn to represent the world.”
Not checkable · 18 Feb 2015 |
|
| Not checkable |
Not checkable |
26 May 2014 |
All three groups are strong and complementary.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I use a lot of math, sometimes at the conceptual level more than at the “detailed proof” level.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
There is nothing magical about margin maximization.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Like Kolmogorov, I am fascinated by the concept of complexity, which is at the root of learning theory, compression, and thermodynamics.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I prefer “Data Science”, which is the automatic (or semi-automatic) extraction of knowledge from data.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I do not believe that classical learning theory with “IID samples, convex optimization, and supervised classification and regression” is…
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Kernel methods are great for many purposes, but they are merely glorified template matching.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Also, I totally hate sampling.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
The principals don’t have a particularly good track record of success.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Physics is about modeling actual systems and processes.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Larger networks tend to work better.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I learned a lot by reading things that are not apparently connected with AI or computer science
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I still think [Vicarious] is mostly hype.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Only the first is interesting to me (the other uses are interesting too, just not on my own radar screen).
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Proofs always come later.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
I like the idea of Boltzmann machines (it’s a beautifully simple concept) but it doesn’t scale well.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
The theory of deep learning is a wide open field.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Ask them what error rate they get on MNIST or ImageNet.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
You can construct pathological cases where the objective function is like a golf course: flat with a tiny hole somewhere.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
DeconvNets are the generative counterpart of feed-forward ConvNets.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Learning with temporal/sequential signals: language, video, speech.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Seriously, I don’t like the phrase “Big Data”.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
They have the advantage of scaling well.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Jeff Hawkins has the right intuition and the right philosophy.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
26 May 2014 |
Few of us believe that we have found a good solution to unsupervised learning.
Not checkable · 26 May 2014 |
|
| Not checkable |
Not checkable |
2010 |
Implementing ConvNets in software is best achieved using the modular, object-oriented approach suggested in [2].
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
It is clear that abs and normalization are cruciala for achieving good performance.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
We describe new unsupervised learning algorithms, and new non-linear stages that allow ConvNets to be trained with very few labeled samples.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
The Convolutional Network architecture is a remarkably versatile, yet conceptually simple paradigm that can be applied to a wide spectrum…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Current and future research will focus on performing unsupervised learning on multiple stages simultaneously, on the integration of…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
The unsupervised method we propose, to learn the filter coefficients in the filter bank layers, is called Predictive Sparse Decomposition…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Good internal representations are hierarchical.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
This is an extremely important fact for users of convolutional networks, which traditionally only use tanh().
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Convolutional Networks (ConvNets) are an answer to the first question.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
One of the key questions of Vision Science (natural and artificial) is how to produce good internal representations of the visual world.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Convolutional Networks [1], [2] are trainable multistage architectures composed of multiple stages.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
This suggests that recognition architectures for vision (and for other modalities such as audio and natural language) should have multiple…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
Intelligent tasks, such as visual perception, auditory perception, and language understanding require the construction of good internal…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
ConvNets are perfect candidates for hardware implementations, and embedded applications, as demonstrated by the increasing amount of work…
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
A major question for Machine Learning is how to learn such good features automatically.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2010 |
It seems that unsupervised pre-training (U , U +) is crucial when newly proposed non-linearities are not in place.
Not checkable · 2010 |
|
| Not checkable |
Not checkable |
2006 |
In the following, we use the convention that small energy values correspond to highly compatible configurations of the variables, while…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
For decision-making tasks, such as steering a robot, it is merely necessary that the system give the lowest energy to the correct answer.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Among the many loss functions proposed in the literature, some are good (with a non-zero margin), and some can be bad.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
However, the convexity of the loss function, while mathematically satisfying, does not seem to be a significant practical advantage.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
For example, there is no reason to believe that the model in Figure 7 is normalizable over Y .
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
A common approach is to view variational methods and energy-based loss functions as approximations to the probabilistic method.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
In a sense, variational methods could be interpreted in the context of EBM as a way to choose a subset of energies to pull up.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
One advantage of the energy-based approach is that it puts very little restrictions on the nature of E.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
To some observers, the recent interest in the linear structured model looks like somewhat of a throw-back to the past, and a regression on…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Energy minimization is a convenient way to represent the general process of reasoning and inference.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Any prior knowledge we may have about the task at hand is embedded into the architecture and into the loss function (particularly the…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
We should emphasize again that this is inconsequential if the model is to be used for prediction, classification, or decision-making.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
The best inference procedure to use often depends on the internal structure of the model.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
It is not obvious that the negative log-likelihood loss satisfies condition 3.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
However, there is no guarantee that all incorrect answers with low energy will be pulled up.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Though this is a trivial problem for a learning machine, it is useful for demonstrating the issues involved in the design of an energy…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
The energy loss is a bad loss function in general, but there are certain forms of energies for which it is a good loss function.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Energy-based learning provides a unified framework for many probabilistic and non-probabilistic approaches to learning, particularly for…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
We conclude that this is a good loss function.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
A GTN can be seen as a multilayer architecture in which the states are trellises, just as a neural net is a multilayer architecture in…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
While the zero margin may limit the robustness of the solution, the perceptron loss seems appropriate as a way to refine a system that was…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
This is best handled by hierarchical, multi-layer architectures that can learn low level features and higher level representations in an…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Hence, at a high level, discriminative probabilistic models can be seen as a special case of EBMs in which:
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
EBMs can be viewed as a form of non-probabilistic factor graphs, and they provide considerably more flexibility in the design of…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Intuitively, the per-sample loss functional should be designed in such a way that it assigns a low loss to well-behaved energy functions…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Our model may give low energy to wrong answers in these regions, but since the inference algorithm cannot find them, they will never appear…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
When the best value of the latent variable for a given X and Y is ambiguous, one may consider combining the contributions of the various…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
In the usual scenario, the energy is minimized with respect to the variables to be predicted Y , given the observed variables X.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Naturally, the ultimate purpose of learning is to produce a model that will give good answers for new input samples that are not seen…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Choosing the combinations of architecture and loss functions that can learn effectively and efficiently is critical to the energy-based…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
It is an interesting advantage of energy-based learning that only the incorrect answers whose energies are pulled up actually matter.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Each factor can be seen as representing soft constraints between the values of its input variables.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
More importantly, the very reason for internally normalizing HMMs and Bayesian nets is somewhat contradictory with the idea of training…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
Fortunately, since those answers are not found by the inference algorithm, we do not need to worry about their energies.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
This section discusses the conditions that a loss function must satisfy so that its minimization will result in a model that produces the…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
The distinction arises during training: we are given the correct value of Y for a number of training samples, but we are never given the…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
There seems to be a bias-variance dilemma similar to the one that influences the generalization performance.
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
GTNs are designed for situations where the sequential structure is so complicated that the corresponding dynamical factor graph cannot be…
Not checkable · 2006 |
|
| Not checkable |
Not checkable |
2006 |
They are merely designed for training models to answer a different kind of question than probabilistic models.
Not checkable · 2006 |
|