| Not checkable |
Not checkable |
Jun 2023 |
The main disadvantage of maximum likelihood as a contrastive EBM is that it wants to make the difference between the energy on the data…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
To avoid this drawback, we need to design an architecture that does not require transferring outputs between its components.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
While it may be surprising that we give up on a probabilistic setting, note that making decisions can be viewed as choosing the option with…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In particular, energies are uncalibrated (i.e., measured in arbitrary units), so combining two separately trained EBMs is not…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Finally, the maximum likelihood can be interpreted as a special case of a contrastive method.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
It means that the minimization of LNLL consists in minimizing the energy of training samples while pulling up the energy of Monte…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Moreover, this representation should be task-agnostic.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
However, in decision-making tasks such as driving a car, it is merely necessary that the system chooses the correct answer.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Ideally, the latent variable represents independent explanatory factors of prediction variation.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, the JEPA can be trained with a loss function that apart from the
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Moreover, the modern ML systems are very specialized (trained for one task) and brittle (make stupid mistakes).
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The key difference is that the world model predicting the future here is learned.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
While the modern large language models sometimes provide an illusion of reasoning, they have no understanding of reality.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
RL works great with games because they can be played over and over in parallel.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Another approach is to add a regularization term that minimizes a certain measure
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Eventually, we hope to achieve a fully autonomous AI, which can perform well on generic tasks by transferring its knowledge and…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The problem of such
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The reason for the limited success of masked AEs in continuous domains like images or videos
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
regularized methods seem more promising for training EBMs of the future.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
All in all, every model that can be multimodal, i.e., have multiple predictions for a single input, is susceptible to collapse.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Negative loglikelihood is then a contrastive method!
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Perhaps surprisingly, this representation encompasses many constrained fitting
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
This limitation renders the Hopfield networks less
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In particular, to prevent the informational energy collapse, we need to make sure that sx and s y carry
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Discrete and continuous Gibbs distribution (or softmax, should be called softargmax):
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, latent variables enable multiple predictions instead of just one.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
So, we can recast the regularized AE as jointly
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
A promising approach is to combine masked AEs with transformers for images [57].
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
An important thing to note is that the energy function discussed here is used only for inference, not learning.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Thus, they give us a way of handling real-world uncertainty.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Finally, we need to minimize or limit the information content of the latent variable to prevent the model from, e.g., relying solely on…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
However, for other architectures, such training may lead to the collapse of the energy function, i.e., given an x the energy landscape…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, the path to fully autonomous human-like intelligence has three main challenges:
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Low energies should be assigned to compatible pairs of x and y (e.g., slow driving when snow is present) and high energies to incompatible…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Here we summarized the main ideas of LeCun from Ref. [6] that address those limitations.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In reality, humans imagine most of the outcomes instead of trying them all.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The proposed solution to tackle multimodality is latent variable models with joint embeddings, and we discuss them in the next section.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The main principle of training an EBM is to sculpt the energy landscape such that data points that are probable or compatible have low…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The global task of this architecture is to take action that minimizes the cost or, even better, minimizes the expected cost in the future.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Such a hierarchical structure allows for multi-level representations of the world and, more
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The possible approach is designing reasoning and planning as energy minimization.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Finally, latent variables are useful in so-called structured prediction problems.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The changes of the critic module model the external influence that is provided by culture, norms, and other people.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In other words, this term decorrelates the representation components, encouraging the learning of different input data aspects.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, there is no need to use a probabilistic framework, as you just need scores for each situation (word/move, etc.), which in…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
If we train our model only to minimize the distance between the outputs of encoders, then the two encoders are likely to ignore inputs…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Note that the marginal distribution is equivalent to Eq. (A1) in which the energy function has merely been redefined
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In such high-dimensional and multi-modal real-world setups, we may need to give up the idea of representing probability distributions over…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Maximum likelihood needs to be then regularized, e.g., with Bayesian prior or by limiting values of model parameters.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Such an energy-guided model only needs to assign the lowest energy to the correct answer and larger energies to the incorrect ones.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In particular, precise predictions of the state of the world, based on the limited percepts of the environment, are possible
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
EBMs, which do not have an immediate probabilistic interpretation, give much more flexibility in the choice of the training setup and, in…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
We believe that self-supervised learning (SSL) is one of the most promising ways to build such background knowledge and approximate a form…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Self-supervised learning enables AI systems to learn from orders of magnitude more data, which is important to recognize and understand…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Without a specific way to do so, the two networks could happily ignore their inputs and always produce identical output embeddings.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Long-term progress will be cumulative.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
We cannot list all possible video frames and associate a score to each of them, because there is an infinite number of them.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Unsupervised learning is an ill-defined and misleading term that suggests that the learning uses no supervision at all.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Practically speaking, it’s impossible to label everything in the world.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Non-contrastive methods applied to joint embedding architectures is possibly the hottest topic in SSL for vision at the moment.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The short answer is that humans rely on their previously acquired background knowledge of how the world works.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Unfortunately, there’s a limit to how far the field of AI can go with supervised learning alone.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
In a way, common sense is the dark matter of artificial intelligence.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Latent-variable models can be trained with contrastive methods.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
No perfect recipe seems to exist to limit the capacity of latent variables.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Part one is simple, but part two is where the difficulty lies.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
In fact, self-supervised learning is not unsupervised, as it uses far more feedback signals than standard supervised and reinforcement…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
There is a way to think about SSL within the unified framework of an energy-based model (EBM).
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
A working hypothesis is that generalized knowledge about the world, or common sense, forms the bulk of biological intelligence in both…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
These results show that we can bring the self-supervised learning paradigm shift to computer vision.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
If the energy is low, x and y are deemed compatible; if it is high, they are deemed incompatible.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The main reason is that it is considerably more difficult to represent uncertainty in the prediction for images than it is for words.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
But contrastive methods have a major issue: They are very inefficient to train.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Supervised learning is a bottleneck for building more intelligent generalist models that can do multiple tasks and acquire new skills…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
This progress demonstrates that self-supervised learning can excel at CV tasks in complex, real-world settings as well.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The method used to train NLP systems by masking or substituting some input words belongs to the category of contrastive methods.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
A particular well-suited deep learning architecture to do so is the so-called Siamese networks or joint embedding architecture.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Self-supervision is one step on the path to human-level intelligence, but there are surely many steps that lie behind this one.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
2021 |
But curiosity just makes the process faster.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
You have to have somewhere representing a sort of infinite number of plausible continuations of multiple frames in a high dimensional…
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I think it's integrated.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
That's called a collapse.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
It is, but we're not there yet.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
So it's easier to learn when your objective is intrinsic.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
So that has to be hardwired.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
Supervised learning requires many samples for learning anything.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I don't know how to make logic-based reasoning
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
It doesn't tell us how to feed ourselves.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
It's not platonic because eventually I want the stuff to get used, but it's okay if it takes five or 10 years for the community to realize…
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I have kind of an orthodox folk speculative hypothesis about consciousness.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I mean, it's a superficial understanding, obviously.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I think the big challenge of AI for the next decade,
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
If we had as many world models as there are situations we encounter, then we could do all of them simultaneously.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
Your model of the world should have your actions as one of the inputs, and that will drive you to learn causal models of the world
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
And that's the answer to this probably is self supervised running.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
The essence of intelligence is the ability to predict
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
And they are extremely inefficient.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
There's nothing wrong with that.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
And the main issue with learning world models or learning predictive models is that the prediction is never a single thing because the…
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
That engine is configurable to the situation at hand.
Not checkable · 2021 |
|