| Resolved true |
Factual |
Jun 2023 |
LeCun's proposed AI architecture consists of multiple interconnected modules.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, latent variables enable multiple predictions instead of just one.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
So, we can recast the regularized AE as jointly
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
A promising approach is to combine masked AEs with transformers for images [57].
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
An important thing to note is that the energy function discussed here is used only for inference, not learning.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Thus, they give us a way of handling real-world uncertainty.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Finally, we need to minimize or limit the information content of the latent variable to prevent the model from, e.g., relying solely on…
Not checkable · Jun 2023 |
|
| Unresolvable |
Prediction |
Jun 2023 |
Contrastive methods are unlikely to lead to future autonomous intelligence because of their unfavorable scaling with data dimension.
Prediction · Jun 2023 |
AI timelines and forecasting |
| Mostly true |
Factual |
Jun 2023 |
With probabilistic models, training is confined to loss functions derived from the negative log-likelihood, such as cross entropy.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
However, for other architectures, such training may lead to the collapse of the energy function, i.e., given an x the energy landscape…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, the path to fully autonomous human-like intelligence has three main challenges:
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Low energies should be assigned to compatible pairs of x and y (e.g., slow driving when snow is present) and high energies to incompatible…
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
For energy-based models with a continuous y domain, the energy function is smooth and differentiable, allowing gradient-based inference.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
Here we summarized the main ideas of LeCun from Ref. [6] that address those limitations.
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
The lecture notes summarize the main ideas of the autonomous intelligence architecture proposed by Yann LeCun.
Factual · Jun 2023 |
Their own work or organisation |
| Not checkable |
Not checkable |
Jun 2023 |
In reality, humans imagine most of the outcomes instead of trying them all.
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
SSL works very well for text, but for images, models trained to make a single prediction predict the average of all possible predictions.
Factual · Jun 2023 |
AI capabilities |
| Not researched |
Citation |
Jun 2023 |
The method for maximizing information content of learned representations is called variance-invariance-covariance regularization (VICReg), per reference [58].
Citation · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
The proposed solution to tackle multimodality is latent variable models with joint embeddings, and we discuss them in the next section.
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
Computing a partition function is very challenging, so probabilistic models in continuous high-dimensional domains are intractable.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
The main principle of training an EBM is to sculpt the energy landscape such that data points that are probable or compatible have low…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The global task of this architecture is to take action that minimizes the cost or, even better, minimizes the expected cost in the future.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Such a hierarchical structure allows for multi-level representations of the world and, more
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
Some regularized autoencoders learn interpretable features that mirror existing hand-crafted data compression approaches.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
The possible approach is designing reasoning and planning as energy minimization.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Finally, latent variables are useful in so-called structured prediction problems.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The changes of the critic module model the external influence that is provided by culture, norms, and other people.
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
A major technique for detecting blacklisted content is nearest-neighbor search against that content.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
In other words, this term decorrelates the representation components, encouraging the learning of different input data aspects.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, there is no need to use a probabilistic framework, as you just need scores for each situation (word/move, etc.), which in…
Not checkable · Jun 2023 |
|
| Mostly true |
Factual |
Jun 2023 |
All modules of the proposed architecture are fully differentiable.
Factual · Jun 2023 |
AI capabilities |
| Not checkable |
Not checkable |
Jun 2023 |
If we train our model only to minimize the distance between the outputs of encoders, then the two encoders are likely to ignore inputs…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Note that the marginal distribution is equivalent to Eq. (A1) in which the energy function has merely been redefined
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In such high-dimensional and multi-modal real-world setups, we may need to give up the idea of representing probability distributions over…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Maximum likelihood needs to be then regularized, e.g., with Bayesian prior or by limiting values of model parameters.
Not checkable · Jun 2023 |
|
| Not researched |
Factual |
Jun 2023 |
Current self-driving models need thousands of footage to learn, while humans can learn to drive in about twenty hours.
Factual · Jun 2023 |
AI capabilities |
| Unresolvable |
Prediction |
Jun 2023 |
Probabilistic models may also fail on the path to autonomous intelligence.
Prediction · Jun 2023 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
Jun 2023 |
Such an energy-guided model only needs to assign the lowest energy to the correct answer and larger energies to the incorrect ones.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In particular, precise predictions of the state of the world, based on the limited percepts of the environment, are possible
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
EBMs, which do not have an immediate probabilistic interpretation, give much more flexibility in the choice of the training setup and, in…
Not checkable · Jun 2023 |
|
| Unresolvable |
Prediction |
4 Mar 2021 |
Researchers may never develop techniques to represent suitable probability distributions over high-dimensional continuous spaces such as all possible video frames.
Prediction · 4 Mar 2021 |
AI capabilities |
| Unresolvable |
Factual |
4 Mar 2021 |
The Siamese network / joint embedding idea goes back to papers from Geoff Hinton's lab and Yann LeCun's group in the early 1990s.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
We believe that self-supervised learning (SSL) is one of the most promising ways to build such background knowledge and approximate a form…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Self-supervised learning enables AI systems to learn from orders of magnitude more data, which is important to recognize and understand…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Without a specific way to do so, the two networks could happily ignore their inputs and always produce identical output embeddings.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Long-term progress will be cumulative.
Not checkable · 4 Mar 2021 |
|
| Resolved true |
Factual |
4 Mar 2021 |
Language models pretrained with self-supervised learning achieve considerably higher performance than models trained solely in a supervised manner.
Factual · 4 Mar 2021 |
AI capabilities |
| Mostly true |
Factual |
4 Mar 2021 |
Facebook has deployed XLM-R, a model based on its RoBERTa architecture, to improve hate speech classifiers in multiple languages across Facebook and Instagram.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
We cannot list all possible video frames and associate a score to each of them, because there is an infinite number of them.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Unsupervised learning is an ill-defined and misleading term that suggests that the learning uses no supervision at all.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Practically speaking, it’s impossible to label everything in the world.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Non-contrastive methods applied to joint embedding architectures is possibly the hottest topic in SSL for vision at the moment.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The short answer is that humans rely on their previously acquired background knowledge of how the world works.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Unfortunately, there’s a limit to how far the field of AI can go with supervised learning alone.
Not checkable · 4 Mar 2021 |
|
| Unresolvable |
Prediction |
4 Mar 2021 |
Perhaps, in the long run, non-contrastive methods with latent-variable predictive models will prove a better alternative.
Prediction · 4 Mar 2021 |
AI capabilities |
| Mostly true |
Factual |
4 Mar 2021 |
LeCun said Facebook AI had created and open sourced SEER, a billion-parameter self-supervised computer vision model that he described as proven to work efficiently with complex, high-dimensional image data.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
In a way, common sense is the dark matter of artificial intelligence.
Not checkable · 4 Mar 2021 |
|
| Mostly true |
Factual |
4 Mar 2021 |
As of March 2021, self-supervised learning had not yet brought computer vision the same improvements it has brought to natural language processing.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
Latent-variable models can be trained with contrastive methods.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
No perfect recipe seems to exist to limit the capacity of latent variables.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Part one is simple, but part two is where the difficulty lies.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
In fact, self-supervised learning is not unsupervised, as it uses far more feedback signals than standard supervised and reinforcement…
Not checkable · 4 Mar 2021 |
|
| Mostly true |
Factual |
4 Mar 2021 |
The Barlow Twins method tries to minimize redundancy between the individual components of the embedding vectors.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
There is a way to think about SSL within the unified framework of an energy-based model (EBM).
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
A working hypothesis is that generalized knowledge about the world, or common sense, forms the bulk of biological intelligence in both…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
These results show that we can bring the self-supervised learning paradigm shift to computer vision.
Not checkable · 4 Mar 2021 |
|
| Resolved true |
Factual |
4 Mar 2021 |
After pretraining on a billion unlabeled Instagram images and fine-tuning on ImageNet, SEER outperformed state-of-the-art self-supervised systems, reaching 84.2 percent top-1 accuracy on ImageNet.
Factual · 4 Mar 2021 |
AI capabilities |
| Not checkable |
Not checkable |
4 Mar 2021 |
If the energy is low, x and y are deemed compatible; if it is high, they are deemed incompatible.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The main reason is that it is considerably more difficult to represent uncertainty in the prediction for images than it is for words.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
But contrastive methods have a major issue: They are very inefficient to train.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Supervised learning is a bottleneck for building more intelligent generalist models that can do multiple tasks and acquire new skills…
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
This progress demonstrates that self-supervised learning can excel at CV tasks in complex, real-world settings as well.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
The method used to train NLP systems by masking or substituting some input words belongs to the category of contrastive methods.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
A particular well-suited deep learning architecture to do so is the so-called Siamese networks or joint embedding architecture.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
4 Mar 2021 |
Self-supervision is one step on the path to human-level intelligence, but there are surely many steps that lie behind this one.
Not checkable · 4 Mar 2021 |
|
| Not checkable |
Not checkable |
2021 |
But curiosity just makes the process faster.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
Contrastive learning with positive and negative pairs does not work when the representation is high-dimensional, because there are too many ways for two things to be different.
Factual · 2021 |
AI capabilities |
| Not researched |
Factual |
2021 |
In MIT experiments where a baby ferret's optic nerve is connected to the auditory cortex, the auditory cortex becomes essentially a visual cortex.
Factual · 2021 |
Science and research methods |
| Not checkable |
Not checkable |
2021 |
You have to have somewhere representing a sort of infinite number of plausible continuations of multiple frames in a high dimensional…
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I think it's integrated.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
That's called a collapse.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
In the first months of life, babies learn how the world works mostly by observation because they can hardly act in the world.
Factual · 2021 |
Philosophy, ethics and society |
| Not researched |
Factual |
2021 |
Across AI sub-fields there has been a continuous evolution from less hand-crafting to more learning.
Factual · 2021 |
AI capabilities |
| Not checkable |
Not checkable |
2021 |
It is, but we're not there yet.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
So it's easier to learn when your objective is intrinsic.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
So that has to be hardwired.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
People have used data augmentation on MNIST digits since the 1990s.
Factual · 2021 |
AI capabilities |
| Not researched |
Factual |
2021 |
Geoffrey Hinton proposed, in the early 1990s with his student Sue Becker, the idea of maximizing mutual information between the outputs of two systems.
Factual · 2021 |
AI capabilities |
| Not checkable |
Not checkable |
2021 |
Supervised learning requires many samples for learning anything.
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
I don't know how to make logic-based reasoning
Not checkable · 2021 |
|
| Not checkable |
Not checkable |
2021 |
It doesn't tell us how to feed ourselves.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
After self-supervised representation learning, recognizing handwritten digits on top of it requires only a few labeled samples.
Factual · 2021 |
AI capabilities |
| Not checkable |
Not checkable |
2021 |
It's not platonic because eventually I want the stuff to get used, but it's okay if it takes five or 10 years for the community to realize…
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
The difference between human and chimp genomes involved in human intelligence is less than 1% of the genome.
Factual · 2021 |
Science and research methods |
| Not checkable |
Not checkable |
2021 |
I have kind of an orthodox folk speculative hypothesis about consciousness.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
The system under discussion worked, but they ended up not using it.
Factual · 2021 |
Technology |
| Not researched |
Factual |
2021 |
LeCun used to call the two-identical-copies network architecture a Siamese network.
Factual · 2021 |
AI capabilities |
| Not checkable |
Not checkable |
2021 |
I mean, it's a superficial understanding, obviously.
Not checkable · 2021 |
|
| Not researched |
Factual |
2021 |
With a differentiable world model such as a neural net, one can do gradient-based model predictive control by backpropagating through it.
Factual · 2021 |
AI capabilities |
| Unresolvable |
Prediction |
2021 |
Training systems on video prediction with self-supervised techniques will provide a path towards some level of physical common sense in machines.
Prediction · 2021 |
AI capabilities |