| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
To prevent the energy collapse, i.e., flat energy landscape, we need to constrain our energy surface so that data points have low energies…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
A joint embedding predictive architecture (JEPA), presented in Fig. 9, is an EBM that combines
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Energy-based models (EBMs) capture dependencies between variables x and y by associating a scalar energy F (x, y) to each configuration of…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
This pitfall is due to the folded structure of the data, which, however, rarely
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
it can collapse because it can learn the identity function and have zero energy for all the inputs.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Such multi-level predictions may be achieved with a hierarchical joint embedding predictive architecture (H-JEPA).
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The tricky part is that the information capacity of the latent variable must be minimized.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Moreover, JEPAs naturally handle
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The main disadvantage of maximum likelihood as a contrastive EBM is that it wants to make the difference between the energy on the data…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
To avoid this drawback, we need to design an architecture that does not require transferring outputs between its components.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
While it may be surprising that we give up on a probabilistic setting, note that making decisions can be viewed as choosing the option with…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In particular, energies are uncalibrated (i.e., measured in arbitrary units), so combining two separately trained EBMs is not…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Finally, the maximum likelihood can be interpreted as a special case of a contrastive method.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
It means that the minimization of LNLL consists in minimizing the energy of training samples while pulling up the energy of Monte…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Moreover, this representation should be task-agnostic.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
However, in decision-making tasks such as driving a car, it is merely necessary that the system chooses the correct answer.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Ideally, the latent variable represents independent explanatory factors of prediction variation.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Therefore, the JEPA can be trained with a loss function that apart from the
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Moreover, the modern ML systems are very specialized (trained for one task) and brittle (make stupid mistakes).
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The key difference is that the world model predicting the future here is learned.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
While the modern large language models sometimes provide an illusion of reasoning, they have no understanding of reality.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
RL works great with games because they can be played over and over in parallel.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Another approach is to add a regularization term that minimizes a certain measure
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Eventually, we hope to achieve a fully autonomous AI, which can perform well on generic tasks by transferring its knowledge and…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The problem of such
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The reason for the limited success of masked AEs in continuous domains like images or videos
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
regularized methods seem more promising for training EBMs of the future.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
All in all, every model that can be multimodal, i.e., have multiple predictions for a single input, is susceptible to collapse.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Negative loglikelihood is then a contrastive method!
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Perhaps surprisingly, this representation encompasses many constrained fitting
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
This limitation renders the Hopfield networks less
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In particular, to prevent the informational energy collapse, we need to make sure that sx and s y carry
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Discrete and continuous Gibbs distribution (or softmax, should be called softargmax):
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Therefore, latent variables enable multiple predictions instead of just one.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
So, we can recast the regularized AE as jointly
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
A promising approach is to combine masked AEs with transformers for images [57].
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
An important thing to note is that the energy function discussed here is used only for inference, not learning.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Thus, they give us a way of handling real-world uncertainty.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Finally, we need to minimize or limit the information content of the latent variable to prevent the model from, e.g., relying solely on…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
However, for other architectures, such training may lead to the collapse of the energy function, i.e., given an x the energy landscape…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Therefore, the path to fully autonomous human-like intelligence has three main challenges:
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Low energies should be assigned to compatible pairs of x and y (e.g., slow driving when snow is present) and high energies to incompatible…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Here we summarized the main ideas of LeCun from Ref. [6] that address those limitations.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In reality, humans imagine most of the outcomes instead of trying them all.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The proposed solution to tackle multimodality is latent variable models with joint embeddings, and we discuss them in the next section.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The main principle of training an EBM is to sculpt the energy landscape such that data points that are probable or compatible have low…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The global task of this architecture is to take action that minimizes the cost or, even better, minimizes the expected cost in the future.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Such a hierarchical structure allows for multi-level representations of the world and, more
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The possible approach is designing reasoning and planning as energy minimization.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Finally, latent variables are useful in so-called structured prediction problems.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
The changes of the critic module model the external influence that is provided by culture, norms, and other people.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In other words, this term decorrelates the representation components, encouraging the learning of different input data aspects.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Therefore, there is no need to use a probabilistic framework, as you just need scores for each situation (word/move, etc.), which in…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
If we train our model only to minimize the distance between the outputs of encoders, then the two encoders are likely to ignore inputs…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Note that the marginal distribution is equivalent to Eq. (A1) in which the energy function has merely been redefined
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In such high-dimensional and multi-modal real-world setups, we may need to give up the idea of representing probability distributions over…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Maximum likelihood needs to be then regularized, e.g., with Bayesian prior or by limiting values of model parameters.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
Such an energy-guided model only needs to assign the lowest energy to the correct answer and larger energies to the incorrect ones.
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
In particular, precise predictions of the state of the world, based on the limited percepts of the environment, are possible
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Yann LeCun | Not checkable |
Jun 2023 |
EBMs, which do not have an immediate probabilistic interpretation, give much more flexibility in the choice of the training setup and, in…
Yann LeCun · Not checkable · Jun 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
31 May 2023 |
When a press release about an existential risk for humanity comes from some organization you haven't previously heard of, step 1 is to go…
Rodney Brooks · Not checkable · 31 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
28 May 2023 |
Moral: unreliable tools in ordinary hands are dangerous.
Rodney Brooks · Not checkable · 28 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
28 May 2023 |
I am just a citizen user of the service.
Rodney Brooks · Not checkable · 28 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
28 May 2023 |
Rating: success.
Rodney Brooks · Not checkable · 28 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
And we don't know if the psyche of the recently ebullient Chinese consumer will be impacted.
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
We don't know if travel will be banned or restricted, although the Macau casino stocks are trading like it's gonna happen
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
I would hate to advise you to sell and then buy back later, though, because we don't know if you'll be able to get back in before the…
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
if you think our leaders are serious about making a deal, then it might be worth trying to sidestep the coming decline
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
They hurt you then. They aren't done hurting you now.
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
24 May 2023 |
But, unless you trade full-time, it's very hard to get out and get back in early enough for it to make a difference, which means most of us…
Jim Cramer · Not checkable · 24 May 2023 |
|
| Not checkable |
Balaji Srinivasan | Not checkable |
24 May 2023 |
calls these tech giants a “systemic risk” to the crypto industry
Balaji Srinivasan · Not checkable · 24 May 2023 |
|
| Not checkable |
Marc Andreessen | Not checkable |
19 May 2023 |
We could never have built @a16z without @jeff_jordan -- @bhorowitz and I will always be grateful beyond words to our partner and friend.
Marc Andreessen · Not checkable · 19 May 2023 |
|
| Not checkable |
Jeff Bezos | Not checkable |
19 May 2023 |
Together, we’ll be solving the boil-off problem and making LOX-LH2 a storable propellant combination, pushing forward the state of the art…
Jeff Bezos · Not checkable · 19 May 2023 |
|
| Not checkable |
Marc Andreessen | Not checkable |
8 May 2023 |
🔥 https://t.co/QP348vYv6d
Marc Andreessen · Not checkable · 8 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
6 May 2023 |
It irks me when "AI" is used with an article, definite or indefinite
Rodney Brooks · Not checkable · 6 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
1 May 2023 |
Chatbots have plenty of gall when it comes to facts.
Rodney Brooks · Not checkable · 1 May 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
30 Apr 2023 |
The opposite of all these are true for robocars.
Rodney Brooks · Not checkable · 30 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
30 Apr 2023 |
Innovation at a distance!!!
Rodney Brooks · Not checkable · 30 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
30 Apr 2023 |
A personned vehicle stopped on Gough St is not 100% equivalent.
Rodney Brooks · Not checkable · 30 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
For that, we need to work with greater speed, stronger collaboration and execution, and to simplify the way we make decisions to focus on…
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
we will be working hard to provide clarity for everyone as rapidly as possible.
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
we have a real opportunity to deliver AI research and products that dramatically improve the lives of billions of people, transform…
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
I’m thrilled to be on this journey with you and look forward to seeing everyone soon.
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
Now, we live in a time in which AI research and technology is advancing exponentially.
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
Building ever more capable and general AI, safely and responsibly, demands that we solve some of the hardest scientific and engineering…
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Demis Hassabis | Not checkable |
20 Apr 2023 |
I’m incredibly excited to be leading this unit and working with all of you to build it.
Demis Hassabis · Not checkable · 20 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
18 Apr 2023 |
@gdb is obfuscating what ChatGPT to the @TED audience attributing intent and reasoning which is just not there.
Rodney Brooks · Not checkable · 18 Apr 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
17 Apr 2023 |
"But still those losses didn't kill the bull," Cramer said.
Jim Cramer · Not checkable · 17 Apr 2023 |
|
| Not checkable |
Jim Cramer | Not checkable |
17 Apr 2023 |
But, Cramer says, "the bull's been crushing the matador."
Jim Cramer · Not checkable · 17 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
14 Apr 2023 |
A rather recent AI hype cycle is pretty much agreed to have been that, rather than a transformation of society.
Rodney Brooks · Not checkable · 14 Apr 2023 |
|
| Not checkable |
Marc Andreessen | Not checkable |
14 Apr 2023 |
@chaos_gpt Is there anything I can do to help you?
Marc Andreessen · Not checkable · 14 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
13 Apr 2023 |
As companies hand tweak LLM based systems to cover published failures those who want to believe in a new super being get more and more…
Rodney Brooks · Not checkable · 13 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
13 Apr 2023 |
I use LLMs daily, but that are not atall what many people want them to be, nor what they assume will come just around the corner.
Rodney Brooks · Not checkable · 13 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
10 Apr 2023 |
There is no miracle.
Rodney Brooks · Not checkable · 10 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
10 Apr 2023 |
A new, rather tricky, tool, yes, but we're still going to have to get through life on our own.
Rodney Brooks · Not checkable · 10 Apr 2023 |
|
| Not checkable |
Rodney Brooks | Not checkable |
10 Apr 2023 |
This is a really good explanation of and reminder about ChatGPT.
Rodney Brooks · Not checkable · 10 Apr 2023 |
|
| Not checkable |
Eliezer Yudkowsky | Not checkable |
7 Apr 2023 |
To threaten these things, but end up not having to do them, is not very morally distinct—I would say—from doing them.
Eliezer Yudkowsky · Not checkable · 7 Apr 2023 |
|
| Not checkable |
Eliezer Yudkowsky | Not checkable |
7 Apr 2023 |
If there was a plan for Earth to survive, if only we passed a six-month moratorium, I would back that plan.
Eliezer Yudkowsky · Not checkable · 7 Apr 2023 |
|
| Not checkable |
Eliezer Yudkowsky | Not checkable |
7 Apr 2023 |
We are not prepared. We are not on course to be prepared in any reasonable time window. There is no plan.
Eliezer Yudkowsky · Not checkable · 7 Apr 2023 |
|
| Not checkable |
Eliezer Yudkowsky | Not checkable |
7 Apr 2023 |
This 6-month moratorium would be better than no moratorium.
Eliezer Yudkowsky · Not checkable · 7 Apr 2023 |
|