| Not checkable |
Not checkable |
8 Mar 2024 |
And some of that work has been incredibly successful.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
It doesn’t think about its answer really. It retrieves it because it’s accumulated a lot of knowledge.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So the only way you’re going to have an AI industry, the only way you’re going to have AI systems that are not uniquely biased is if you…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I think it should not be abandoned completely, but I think it’s use should be minimized because it’s incredibly inefficient in terms of…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I would argue most of our knowledge is derived from that interaction with the physical world.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So the short answer is no
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
There is an abstract representation of imagining the reaction of a reader to that text.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
They’re not going to be able to plan for situations where that they never encountered before.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So making people smarter, we just can only be better.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
But there are limitations to those contrastive methods.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Alan Turing would decide that a Turing test is a really bad test, okay?
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
The design is all about safety, because a better turbojet is also a safer turbojet, so a more reliable one.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
That’s just not acceptable, it’s a danger to democracy regardless of how well-intentioned those companies are, and it’s also a danger to…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Well, that’s exactly why we need open source platforms.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I’m clearly in the camp that yes, intelligence cannot appear without some grounding in some reality.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So that, I think, is an essential component of an intelligent system.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Don’t think it’s just Moravec’s paradox.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So now you can do what LMS cannot do, which is planning what you’re going to do.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Well, there’s a lot of situations that might be difficult to, for a purely language based system to know.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
No, it’s basically a struggle against the curse of dimensionality.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
A lot of it is similar, and LLMs don’t have that.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
There is another set of methods which are non-contrastive, and I prefer those
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So certainly LLMs are going to be able to do this, but then how you link this from the low level of actions, that needs to be done with…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So having machines that are smarter than us, that assist us in all of our tasks, our daily lives, whether it’s professional or personal, I…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I don’t think anyone today would say that the invention of the printing press had a overall negative effect despite the fact that it…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Big companies are really careful about not producing things of this type because they don’t want to hurt anyone, first of all, and then…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And we’re talking about inference, we’re not talking about training, the system has been trained already.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
No, I don’t think the issue has to do with the political leaning of the people designing those systems.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So we can’t just only model everything at the lowest level.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
We specify what the proper levels of abstraction, the representation at each level of abstraction have to be.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
But it’s not a question of being impressed or not, it’s the question of knowing what the limit of those systems can do.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
science makes progress when people disagree and they come up with an answer and consensus forms
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
That’s one of the reasons you need experience of the physical world, which is much higher bandwidth than what you can express in words, in…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Even if you have a team of 50 engineers that are really experienced building it, you’re still going to have to blow up a dozen of them…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
There’s a few breakthroughs that we have to basically go through before we can get there, but you’ll be able to monitor our progress…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
if you expect the system to become intelligent just without having the possibility of doing those things, you’re making a mistake.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I don’t hate reinforcement learning
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
It doesn’t need to be physical reality. It could be simulated, but the environment is just much richer than what you can express in…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And the answer is the same answer that we found in liberal democracy about the press, the press needs to be free and diverse.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
I think people are fundamentally good and in fact, a lot of doomers are doomers because they don’t think that people are fundamentally good.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
but there is a lot of things they cannot do, and we have to realize what they cannot do and then figure out how we get there.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And if some of those top systems are open source, anybody can use them, anybody can fine tune them.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And so you cannot have an unbiased system, that’s just an impossibility.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So what you want is your encoder to basically eliminate all those details.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
The kind of thinking you’re doing has absolutely nothing to do with language really.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Hans Marvek would say that Marvek Paradox still applies.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Well, there’s all kinds of incentive to make AI systems submissive to humans.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
it is very important for people in India.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Mark is right about a number of things that you list that indeed scare large companies.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
intelligence is not a linear thing that you can measure with a scale or with a single number.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
we don’t want all of our information to come from a unique source because that’s opposite to the whole idea of democracy and progressive…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
in the end, the system ends up being a giant lookup table essentially, which is not really what you want, you want systems that can reason…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
But the point is, if you have a big enough potential customer base and you need to build that system anyway for them, it doesn’t hurt you…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And in fact, a lot of doomers are doomers because they don’t think that people are fundamentally good, and they either don’t trust people…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
It’s going to be you push it in one way, one set of people are going to see it as biased, and then you push it the other way and another…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
There has to be some limit to the kind of stuff that those systems might be authorized to produce, some guardrails.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And I call this a JEPA, so that means joint embedding predictive architecture
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So I’m super excited because I see a path towards potentially human-level intelligence with systems that can understand the world…
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
It has to do with the acceptability or political leanings of their customer base or audience.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
those are four essential characteristics of intelligent systems or entities, humans, animals.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So intelligence is a collection of skills and an ability to acquire new skills efficiently.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
No, of course everybody would be impressed.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
And they do this optimization in a horribly inefficient way, which is generate a lot of hypothesis and then select the best ones.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
Well, eventually, yes. But I think if we do this too early, we run the risk of being tempted to cheat.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
The idea somehow that we can’t get it slightly wrong because if we get it slightly wrong, we’ll die is ridiculous.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
8 Mar 2024 |
So there’s going to be various versions of LLaMA that are improvements of previous LLaMAs, bigger, better, multimodal, things like that.
Not checkable · 8 Mar 2024 |
|
| Not checkable |
Not checkable |
28 Jun 2023 |
Also, my ears are not pointy, and my archer skills are feeble.
Not checkable · 28 Jun 2023 |
|
| Not checkable |
Not checkable |
27 Jun 2023 |
Nice piece. But it should have talked about @NablaTech one of the first players in that space (which the NYT wrote about a while back).
Not checkable · 27 Jun 2023 |
|
| Not checkable |
Not checkable |
26 Jun 2023 |
We are missing a number of basic concepts to get human-level AI.
Not checkable · 26 Jun 2023 |
|
| Not checkable |
Not checkable |
26 Jun 2023 |
I also agree that many of the AI safety policies that have been floating around are naïve.
Not checkable · 26 Jun 2023 |
|
| Not checkable |
Not checkable |
26 Jun 2023 |
Good thread on AI policy from @jackclarkSF
Not checkable · 26 Jun 2023 |
|
| Not checkable |
Not checkable |
26 Jun 2023 |
I agree that limiting compute or requiring a license above a certain threshold of compute power would be bad policy.
Not checkable · 26 Jun 2023 |
|
| Not checkable |
Not checkable |
25 Jun 2023 |
Thanks for clarifying.
Not checkable · 25 Jun 2023 |
|
| Not checkable |
Not checkable |
25 Jun 2023 |
On le construira pour être bénéfique.
Not checkable · 25 Jun 2023 |
|
| Not checkable |
Not checkable |
23 Jun 2023 |
He doesn't seem worried about the probability that he will produce intelligent evil psychopaths.
Not checkable · 23 Jun 2023 |
|
| Not checkable |
Not checkable |
23 Jun 2023 |
We can do the same with AI: we can thoroughly test their benevolence before we give them any knowledge and agency.
Not checkable · 23 Jun 2023 |
|
| Not checkable |
Not checkable |
22 Jun 2023 |
Automated manufacturing makes handcrafted goods more valuable.
Not checkable · 22 Jun 2023 |
|
| Not checkable |
Not checkable |
21 Jun 2023 |
Fun CIFAR workshop, as usual.
Not checkable · 21 Jun 2023 |
|
| Not checkable |
Not checkable |
19 Jun 2023 |
I can't disagree!
Not checkable · 19 Jun 2023 |
|
| Not checkable |
Not checkable |
18 Jun 2023 |
How to shoot yourself in the foot with a bazooka in the form of insane immigration laws.
Not checkable · 18 Jun 2023 |
|
| Not checkable |
Not checkable |
15 Jun 2023 |
A good article by the BBC about my views on the future of AI.
Not checkable · 15 Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Human learning, in contrast, is often highly efficient.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
However, which information is important may be scale-dependent.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The proposed solution is, therefore, to replace probabilistic models with energy-based models (EBMs) and combine them with latent variables…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
In the case of the latent variable EBMs (Fig. 4(b)), such
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Contrastive methods aim to push down the energy of training samples and to pull up the energy of suitably-generated contrastive samples…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Current automated systems have crucial limitations that need to be addressed before artificial intelligence can reach human-like levels and…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
An important feature of latent variable models is that while they are composed of deterministic functions, they are non-deterministic as…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Therefore, the energy function is not the objective function to minimize within the learning!
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
This proposal is important to the whole ML community
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
SSL is the cake (provides millions of information bits per sample)
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Firstly, it is challenging to understand uncertainty in an energy-based setting.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
To prevent the energy collapse, i.e., flat energy landscape, we need to constrain our energy surface so that data points have low energies…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
A joint embedding predictive architecture (JEPA), presented in Fig. 9, is an EBM that combines
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Energy-based models (EBMs) capture dependencies between variables x and y by associating a scalar energy F (x, y) to each configuration of…
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
This pitfall is due to the folded structure of the data, which, however, rarely
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
it can collapse because it can learn the identity function and have zero energy for all the inputs.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Such multi-level predictions may be achieved with a hierarchical joint embedding predictive architecture (H-JEPA).
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
The tricky part is that the information capacity of the latent variable must be minimized.
Not checkable · Jun 2023 |
|
| Not checkable |
Not checkable |
Jun 2023 |
Moreover, JEPAs naturally handle
Not checkable · Jun 2023 |
|