| Not checkable |
Not checkable |
28 Feb 2018 |
These two questions together form a paperclip maximizer. You don’t need anything else.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the idea that human minds are secure is loony.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
That’s what the AI-box experiment illustrates. You don’t know what went on in there, and that’s exactly the position you’d be in with…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
You only get oughts from other oughts. That’s the Hume version of the thesis.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would describe myself as a decision theorist.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’m worried that it’s not happening fast enough
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Parenthetically, I do want to be slightly careful about using the word “rational” to describe the behavior of people stuck in the system…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Collectively. Like, if people who slow down and do things right finish their work two years after the universe has been destroyed, that’s…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the way I would phrase it is that it’s not that the paperclip maximizer has a different set of oughts, but that we can see it as running…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
all of the ones who are not utter idiots are past the point where they would deliberately enact Hollywood movie plots.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
We know humans are conscious, so the computations that you run to build very accurate predictive models of humans are among the parts that…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And I would say this is a computer science issue.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
So, one way to look at the book is that it’s about how you can get crazy, stupid, evil large systems without any of the people inside them…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
paperclips are not one of those things. No matter how long you stare at a paperclip, it still seems pretty pointless from our perspective.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The original “orthogonal” means “at right angles.”
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And I find that convincing, but other people may want additional details.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’d say that Neil de Grasse Tyson is failing to respect the AI’s intelligence to the point of asking what he would do if he were inside a…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’d say this is a thesis of capability gain.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I don’t think we’re looking at an economic problem.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
A coordination problem is where there’s a better way to do it, but you have to change more than one thing at a time.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
They’re not imagining enough changing simultaneously.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I don’t think that’s true, but I’m worried about the sentiment behind it.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It could be 50 years away. It could be two years away.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The people at Google are not dumb enough to build an AI and tell it to make paperclips.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
that pursues arbitrary tractable goals. Including, for example, making paperclips.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
you can have arbitrarily powerful intelligence, with no defects of that intelligence
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I wouldn’t say disparagement. These are just initial reactions. These are people we haven’t been talking to yet.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The main way in which I would be worried about conscious systems emerging within the system without that happening on purpose would be if…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But the three laws, I mean, they’re deontology. Decision theory requires quantitative weights on your goals.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would mostly say intelligence.org.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Instrumental convergence! Which means that a lot of times, across a very broad range of final goals, there are similar strategies (we…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
We aren’t smart enough to treat the brain as something that from our perspective is a mechanical system and just navigate it to where you…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s more about how to predict which parts of society will perform poorly to the point where you as an individual can manage to do better…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And if you don’t have a business plan for this sort of thing, then you’re obviously just using it as an excuse.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
AlphaZero seems to me like a genuine case in point.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
We’re still on the same map. I agree with most of it.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It is certainly possible to have agents such that as they get smarter, what they would report as their utility functions will change. A…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I didn’t have any super-secret special trick that makes it all make sense in retrospect. I just did it the hard way.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I mean, sort of crowdfunding, I guess, although the hammer currently isn’t very large.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But this is a fairy tale. This is not a prediction.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
there seems to have been a very sharp jump in generality between chimpanzees and humans.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The thing I’m worried about is that the way that natural selection does it—it’s not stable.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The sentiments behind this are something that I have a great deal of sympathy for.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But broadly speaking, it’s an AI where you can say what it’s trying to do.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Those are plot devices. If they worked, Asimov would have had no stories.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And that in turn is an example that refutes another thesis about how artificial general intelligence develops slowly and gradually
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
they couldn’t all jump to the new system because they couldn’t jump one at a time.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The thing that makes something smarter than you dangerous is you cannot foresee everything it might try.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’m not concerned that you’re going to teach the AI to deceive humans. I’m concerned that someone somewhere is going to get to the point of…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the real story wasn’t in how AlphaGo beat human Go players. It’s in how AlphaZero beat human Go system programmers and human chess system…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The system didn’t fit me that well, and I’m good at self-teaching.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
This is the special case of the system that is smart in the way that you are smart and that mice are not smart.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The problem I’m worried about is that it’s technically difficult to get the AI to have a particular goal set and keep that goal set and…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The way I would rephrase the fact/values thing is: We all know about David Hume and Hume’s Razor
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Okay, what changes your mind short of the world actually ending?
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
This is not here to fit into some kind of grand moral lesson that I have for you about how civilization ought to work.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Yeah, that’s not going to save you from something that’s significantly smarter than you are.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
then the feeling of something being far away because you don’t know how to do it yet is just not very informative.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the key to that is our ability to learn things that natural selection did not preprogram us with; so learning is the key to generality.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But the more complicated the system is and the less you understand the system, the more something smarter than you may have what is simply…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
That’s actually a surprisingly good plot
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
One is: “by definition”? We don’t care what’s true by definition here.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It doesn’t scale.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
So the orthogonality thesis is that an intelligence that pursues paperclips for their own sake, because that’s what its utility function…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
That’s one of the fundamental foundational ways that systems can get stuck.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think that the main thing you should be trying to do with the first artificial general intelligence ever built is a very narrow…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
If there was a way to make an enormous profit by knowing this, the profit would probably already be taken.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think, by the way, that that’s a pretty terrible term.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
if alignment is technically difficult, if it is easy to screw up, if it requires a bunch of additional effort
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
a paperclip maximizer is built so as to respond to “is” questions about paperclips, not about what is right and what is good
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I don’t technically agree with it either. What I would say is that the facts are not motivating.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
for AlphaGo to play better Go than the best human Go players, it must be the case that the best human Go players cannot predict exactly…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The thing the chimp is better at doesn’t play a big role in our global economy.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But humanly speaking—not so much in terms of ideal rationality, but in terms of what people can actually do and the options they actually…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
my claim is not that there is a grand narrative that makes it emotionally consonant that paperclip maximizers are a thing. I’m claiming…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But it was clear that I could self-teach, so that’s what I did.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It can’t be settled by definitions; it can’t be settled by how you define “intelligence.”
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s possible that if they were all really rational and had common knowledge of rationality, they would be able to solve the coordination…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’m not concerned that alignment is impossible. I’m concerned that it’s difficult.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Nobody on the sane alignment-is-necessary side of this argument is postulating that the CPUs are disobeying the laws of physics to…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It seems to me to be only slightly more reasonable than the old theory that if you got dirty shirts and straw, they would spontaneously…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would call that “AI alignment,” following Stuart Russell.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
by “achieve goals” we mean something like “squeezing the measure of possible futures higher in your preference ordering.”
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I do think that a lot of people listening may want us to back up and talk about the question of whether you can have something that feels…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
There’s no continuity between what you have to do to fend off little pieces of code trying to break into your computer, and what you have…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s in this sense that I mean that there’s no fire alarm for artificial general intelligence.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
both the narrow and the ambitious alignment have in common that you’re trying to have the AI do that thing rather than making a lot of…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think that there are multiple narratives about AI, and I think that the technical truth is something that doesn’t fit into any of the…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I have much clearer ideas about how to go around tackling the technical problem than tackling the social problem.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think, for example, that if you have humans and you make the human smarter, this is not orthogonal to the humans’ values.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
You could build a mind that would not deceive you; you could build a mind that maximizes the flourishing of a happy intergalactic…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would say that the real function of the fire alarm is the social function of telling you that everyone else knows there’s a fire and you…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Even if all you are trying to do is end up with two identical strawberries on a plate without destroying the universe, I think that’s…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Even the most talented Go player, if you plopped them down in front of a Go board and gave them only a day, would play garbage.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And that’s the thing that I think is true as a matter of computer science. Not as a matter of fitting with a particular narrative; that’s…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s not really a very hopeful book in that regard.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
There’s no point in even talking about the arms race if the arms race is between a set of unfriendly AIs with no friendly AI in the mix.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And this follows from the rapid capability gain thesis and at least the current landscape for how these things are developing.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the problem isn’t that paperclip factory AIs spontaneously wake up.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
They’re just wrong as to where the problem lies.
Not checkable · 28 Feb 2018 |
|