| Not checkable |
Not checkable |
28 Feb 2018 |
Okay, what changes your mind short of the world actually ending?
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
This is not here to fit into some kind of grand moral lesson that I have for you about how civilization ought to work.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Yeah, that’s not going to save you from something that’s significantly smarter than you are.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
then the feeling of something being far away because you don’t know how to do it yet is just not very informative.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the key to that is our ability to learn things that natural selection did not preprogram us with; so learning is the key to generality.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But the more complicated the system is and the less you understand the system, the more something smarter than you may have what is simply…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
That’s actually a surprisingly good plot
Not checkable · 28 Feb 2018 |
|
| Own records only |
Factual |
28 Feb 2018 |
MIRI has some communication with DeepMind's upper people, though the bandwidth is not optimal.
Factual · 28 Feb 2018 |
Their own work or organisation |
| Not checkable |
Not checkable |
28 Feb 2018 |
One is: “by definition”? We don’t care what’s true by definition here.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It doesn’t scale.
Not checkable · 28 Feb 2018 |
|
| Not researched |
Factual |
28 Feb 2018 |
Economists already know about market failures and have that concept.
Factual · 28 Feb 2018 |
Economics and markets |
| Unresolvable |
Prediction |
28 Feb 2018 |
An AI built narrowly enough to be understood will not be able to defend against arbitrary unrestrained superintelligences.
Prediction · 28 Feb 2018 |
AI safety and alignment |
| Not checkable |
Not checkable |
28 Feb 2018 |
So the orthogonality thesis is that an intelligence that pursues paperclips for their own sake, because that’s what its utility function…
Not checkable · 28 Feb 2018 |
|
| Not researched |
Factual |
28 Feb 2018 |
MIRI is the original AI alignment organization and today works primarily on the technical parts of the alignment problem.
Factual · 28 Feb 2018 |
Their own work or organisation |
| Not checkable |
Not checkable |
28 Feb 2018 |
That’s one of the fundamental foundational ways that systems can get stuck.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think that the main thing you should be trying to do with the first artificial general intelligence ever built is a very narrow…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
If there was a way to make an enormous profit by knowing this, the profit would probably already be taken.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think, by the way, that that’s a pretty terrible term.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
if alignment is technically difficult, if it is easy to screw up, if it requires a bunch of additional effort
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
a paperclip maximizer is built so as to respond to “is” questions about paperclips, not about what is right and what is good
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I don’t technically agree with it either. What I would say is that the facts are not motivating.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
for AlphaGo to play better Go than the best human Go players, it must be the case that the best human Go players cannot predict exactly…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
The thing the chimp is better at doesn’t play a big role in our global economy.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But humanly speaking—not so much in terms of ideal rationality, but in terms of what people can actually do and the options they actually…
Not checkable · 28 Feb 2018 |
|
| Not researched |
Factual |
28 Feb 2018 |
Eliezer Yudkowsky wrote two to three years of daily blog posts on human rationality, which were collected into the book Rationality: From AI to Zombies.
Factual · 28 Feb 2018 |
Their own work or organisation |
| Not checkable |
Not checkable |
28 Feb 2018 |
my claim is not that there is a grand narrative that makes it emotionally consonant that paperclip maximizers are a thing. I’m claiming…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
But it was clear that I could self-teach, so that’s what I did.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It can’t be settled by definitions; it can’t be settled by how you define “intelligence.”
Not checkable · 28 Feb 2018 |
|
| Unresolvable |
Prediction |
28 Feb 2018 |
Yudkowsky thinks AGI will be built on purpose about 10 years before it would otherwise happen by accident.
Prediction · 28 Feb 2018 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s possible that if they were all really rational and had common knowledge of rationality, they would be able to solve the coordination…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I’m not concerned that alignment is impossible. I’m concerned that it’s difficult.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Nobody on the sane alignment-is-necessary side of this argument is postulating that the CPUs are disobeying the laws of physics to…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It seems to me to be only slightly more reasonable than the old theory that if you got dirty shirts and straw, they would spontaneously…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would call that “AI alignment,” following Stuart Russell.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
by “achieve goals” we mean something like “squeezing the measure of possible futures higher in your preference ordering.”
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I do think that a lot of people listening may want us to back up and talk about the question of whether you can have something that feels…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
There’s no continuity between what you have to do to fend off little pieces of code trying to break into your computer, and what you have…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s in this sense that I mean that there’s no fire alarm for artificial general intelligence.
Not checkable · 28 Feb 2018 |
|
| Mixed |
Factual |
28 Feb 2018 |
AlphaZero surpassed human Go knowledge in less than a day starting from scratch, without human games, theories or Go-specific code beyond what chess used.
Factual · 28 Feb 2018 |
AI capabilities |
| Not checkable |
Not checkable |
28 Feb 2018 |
both the narrow and the ambitious alignment have in common that you’re trying to have the AI do that thing rather than making a lot of…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think that there are multiple narratives about AI, and I think that the technical truth is something that doesn’t fit into any of the…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I have much clearer ideas about how to go around tackling the technical problem than tackling the social problem.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I think, for example, that if you have humans and you make the human smarter, this is not orthogonal to the humans’ values.
Not checkable · 28 Feb 2018 |
|
| Unresolvable |
Prediction |
28 Feb 2018 |
The first artificial general intelligence will be built in a research lab specifically dedicated to building it, not arise spontaneously from an unrelated system.
Prediction · 28 Feb 2018 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
28 Feb 2018 |
You could build a mind that would not deceive you; you could build a mind that maximizes the flourishing of a happy intergalactic…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
I would say that the real function of the fire alarm is the social function of telling you that everyone else knows there’s a fire and you…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Even if all you are trying to do is end up with two identical strawberries on a plate without destroying the universe, I think that’s…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
Even the most talented Go player, if you plopped them down in front of a Go board and gave them only a day, would play garbage.
Not checkable · 28 Feb 2018 |
|
| Unresolvable |
Prediction |
28 Feb 2018 |
Yudkowsky said he thinks the artificial general intelligence timeline is likely to run out before humanity gets much better at solving inadequacy and systemic problems in general.
Prediction · 28 Feb 2018 |
AI timelines and forecasting |
| Not researched |
Factual |
28 Feb 2018 |
Eliezer Yudkowsky conducted the experiment that became known as the AI-box experiment, to demonstrate human limits in dealing with smarter minds.
Factual · 28 Feb 2018 |
AI safety and alignment |
| Not checkable |
Not checkable |
28 Feb 2018 |
And that’s the thing that I think is true as a matter of computer science. Not as a matter of fitting with a particular narrative; that’s…
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
It’s not really a very hopeful book in that regard.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
There’s no point in even talking about the arms race if the arms race is between a set of unfriendly AIs with no friendly AI in the mix.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
And this follows from the rapid capability gain thesis and at least the current landscape for how these things are developing.
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
the problem isn’t that paperclip factory AIs spontaneously wake up.
Not checkable · 28 Feb 2018 |
|
| Not researched |
Factual |
28 Feb 2018 |
Yudkowsky said that AlphaZero is not reflecting on itself, is not learning things, does not have a general model of the world, and is not operating in new contexts or making new contexts for itself to be in.
Factual · 28 Feb 2018 |
AI capabilities |
| Not checkable |
Not checkable |
28 Feb 2018 |
They’re just wrong as to where the problem lies.
Not checkable · 28 Feb 2018 |
|
| Unresolvable |
Prediction |
28 Feb 2018 |
Better anti-virus software will not give automatic defense against superintelligence.
Prediction · 28 Feb 2018 |
AI safety and alignment |
| Not checkable |
Not checkable |
28 Feb 2018 |
I think they’re wrong
Not checkable · 28 Feb 2018 |
|
| Not checkable |
Not checkable |
28 Feb 2018 |
it seems to me that it is perfectly coherent to imagine an AI that is better at everything (or almost everything) than we are
Not checkable · 28 Feb 2018 |
|
| Not researched |
Factual |
28 Feb 2018 |
Two years after the Wright Flyer, people could still be found saying heavier-than-air flight is impossible.
Factual · 28 Feb 2018 |
Other global risks |
| Research failed |
Factual |
28 Feb 2018 |
The experiment in which people stay in a smoke-filling room when others look calm is pretty well replicated, with many variations finding basically the same result, though he said he did not want to put absolute faith in it because of the replication crisis.
Factual · 28 Feb 2018 |
Science and research methods |
| Mostly true |
Factual |
30 Jan 2013 |
Nick Bostrom does meta-strategy work as well as object-level work on anthropic probabilities and the parliamentary model for normative uncertainty.
Factual · 30 Jan 2013 |
Field building and careers |
| Mostly true |
Factual |
30 Jan 2013 |
Nick Bostrom hosts researchers such as Anders Sandberg, who write papers about uploading timelines relevant to policy decisions.
Factual · 30 Jan 2013 |
Field building and careers |
| Not checkable |
Not checkable |
30 Jan 2013 |
Are you sure the same way of thinking wouldn’t prevent you from working on any important problem?
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
They didn’t work like that.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
If you’re still like, “No, no, we must think more meta” after a year, then I don’t believe you’re the sort of person who will ever act.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
When people constantly say “maybe we should do some other thing,” I would say, “Come to an interim decision, start acting on the interim…
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
If he then came up with some particular intervention which he thought was higher priority, I’d ask about the specific case.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
Work on whatever your current priority is, after an hour of meta reasoning but not a year of meta reasoning.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
Real computers don’t work very much like the formalism, but Turing’s work was useful anyway.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
Nick Bostrom isn’t making this mistake, either.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
No, this issue of trying to have an AI write a similar AI seems like a fairly fundamental one, and the Löb Problem blocks it.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
But if you’re the person who always tries to go meta and only thinks meta because there might be some better thing, you’re not ever going…
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
I’d start by having them read a description of a particular technical problem we’re working on, for example the “Löb Problem.”
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
The fact that we can’t figure out how to do these things — even given infinite computing power — is alarming.
Not checkable · 30 Jan 2013 |
|
| Resolved true |
Factual |
30 Jan 2013 |
Paul Christiano is working on actual Friendly AI problems while also looking for other promising interventions.
Factual · 30 Jan 2013 |
Field building and careers |
| Not checkable |
Not checkable |
30 Jan 2013 |
Are you sure you wouldn’t be going back in time and telling Alan Turing to not invent Turing machines because who knows whether computers…
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
30 Jan 2013 |
I don’t have much objection to that.
Not checkable · 30 Jan 2013 |
|
| Not checkable |
Not checkable |
2008 |
I would be pleasantly surprised to be proven wrong about this.
Not checkable · 2008 |
|
| Unresolvable |
Factual |
2008 |
The Dartmouth Proposal on AI makes no mention of building nice, good or benevolent AI.
Factual · 2008 |
AI policy and governance |
| Not checkable |
Not checkable |
2008 |
No system is secure that searches for ways to defeat its own security.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Wason's 2-4-6 task is a "cold" form of confirmation bias; people seek confirming but not falsifying evidence.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
it would be nice if we knew how to build a nice AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This statement now seems to me like complete gibberish.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This does not imply the AI has the motive to change its own motives.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I have given you another opportunity to fail each time you face the challenge of changing your mind.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Such competence is not historically typical of human institutions, no matter how hard they try.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
However, ranking (8) as more probable than (6) violates the conjunction rule of probability theory which states that p(A & B) ≤ p(A).
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Subjects in Wason's 2-4-6 task try to confirm their hypotheses rather than falsify them.
Factual · 2008 |
Science and research methods |
| Not researched |
Factual |
2008 |
Flaws have occasionally been discovered in previously accepted informal human mathematical proofs.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Thus, all else being equal, I would greatly prefer that Friendly AI precede nanotechnology in the ordering of technological developments.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Information known to be irrelevant still anchors people's judgments and contaminates their guesses.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
Moore’s Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world drops by one point.
Not checkable · 2008 |
|
| Mostly true |
Factual |
2008 |
Overconfidence is no longer noteworthy in the modern field but continues to show up in nearly any experiment allowing extreme probabilities.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
I should not have had to stumble across that reference by accident.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The first AI must be Friendly.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If an AI will hurt you given magic, the Friendliness architecture is wrong.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Lacking the space for exposition, I can only plead with the reader to consult the primary literature.
Not checkable · 2008 |
|