| Not checkable |
Not checkable |
Feb 2023 |
I think they lack *security mindset*.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
Nobody knows how to create any good AIs at all.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
The politicians do not understand it.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
A whole lot of this problem is that even if you do build an AI that's limited in some way, somebody else steals it, copies it, runs it…
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
A superintelligence is like that with respect to everything, with respect to all of humanity. It is relatively efficient to humanity.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
Current heads of major labs seem to me to be openly contemptuous of these issues.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
The tools are too blunt.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
I basically don't see on-model hopeful outcomes at this point.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
If you're the kind of person who can think of that from scratch, maybe take your hand at alignment.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
So maybe the people finding weird stuff to do with AIs, maybe those people will come up with some truth about these systems that makes them…
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
Mostly what you've got to do is invent some entirely new AI paradigm that isn't the giant inscrutable matrices of floating point numbers on…
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
If we got unlimited free retries without destroying the world, it'd play out the same way that ChatGPT played out.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
if you donate that to MIRI, then you at least have the assurance that we will not randomly spray money on looking like we're doing stuff
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
It's good for applications where it works 80% of the time, but not where it needs to work 99.999% of the time.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
And the AI there goes over a critical threshold, which most obviously could be like, "can write the next AI".
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There’s no plan.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level, opening up new…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
But they’ll die with more dignity than this.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There are no pivotal weak acts.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It probably relates to ‘security mindset‘, and a mental motion where you refuse to play out scripts
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
That’s not what surviving worlds look like.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
is an option that first gets evaluated for real at the point where an AGI fully expects it can defeat its creators.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You don’t get 1000 failed tries at burning all GPUs – because people will notice, even leaving out the consequences of capabilities success…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
If cognitive machinery doesn’t generalize far out of the distribution where you did tons of training, it can’t solve problems on the order…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We can’t just “decide not to build AGI” because GPUs are everywhere, and knowledge of algorithms is constantly being improved and published
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There is no pivotal act this weak
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Many alignment problems of superintelligence will not naturally appear at pre-dangerous, passively-safe levels of capability.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Earthlings have no word and indeed no standard native concept that means ‘actually useful cognitive power’.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A large amount of failure to panic sufficiently, seems to me to stem from a lack of appreciation for the incredible potential lethality of…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s not just non-hand-codable, it is unteachable on-the-first-try because the thing you are trying to teach is too weird and complicated.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Once you start to suspect you’re in that situation, you have to do the Bayesian thing and update now to the view you will predictably…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You could pump a billion dollars into it and it would produce mostly noise to drown out what little progress was being made elsewhere.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s a genuinely difficult problem, to solve something like that on your first try.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
What’s lethal is that we do not have the Textbook From The Future telling us all the simple solutions that actually in real life just work…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
In this non-surviving world, there are no candidate plans that do not immediately fall to Eliezer instantly pointing at the giant visible…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
That requires, not the ability to read this document and nod along with it, but the ability to spontaneously write it from scratch without…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We need to get alignment right on the ‘first critical try’ at operating at a ‘dangerous’ level of intelligence, where unaligned operation…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The worlds of humanity that survive have plans.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
None of this is about anything being impossible in principle.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Running AGIs doing something pivotal are not passively safe, they’re the equivalent of nuclear cores that require actively maintained…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There’s no reliable Cartesian-sensory ground truth (reliable loss-function-calculator) about whether an output is ‘aligned’
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We can’t just build a very weak system, which is less dangerous because it is so weak, and declare victory
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It is a written plan.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Anybody telling you I’m asking for stricter ‘alignment’ than this has failed at reading comprehension.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
people who might not be able to do equally great work away from tight feedback loops
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
you can’t build a system that only has the capability to drive red cars and not blue cars, because all red-car-driving algorithms…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you’re playing them against each…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Knowing that a medium-strength system of inscrutable matrices is planning to kill us, does not thereby let us build a high-strength system…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Almost all of them are there to tackle problems on which they can appear to succeed and publish a paper claiming success
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Words are not an AGI-complete data representation in its native style.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A powerful AI searches parts of the option space we don’t, and we can’t foresee all its options.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A strategically aware intelligence can choose its visible outputs to have the consequence of deceiving you
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
probably don’t have the mysterious gears simply because they’re rare.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The AI does not think like you do, the AI doesn’t have thoughts built up from the same concepts you use, it is utterly alien on a…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
If we had unlimited retries – if every time an AGI destroyed all the galaxies we got to go back in time four years and try again – we would…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
if you’re fighting it in an incredibly complicated domain you understand poorly, like human minds, you should expect to be defeated by…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I concede that real high-powered talents, especially if they’re still in their 20s, genuinely interested, and have done their reading, are…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The metaphor I usually use is that if a textbook from one hundred years in the future fell into our hands, containing all of the simple…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
a “multipolar” system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This is less of a viable survival plan for your planet if the first major failure of the bright-eyed youngsters kills literally everyone…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m assuming you are already familiar with some basics, and already know what ‘orthogonality’ and ‘instrumental convergence’ are and why…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You can’t train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Half the people going into string theory shifted into AI alignment instead and made real progress there.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Key people are taking internal and real responsibility for finding flaws in their own plans, instead of considering it their job to propose…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m not particularly hopeful of this turning out to be true in real life
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This situation you see when you look around you is not what a surviving world looks like.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m not particularly happy with this list; the alternative was publishing nothing, and publishing this seems marginally more dignified.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Practically all of the difficulty is in getting to “less than certainty of killing literally everyone”.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
These problems are in fact out of reach
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s guaranteed that some of my analysis is mistaken, though not necessarily in a hopeful direction.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The central result: Capabilities generalize further than alignment once capabilities start to generalize far.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The example I usually give is “burn all GPUs”.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The second thing looks unworkable (less so than CEV, but still lethally unworkable) because corrigibility runs actively counter to…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I figured this stuff out using the null string as input
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Reading this document cannot make somebody a core alignment researcher.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I have a hard time myself feeling hopeful about getting real alignment work out of somebody who previously sat around waiting for somebody…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This is sufficient on its own, even ignoring many other items on this list, to trash entire categories of naive alignment proposals which…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
So most organizations don’t have plans, because I haven’t taken the time to personally yell at them.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Human beings cannot inspect an AGI’s output to determine whether the consequences will be good.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This field is not making real progress and does not have a recognition function to distinguish real progress if it took place.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Corrigibility is anti-natural to consequentialist reasoning
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This is a very lethal problem, it has to be solved one way or another, it has to be solved at a minimum strength and difficulty level…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
When you explicitly optimize against a detector of unaligned thoughts, you’re partially optimizing for more aligned thoughts, and partially…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
But I’d have more hope – not significant hope, but more hope – in separating the concerns of (a) credibly promising to pay big money…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You don’t get any cynical old veterans, in this case, because everybody on Earth is dead.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
when it comes to a choice of utility function, there are unbounded degrees of freedom and multiple reflectively coherent fixpoints.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
So far as I’m concerned, if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It does not appear to me that the field of ‘AI safety’ is currently being remotely productive on tackling its enormous lethal problems.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Nobody knows what the hell GPT-3 is thinking
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Anticipating all of the really dangerous ones, and then successfully materializing them, in the correct form for early solutions to…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Human beings can figure out pretty difficult things over time, when they get lots of tries; when a failed guess kills literally everyone…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
If all you need is an object that doesn’t do dangerous things, you could try a sponge; a sponge is very passively safe.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s not like people will yell at themselves about prospective alignment difficulties, they don’t have an internal voice of caution.
Not checkable · 10 Jun 2022 |
|