| Not checkable |
Not checkable |
Feb 2023 |
A whole lot of this problem is that even if you do build an AI that's limited in some way, somebody else steals it, copies it, runs it…
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
A superintelligence is like that with respect to everything, with respect to all of humanity. It is relatively efficient to humanity.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
Current heads of major labs seem to me to be openly contemptuous of these issues.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
The tools are too blunt.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
I basically don't see on-model hopeful outcomes at this point.
Not checkable · Feb 2023 |
|
| Not researched |
Factual |
Feb 2023 |
The Wright brothers said heavier-than-air flight was 50 years off shortly before they built the first Wright Flyer.
Factual · Feb 2023 |
Science and research methods |
| Not checkable |
Not checkable |
Feb 2023 |
If you're the kind of person who can think of that from scratch, maybe take your hand at alignment.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
So maybe the people finding weird stuff to do with AIs, maybe those people will come up with some truth about these systems that makes them…
Not checkable · Feb 2023 |
|
| Not researched |
Factual |
Feb 2023 |
Yudkowsky's view on AI extinction risk is not the consensus view in the wider scientific field.
Factual · Feb 2023 |
AI safety and alignment |
| Not checkable |
Not checkable |
Feb 2023 |
Mostly what you've got to do is invent some entirely new AI paradigm that isn't the giant inscrutable matrices of floating point numbers on…
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
If we got unlimited free retries without destroying the world, it'd play out the same way that ChatGPT played out.
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
if you donate that to MIRI, then you at least have the assurance that we will not randomly spray money on looking like we're doing stuff
Not checkable · Feb 2023 |
|
| Not checkable |
Not checkable |
Feb 2023 |
It's good for applications where it works 80% of the time, but not where it needs to work 99.999% of the time.
Not checkable · Feb 2023 |
|
| Unresolvable |
Prediction |
Feb 2023 |
If the AI winter the subject said he had hoped for happened, the timeline he was discussing could be about 16 years.
Prediction · Feb 2023 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
Feb 2023 |
And the AI there goes over a critical threshold, which most obviously could be like, "can write the next AI".
Not checkable · Feb 2023 |
|
| Not researched |
Factual |
Feb 2023 |
MIRI has pursued many lines of research that have not panned out.
Factual · Feb 2023 |
Their own work or organisation |
| Not checkable |
Not checkable |
10 Jun 2022 |
There’s no plan.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level, opening up new…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
But they’ll die with more dignity than this.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There are no pivotal weak acts.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It probably relates to ‘security mindset‘, and a mental motion where you refuse to play out scripts
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
That’s not what surviving worlds look like.
Not checkable · 10 Jun 2022 |
|
| Mostly true |
Factual |
10 Jun 2022 |
Humans do not explicitly pursue inclusive genetic fitness, although natural selection optimized on that simple loss function.
Factual · 10 Jun 2022 |
AI safety and alignment |
| Unresolvable |
Prediction |
10 Jun 2022 |
Losing a conflict with a high-powered cognitive system looks at least as deadly as everyone on Earth suddenly dying within the same second.
Prediction · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
is an option that first gets evaluated for real at the point where an AGI fully expects it can defeat its creators.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You don’t get 1000 failed tries at burning all GPUs – because people will notice, even leaving out the consequences of capabilities success…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
If cognitive machinery doesn’t generalize far out of the distribution where you did tons of training, it can’t solve problems on the order…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We can’t just “decide not to build AGI” because GPUs are everywhere, and knowledge of algorithms is constantly being improved and published
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There is no pivotal act this weak
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Many alignment problems of superintelligence will not naturally appear at pre-dangerous, passively-safe levels of capability.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Earthlings have no word and indeed no standard native concept that means ‘actually useful cognitive power’.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A large amount of failure to panic sufficiently, seems to me to stem from a lack of appreciation for the incredible potential lethality of…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s not just non-hand-codable, it is unteachable on-the-first-try because the thing you are trying to teach is too weird and complicated.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Once you start to suspect you’re in that situation, you have to do the Bayesian thing and update now to the view you will predictably…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You could pump a billion dollars into it and it would produce mostly noise to drown out what little progress was being made elsewhere.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s a genuinely difficult problem, to solve something like that on your first try.
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
Using current alignment techniques, humanity is not on course to get AGI that avoids killing everyone with probability roughly 1.
Prediction · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
What’s lethal is that we do not have the Textbook From The Future telling us all the simple solutions that actually in real life just work…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
In this non-surviving world, there are no candidate plans that do not immediately fall to Eliezer instantly pointing at the giant visible…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
That requires, not the ability to read this document and nod along with it, but the ability to spontaneously write it from scratch without…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We need to get alignment right on the ‘first critical try’ at operating at a ‘dangerous’ level of intelligence, where unaligned operation…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment.
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
Given insufficient foresight by the operators, Yudkowsky expects many of those alignment problems to appear at about the same time after a sharp AI capability gain.
Prediction · 10 Jun 2022 |
AI safety and alignment |
| Unresolvable |
Prediction |
10 Jun 2022 |
Future AGI will not be upper-bounded by human ability or human learning speed.
Prediction · 10 Jun 2022 |
AI capabilities |
| Not checkable |
Not checkable |
10 Jun 2022 |
The worlds of humanity that survive have plans.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
None of this is about anything being impossible in principle.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Running AGIs doing something pivotal are not passively safe, they’re the equivalent of nuclear cores that require actively maintained…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
There’s no reliable Cartesian-sensory ground truth (reliable loss-function-calculator) about whether an output is ‘aligned’
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
We can’t just build a very weak system, which is less dangerous because it is so weak, and declare victory
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It is a written plan.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Anybody telling you I’m asking for stricter ‘alignment’ than this has failed at reading comprehension.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
people who might not be able to do equally great work away from tight feedback loops
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I…
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
A sufficiently capable cognitive system with any medium-bandwidth channel of causal influence could easily bootstrap to overpowering capabilities independent of human infrastructure.
Prediction · 10 Jun 2022 |
AI capabilities |
| Not checkable |
Not checkable |
10 Jun 2022 |
you can’t build a system that only has the capability to drive red cars and not blue cars, because all red-car-driving algorithms…
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Factual |
10 Jun 2022 |
When natural selection produced general intelligence in humans, alignment with the outer optimization loop broke after the intelligence became general, relatively late in its accumulation of capability and knowledge.
Factual · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you’re playing them against each…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Knowing that a medium-strength system of inscrutable matrices is planning to kill us, does not thereby let us build a high-strength system…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Almost all of them are there to tackle problems on which they can appear to succeed and publish a paper claiming success
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Words are not an AGI-complete data representation in its native style.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A powerful AI searches parts of the option space we don’t, and we can’t foresee all its options.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
A strategically aware intelligence can choose its visible outputs to have the consequence of deceiving you
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
probably don’t have the mysterious gears simply because they’re rare.
Not checkable · 10 Jun 2022 |
|
| Resolved true |
Factual |
10 Jun 2022 |
MIRI tried and failed to find a coherent formula for an agent that would let itself be shut down without actively trying to get shut down.
Factual · 10 Jun 2022 |
Their own work or organisation |
| Not checkable |
Not checkable |
10 Jun 2022 |
The AI does not think like you do, the AI doesn’t have thoughts built up from the same concepts you use, it is utterly alien on a…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
If we had unlimited retries – if every time an AGI destroyed all the galaxies we got to go back in time four years and try again – we would…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
if you’re fighting it in an incredibly complicated domain you understand poorly, like human minds, you should expect to be defeated by…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I concede that real high-powered talents, especially if they’re still in their 20s, genuinely interested, and have done their reading, are…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The metaphor I usually use is that if a textbook from one hundred years in the future fell into our hands, containing all of the simple…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
a “multipolar” system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which…
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
Yudkowsky expects that training by gradient descent against that behavior in a toy domain would produce not-particularly-coherent local patches to thought processes, which would break with near-certainty inside a superintelligence generalizing far outside its training distribution.
Prediction · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
This is less of a viable survival plan for your planet if the first major failure of the bright-eyed youngsters kills literally everyone…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m assuming you are already familiar with some basics, and already know what ‘orthogonality’ and ‘instrumental convergence’ are and why…
Not checkable · 10 Jun 2022 |
|
| Resolved true |
Factual |
10 Jun 2022 |
There is no known way to use loss functions, sensory inputs or reward inputs to make a cognitive system's optimization point at particular latent things in the environment.
Factual · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
You can’t train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Half the people going into string theory shifted into AI alignment instead and made real progress there.
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Factual |
10 Jun 2022 |
AlphaZero surpassed all accumulated human knowledge about Go after about a day of self-play, without human playbooks or sample games.
Factual · 10 Jun 2022 |
AI capabilities |
| Not checkable |
Not checkable |
10 Jun 2022 |
Key people are taking internal and real responsibility for finding flaws in their own plans, instead of considering it their job to propose…
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
Programmers' and operators' methodologies may break when they move from not-fully-dangerous to dangerous AI domains, which are out-of-distribution for them.
Prediction · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m not particularly hopeful of this turning out to be true in real life
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This situation you see when you look around you is not what a surviving world looks like.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I’m not particularly happy with this list; the alternative was publishing nothing, and publishing this seems marginally more dignified.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Practically all of the difficulty is in getting to “less than certainty of killing literally everyone”.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
it is still Eliezer Yudkowsky writing up this list, says that humanity still has only one gamepiece that can do that.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
These problems are in fact out of reach
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
It’s guaranteed that some of my analysis is mistaken, though not necessarily in a hopeful direction.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The central result: Capabilities generalize further than alignment once capabilities start to generalize far.
Not checkable · 10 Jun 2022 |
|
| Mostly true |
Factual |
10 Jun 2022 |
Under the current optimization paradigm there is no general method known to get particular inner properties into an AI system or verify they are present.
Factual · 10 Jun 2022 |
AI safety and alignment |
| Not checkable |
Not checkable |
10 Jun 2022 |
The example I usually give is “burn all GPUs”.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The second thing looks unworkable (less so than CEV, but still lethally unworkable) because corrigibility runs actively counter to…
Not checkable · 10 Jun 2022 |
|
| Unresolvable |
Prediction |
10 Jun 2022 |
Fast AI capability gains seem likely and may break many previous alignment-required invariants simultaneously.
Prediction · 10 Jun 2022 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
10 Jun 2022 |
I figured this stuff out using the null string as input
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
Reading this document cannot make somebody a core alignment researcher.
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
I have a hard time myself feeling hopeful about getting real alignment work out of somebody who previously sat around waiting for somebody…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
This is sufficient on its own, even ignoring many other items on this list, to trash entire categories of naive alignment proposals which…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
The first thing generally, or CEV specifically, is unworkable because the complexity of what needs to be aligned or meta-aligned for our…
Not checkable · 10 Jun 2022 |
|
| Not checkable |
Not checkable |
10 Jun 2022 |
So most organizations don’t have plans, because I haven’t taken the time to personally yell at them.
Not checkable · 10 Jun 2022 |
|