| Not checkable |
Not checkable |
2008 |
When using long, complex chains of reasoning to argue in favor of an anthropomorphic conclusion, each and every step of the reasoning is…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
To get a really good answer you must sacrifice your ability to predict the answer, albeit not your ability to say what is the question.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Vivid, specific scenarios can inflate our probability estimates of security, as well as misdirecting defensive investments into needlessly…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The glorious thing about formal mathematical proof, is that a proof of ten billion steps is just as reliable as a proof of ten steps.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We would last only until the probabilities caught up with us.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Such is the power of creativity.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
It is no law of Nature that physical operations must crawl at the pace of long seconds.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
“Fairly good” is not good enough to last another million years.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Sufficiently tall skyscrapers don’t potentially start doing their own engineering.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If we suppose the g-factor metaphor, then global catastrophic risks of this scenario are relatively mild; a hostile AI is not much more of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Anthropomorphic bias can be classed as insidious: it takes place with no deliberate intent, without conscious realization, and in the face…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
To improve the truth of our beliefs we must change our beliefs.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A success on human intelligence enhancement would make Friendly AI easier, and also help on other technologies.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If you throw too much money at a problem that a field is not prepared to solve, the excess money is more likely to produce anti-science…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Molecular nanotechnology fits this criterion, first because its elementary operations are fast, and second because there exists a ready…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The prospect of AI interacts with global risk in more complex ways than that; if AI were a pure liability, matters would be simple.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
All that vast potential hinges on our survival here, now, in the days when the realm of humankind is a single planet orbiting a single star.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
EP is a powerful, still maturing technique that is intrinsically unsuited to the demands of Friendly AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
An optimization process steers the future into particular regions of the possible.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But if people are liable to confidently believe that the button does something quite different from its actual consequence, that is cause…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I don’t think this exact scenario is plausible, mostly because I don’t expect the curve of recursive self-improvement to move at a linear…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
There is a taboo against talking about humanlevel capabilities.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Nature does not match her challenges to your skill, or your resources, or how much free time you have to think about the problem.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Storytellers obey strict rules of narrative unrelated to reality.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Any technology which increases available computing power decreases the minimum theoretical sophistication necessary to develop Artificial…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Of all possible configurations for the atoms making up the Corolla, only an infinitesimal fraction qualify as a useful working car.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Yet for a futurist, disjunctions make for an awkward and unpoetic-sounding prophecy.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The danger of confusing general intelligence with g-factor is that it leads to tremendously underestimating the potential impact of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The subjects in these experiments don't make mistakes on purpose; they make mistakes because they don't know how to do better.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
To build an AI that discovers the orbits of the planets, the programmers need not know the math of Newtonian mechanics, only the math of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The above methodology for expert elicitation looks perfectly reasonable
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This may have been one contributing factor to the embarrassing history of AI, which stems not from the difficulty of AI as such, but from the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The scenario of humanity going extinct in the next century is a disjunctive event.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Since the given description of Linda was chosen to be similar to a feminist and dissimilar to a bank teller, (8) is more representative of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
g-factor metaphors seem most common in popular futurism: when people think of “intelligence” they think of human geniuses instead of humans.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The second threat scenario is less vivid and alarming, but the defense is more useful because it is more vague.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Natural selection isn’t friendly, nor does it hate you, nor will it leave you alone.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Often Nature poses requirements that are grossly unfair, even on tests where the penalty for failure is death.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But human beings are not trustworthy to peer over a purported proof of ten billion steps; we have too high a chance of missing an error.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We must execute the creation of Artificial Intelligence as the exact application of an exact art.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
People may be overconfident and over-optimistic.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Awareness of human fallibility is a dangerous knowledge, if you remind yourself of the fallibility of those who disagree with you.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The first moral is that confusing the speed of AI research with the speed of a real AI once built is like confusing the speed of physics…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A majoritarian strategy only works if it is not possible for a single defector to cause global catastrophic damage.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
And when you run into a lethal challenge too difficult for you, you die.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Every true idea which discomforts you will seem to match the pattern of at least one psychological error.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The main advice the metaphor gives us is that we had better get Friendly AI right, which is good advice in any case.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But in forecasting, every added detail is an extra burden!
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The elapsed turnaround time would be, imaginably, on the order of a week from when the fast intelligence first became able to solve the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Familiarity with heuristics and biases can enable us to detect a wide variety of logical flaws that might otherwise evade our inspection.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A majoritarian strategy is sometimes workable, if you have decades in which to do your work.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
As logic, the above argument contains holes.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Few people would push a button that they clearly knew would cause a global catastrophe.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I introduce the concept of an optimization process
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Because of hindsight bias, we learn overly specific lessons.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We would expect a true AI, an Artificial General Intelligence, to be capable of changing its empirical beliefs
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
And the stakes of existential risk extend beyond even the six billion humans alive today, to all the stars in all the galaxies that…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But if they had said might, I would have no objection.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The raw materials for AI are already everywhere.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If the AI tribe wins in military conflict and wipes out the humans, that is an existential catastrophe of the Bang variety
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Substantially larger numbers, such as 500 million deaths, and especially qualitatively different scenarios such as the extinction of the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I do not assign strong confidence to the assertion that Friendly AI is easier than human augmentation, or that it is safer.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
It only takes a single counterexample to falsify a universal quantifier.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
They are somewhat less accessible to a general audience.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Otherwise you become slower to react to incoming evidence.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But one should not so quickly write off the challenge, especially considering the stakes.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Intelligence is the foundation of human power, the strength that fuels our other arts.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
When we talk about “AIs” we are really talking about minds-in-general, or optimization processes in general.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A dead planet, lifelessly orbiting its star, is also stable.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But the stakes are much, much higher.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Absolute power is a conservative engineering assumption in Friendly AI, exposing broken designs.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I cannot see how nanotechnology would make it easier to develop Friendly AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
There is no way to obtain more truth for a proposition by bribery, flattery, or the most passionate argument - you can make more people…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Friendly AI is not about brute-forcing the problem.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Any two AI designs might be less similar to one another than you are to a petunia.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The AI might transcend the factual error of its programmers, provided that the programmers understood inference rather better than they…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A technical failure occurs when the code does not do what you think it does, though it faithfully executes as you programmed it.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
To live even a million years, as an unaging human in a world as risky as our own, you must somehow drive your annual probability of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Someone should know that when an expert says the probability is "a million to one" without using actuarial data or calculations from a…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The first AI built cannot by itself do catastrophic damage.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
It once occurred to me that modern civilization occupies an unstable state.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Heuristics and biases are widespread in human reasoning.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Friendly AI programmers should assume that the AI has total access to its own source code.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Do-What-I-Mean is a major, nontrivial technical challenge of Friendly AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
AIs will swiftly invent the kind of capabilities that cliche would attribute to human civilization a century or millennium from now
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
They may conflate positive information about the benefits of a technology as negative information about its risks.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This is the way out of the high-mortality phase of the life cycle, the way to close the window of vulnerability; it is also probably the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We don’t instinctively see that alchemy won’t work.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Human emotions take place within an analog brain.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But only if intelligence ceases to be a sacred mystery to us, as life was a sacred mystery to Lord Kelvin.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
What is not acceptable is succeeding at AI and failing at Friendly AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
In computer security, a "trusted system" is one that you are in fact trusting, not one that is in fact trustworthy.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Querying your own human brain works fine, as an adaptive instinct, if you need to predict other humans.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A Friendly AI plus molecular nanotechnology is presumptively powerful enough to solve any problem which can be solved either by moving…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Note also that an expert strongly confident in their opinion, is quite a different matter from a calculation made strictly from actuarial…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Friendly AI is absent from the conceptual landscape, not just unpopular or unfunded.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Viewing history through the lens of hindsight, we vastly underestimate the cost of preventing catastrophe.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I can't feel our future.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The indispensable protection is an AI that does not want to hurt you.
Not checkable · 2008 |
|