| Not checkable |
Not checkable |
2008 |
AIs will swiftly invent the kind of capabilities that cliche would attribute to human civilization a century or millennium from now
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
They may conflate positive information about the benefits of a technology as negative information about its risks.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This is the way out of the high-mortality phase of the life cycle, the way to close the window of vulnerability; it is also probably the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We don’t instinctively see that alchemy won’t work.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Human emotions take place within an analog brain.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But only if intelligence ceases to be a sacred mystery to us, as life was a sacred mystery to Lord Kelvin.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
What is not acceptable is succeeding at AI and failing at Friendly AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
In computer security, a "trusted system" is one that you are in fact trusting, not one that is in fact trustworthy.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Intelligence is not the first thing human science has encountered that proved difficult to understand.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
Querying your own human brain works fine, as an adaptive instinct, if you need to predict other humans.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
A Friendly AI plus molecular nanotechnology is presumptively powerful enough to solve any problem which can be solved either by moving…
Not checkable · 2008 |
|
| Unresolvable |
Prediction |
2008 |
Human uploading and upgrading will very probably not be the ground on which humanity first confronts smarter-than-human intelligence.
Prediction · 2008 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
2008 |
Note also that an expert strongly confident in their opinion, is quite a different matter from a calculation made strictly from actuarial…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Friendly AI is absent from the conceptual landscape, not just unpopular or unfunded.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Viewing history through the lens of hindsight, we vastly underestimate the cost of preventing catastrophe.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I can't feel our future.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The indispensable protection is an AI that does not want to hurt you.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The easy scenario would hold if e.g., human institutions can reliably distinguish Friendly AIs from unFriendly, and give revocable power…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Mistakenly believing that the past was predictable, people conclude that the future is predictable.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Another moral is that there’s a huge difference between one self-improvement triggering 0.9994 further improvements on average, and one…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
An AI of superhuman intelligence surely could not be modified without its consent.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
All I can do is try to defend it.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Intelligence must cease to be any kind of mystery whatever, sacred or not.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The first flaw you should notice is a Giant Cheesecake Fallacy.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I cannot regard this verdict as good news.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The question is whether the superintelligence wants to build giant cheesecakes.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
The letter R appears in the third-letter position of more English words than in the first-letter position.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
Rather the civil engineer overcomes the burden of improbability through specific choice guided by specific understanding.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Analyses should finally center on testable real-world assertions.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
After September 11th, the U.S. learned not to allow hijacked planes to overfly its cities, but did not learn the broader lesson that Black Swans occur and one should prepare for the unanticipated.
Factual · 2008 |
Other global risks |
| Not checkable |
Not checkable |
2008 |
I think it would be well for estimators of existential risks to know something about heuristics and biases in general, and disconfirmation…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If you believe someone is guilty of a psychological error, then demonstrate your competence by first demolishing their consequential…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I am not willing to embrace a strategy which fails catastrophically if human intelligence enhancement takes longer than building AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This program has made discoveries highly relevant to assessors of global catastrophic risks.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The challenge is to chain your capabilities—the physical-world analogue of combining weak vulnerabilities in a computer system to obtain…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
There are subtler defeats, but this is outright surrender.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Anyone seeking to reduce anthropomorphic bias in themselves would be well-advised to study evolutionary biology for practice, preferably…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
It is a terribly frightening thing, but people do not become any smarter, just because the survival of humankind is at stake.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Stars, chemistry and biology were once mysteries to science.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
The human species, Homo sapiens, is a first mover.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Wishful thinking adds detail, constrains prediction, and thereby creates a burden of improbability.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But preventing the Challenger disaster would have required, not attending to the problem with the O-rings, but attending to every warning…
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Eliezer Yudkowsky proposed the term "Friendly AI" to describe the field of knowledge needed to build an AI with beneficial motivations.
Factual · 2008 |
AI safety and alignment |
| Mostly true |
Factual |
2008 |
The bystander effect is usually explained as resulting from diffusion of responsibility and pluralistic ignorance.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
then no scientist will ever again make a suggestion, which is worse.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
That is why - despite frequent great temptation - I avoid discussing the research on calibration unless I have previously spoken of the…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The wishful thinking involved was probably more analogous to alchemy than civil engineering.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Truth is hard work, and not the kind of hard work done by storytellers.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
The conjunction fallacy has survived all experimental tests meant to disprove it.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
And maybe then we can win.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Not knowing how to build a friendly AI is not deadly, of itself, in any specific instance, if you know you don’t know.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If you deal out cards from a deck, one after another, you will eventually deal out the ace of clubs.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
by the time your neurons finish thinking the words “I should do something” you have already lost.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But this makes AI catastrophes more worrisome, not less.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Therefore I strongly recommend reading
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I’m saying that it would be easier to build the 747, and then have the 747, metaphorically speaking, upgrade the bird.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Scope neglect is the hazard of being a biological human, running on an analog brain; the brain cannot multiply by six billion.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
anthropomorphism is flypaper for unwary scientists, a trap so sticky that it takes a Darwin to escape.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We are not aware of substituting judgment of representativeness for judgment of probability.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
People refuse to buy flood insurance even when it is heavily subsidized and priced far below actuarially fair value.
Factual · 2008 |
Economics and markets |
| Not checkable |
Not checkable |
2008 |
Unanimous strategies are unworkable, which does not stop people from proposing them.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This is not a domain in which I am willing to give narrow confidence intervals
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The value of information emerges from the value of whatever it is information about; if you double the stakes, you double the value of…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
AI has safety issues; so does human intelligence enhancement.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We change our minds less often than we think.
Not checkable · 2008 |
|
| Unresolvable |
Prediction |
2008 |
An AI might make a huge jump in intelligence after reaching some threshold of criticality.
Prediction · 2008 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
2008 |
It should be obvious that this is neither logically true, nor practically a good philosophy.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But the student of heuristics and biases will recognize at least two major mistakes in the method - not logical flaws, but conditions…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
I do not want to play down the colossal audacity of trying to build, to a precise purpose and design, something smarter than ourselves.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
But the human mind is not the limit of the possible.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
"Hot" (emotionally charged) confirmation biases are stronger, with larger effects and more resistance to change, than "cold" ones.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
I tend to assume arbitrarily large potential jumps for intelligence because (a) this is the conservative assumption;
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The two statements, sadly, are not remotely equivalent.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Black Swans are an especially difficult version of the problem of the fat tails
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Natural selection did not design the human brain with multiple rings of precautionary measures, conservative decision processes, and orders of magnitude of safety margin.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
The critical inference is not that AI is hard, but that, for some reason, it is very easy for people to think they know far more about…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This form of failure is especially dangerous because it will appear to work within a fixed context, then fail when the context changes.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The human species, Homo sapiens, is unaging but not immortal.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Cognitive biases are settled science; one need simply quote the literature.
Not checkable · 2008 |
|
| Unresolvable |
Factual |
2008 |
The people who first invented and spread communism were idealists acting with good intentions.
Factual · 2008 |
Philosophy, ethics and society |
| Not checkable |
Not checkable |
2008 |
The species metaphor would seem the nearest analogy a priori, but it does not lend itself to making up detailed stories.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
This book aims to introduce heuristics and biases to an intelligent general audience.
Not checkable · 2008 |
|
| Unresolvable |
Prediction |
2008 |
An Artificial Intelligence could rewrite its code from scratch and change the underlying dynamics of optimization.
Prediction · 2008 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
2008 |
The vision leaps directly from capability to actuality, without considering the necessary intermediate of motive.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
If you realize that expert opinions have less force than you thought, you had better also realize that your own thoughts have much less…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
The better the computing hardware, the less understanding you need to build an AI.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Asking people for a point estimate and then confidence bounds leads them to adjust from the estimate until values sound implausible, producing under-adjustment and too-narrow confidence bounds.
Factual · 2008 |
Science and research methods |
| Not checkable |
Not checkable |
2008 |
Risks of human extinction may tend to be underestimated since, obviously, humanity has never yet encountered an extinction event.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
Because we remember future civilizations surprising past civilizations, it has become cliche that we can’t put limits on our…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
When you read a paper which proposes that a potential global catastrophe is impossible, or has a specific annual probability, or can be…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
It is this enormous space of possibilities which outlaws anthropomorphism as legitimate reasoning.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
Taber and Lodge's experiments confirmed all six of the authors' prior hypotheses.
Factual · 2008 |
Science and research methods |
| Not researched |
Citation |
2008 |
Studies, including Carson and Mitchell (1995), show scope neglect applies to human lives in willingness to pay.
Citation · 2008 |
Science and research methods |
| Unresolvable |
Prediction |
2008 |
Yudkowsky said he thinks it is possible for fallible humans to succeed at building Friendly AI, but only if intelligence ceases to be a sacred mystery to us.
Prediction · 2008 |
AI safety and alignment |
| Unresolvable |
Prediction |
2008 |
There will not necessarily be distant advance warning before AI is created, so it cannot be relied upon.
Prediction · 2008 |
AI timelines and forecasting |
| Not checkable |
Not checkable |
2008 |
But the code is not given to the AI, for the AI to look over and hand back if it does the wrong thing. The code is the AI.
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We should avoid, not only being duped by fiction - failing to expend the mental effort necessary to 'unbelieve' it - but also being…
Not checkable · 2008 |
|
| Not checkable |
Not checkable |
2008 |
We only need to get Friendly AI right in one place and one time, not every time everywhere.
Not checkable · 2008 |
|
| Not researched |
Factual |
2008 |
As of 2006, the AI research community did not see Friendly AI as part of the problem.
Factual · 2008 |
AI safety and alignment |
| Not checkable |
Not checkable |
2008 |
And we need to solve the Friendly AI challenge before Artificial General Intelligence is created, not afterward;
Not checkable · 2008 |
|