Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 70 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
60 42 78
All types combined: 74 (likely 68 to 80, high confidence), from 79 claims: 71 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Eliezer Yudkowsky's claims
46 claims
“In the modern field, overconfidence is no longer noteworthy; but it continues to show up, in passing, in nearly any experiment where subjects are allowed to assign extreme probabilities.”
Eliezer Yudkowsky, essay author 2008 Cognitive biases potentially affecting judgment of global risks (UC Berkeley (hosted))
Overconfidence is no longer noteworthy in the modern field but continues to show up in nearly any experiment allowing extreme probabilities.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge as of 2008 against the judgement-and-decision-making literature. True: the literature at the time treated overconfidence as a well-established, routine finding (no longer a surprising result in its own right), and it appeared widely in calibration experiments where subjects could give extreme probabilities or confidence levels (e.g. Lichtenstein, Fischhoff & Phillips; Alpert & Raiffa; Griffin & Tversky). Mostly true: the finding is robust, but 'nearly any' overstates it, e.g. because of known exceptions such as the hard-easy effect, underconfidence on easy items, or calibrated experts like weather forecasters. Mixed: the evidence was substantially contested (e.g. the ecological-sampling critiques of Gigerenzer and Juslin) so that the generality holds only in some paradigms. Mostly false: overconfidence appeared only in narrow settings. False: the literature showed overconfidence was not a typical finding.
Evidence
- ERIC - EJ450925 - The Weighing of Evidence and the Determinants of Confidence., Cognitive Psychology, 1992 ERIC (Education Resources Information Center), abstract for Griffin & Tversky, Cognitive Psychology · 1992
“Tests the hypotheses that overconfidence occurs when strength is high and weight is low and that underconfidence occurs when weight is high and strength is low in 4 experiments with 153 students.”
- The Trouble With Overconfidence Psychological Review (Moore & Healy; PDF hosted at Ohio State University) · 2008
“Noprobleminjudgmentanddecisionmakingismoreprevalentandmorepotentiallycatastrophicthanoverconfidence”
- The Trouble With Overconfidence Psychological Review (Moore & Healy; PDF hosted at Ohio State University) · 2008
“Easytasks,whichproducethemostunderestimation,alsoproducethemostoverplacement.”
- Chapter 20 THE COGNITIVE ILLUSION CONTROVERSY Hertwig & Ortmann (book chapter hosted at Max Planck Society PuRe repository)
“Gigerenzer et al. (1991) conjectured that the overconfidence effect observed in psychology studies stemmed from the fact that the researchers did not sample general-knowledge questions randomly”
- Calibration of probabilities: The state of the art to 1980 (Chapter 22) - Judgment under Uncertainty Cambridge University Press · 5 May 2013
“This chapter reviews the literature concerning yet another aspect of goodness, called calibration.”
Reasoning
Moore & Healy (2008) state that no problem in judgment and decision making is more prevalent than overconfidence, which supports the claim that it was a widespread, established finding by 2008 [E2]. The same literature also records qualifications: Griffin & Tversky (1992) found underconfidence when weight is high and strength is low [E1], and Moore & Healy note that easy tasks produce the most underestimation while also producing the most overplacement [E3]. Gigerenzer et al. (1991) argued that the effect came partly from non-random item selection [E4]. Together this shows a robust finding with documented exceptions and some dispute, so "nearly any experiment" overstates it somewhat, which fits mostly true.
Cite this claim
Eliezer Yudkowsky, “Overconfidence is no longer noteworthy in the modern field but continues to show up in nearly any experiment allowing extreme probabilities.” (said 2008). Verdict: mostly true. Reliability API, claim c_0a137aa38251, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_0a137aa38251
Claims without a verdict
Is a verdict here wrong, or do you speak for Eliezer Yudkowsky? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.