Claims

Factual · Science and research methods

“In the modern field, overconfidence is no longer noteworthy; but it continues to show up, in passing, in nearly any experiment where subjects are allowed to assign extreme probabilities.”

Eliezer Yudkowsky, essay author · 2008 · Cognitive biases potentially affecting judgment of global risks (UC Berkeley (hosted))

Overconfidence is no longer noteworthy in the modern field but continues to show up in nearly any experiment allowing extreme probabilities.

Mostly true True in substance, wrong in a detail.

What would count as true or false

Judge as of 2008 against the judgement-and-decision-making literature. True: the literature at the time treated overconfidence as a well-established, routine finding (no longer a surprising result in its own right), and it appeared widely in calibration experiments where subjects could give extreme probabilities or confidence levels (e.g. Lichtenstein, Fischhoff & Phillips; Alpert & Raiffa; Griffin & Tversky). Mostly true: the finding is robust, but 'nearly any' overstates it, e.g. because of known exceptions such as the hard-easy effect, underconfidence on easy items, or calibrated experts like weather forecasters. Mixed: the evidence was substantially contested (e.g. the ecological-sampling critiques of Gigerenzer and Juslin) so that the generality holds only in some paradigms. Mostly false: overconfidence appeared only in narrow settings. False: the literature showed overconfidence was not a typical finding.

Evidence

1. ERIC - EJ450925 - The Weighing of Evidence and the Determinants of Confidence., Cognitive Psychology, 1992 ERIC (Education Resources Information Center), abstract for Griffin & Tversky, Cognitive Psychology · 1992

   “Tests the hypotheses that overconfidence occurs when strength is high and weight is low and that underconfidence occurs when weight is high and strength is low in 4 experiments with 153 students.”

2. The Trouble With Overconfidence Psychological Review (Moore & Healy; PDF hosted at Ohio State University) · 2008

   “Noprobleminjudgmentanddecisionmakingismoreprevalentandmorepotentiallycatastrophicthanoverconfidence”

3. The Trouble With Overconfidence Psychological Review (Moore & Healy; PDF hosted at Ohio State University) · 2008

   “Easytasks,whichproducethemostunderestimation,alsoproducethemostoverplacement.”

4. Chapter 20 THE COGNITIVE ILLUSION CONTROVERSY Hertwig & Ortmann (book chapter hosted at Max Planck Society PuRe repository)

   “Gigerenzer et al. (1991) conjectured that the overconfidence effect observed in psychology studies stemmed from the fact that the researchers did not sample general-knowledge questions randomly”

5. Calibration of probabilities: The state of the art to 1980 (Chapter 22) - Judgment under Uncertainty Cambridge University Press · 5 May 2013

   “This chapter reviews the literature concerning yet another aspect of goodness, called calibration.”

Reasoning

Moore & Healy (2008) state that no problem in judgment and decision making is more prevalent than overconfidence, which supports the claim that it was a widespread, established finding by 2008 [E2]. The same literature also records qualifications: Griffin & Tversky (1992) found underconfidence when weight is high and strength is low [E1], and Moore & Healy note that easy tasks produce the most underestimation while also producing the most overplacement [E3]. Gigerenzer et al. (1991) argued that the effect came partly from non-random item selection [E4]. Together this shows a robust finding with documented exceptions and some dispute, so "nearly any experiment" overstates it somewhat, which fits mostly true.

Cite this claim

Eliezer Yudkowsky, “Overconfidence is no longer noteworthy in the modern field but continues to show up in nearly any experiment allowing extreme probabilities.” (said 2008). Verdict: mostly true. Reliability API, claim c_0a137aa38251, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_0a137aa38251