Claims

Factual · Science and research methods

“Like, almost nothing is 40% causative on anything out of a very small number of exceptions.”

Marc Andreessen, guest · 2 September 2023 · Marc Andreessen on Why AI Will Save the World (EconTalk)

Almost nothing is 40% causative of anything, apart from a very small number of exceptions.

Mostly true True in substance, wrong in a detail.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

The claim is a general statement about effect sizes: very few single factors account for about 40% or more of the causation (or variance) of any outcome. True: reviews or meta-analyses of effect sizes across fields (social science, medicine, genetics) show that single factors explaining about 40% or more of an outcome are rare exceptions. Mostly true: such effects are uncommon but somewhat more frequent than 'a very small number', or this holds in the fields the claim plausibly refers to but not universally. Mixed: the evidence is split, with large single-factor effects common in some major domains and rare in others. Mostly false: the claim is stated as established but the literature cannot settle it, or it overstates a real tendency for effects to be small. False: the literature shows single-factor effects of about 40% or more are common across many domains.

Evidence

1. Evaluating Effect Size in Psychological Research: Sense and Nonsense Association for Psychological Science (Advances in Methods and Practices in Psychological Science) · 1 June 2019

   “A very large effect size ( r = .40 or greater) in the context of psychological research is likely to be a gross overestimate that will rarely be found in a large sample or in a replication.”

2. Empirical effect size comparison studies (2020 overview) Emil Kirkegaard (personal blog compilation) · 4 March 2020

   “The analysis reveals that the 25th, 50th, and 75th percentiles corresponded to correlation coefficients values of 0.12, 0.25, and 0.42 and to Hedges’ g values of 0.15, 0.38, and 0.69, respectively.”

3. Is psychological science progressing? Explained variance in PsycINFO articles during the period 1956 to 2022 PubMed Central (NCBI); journal name not shown in fetched text · 21 December 2022

   “While the average ratio is lower, there are enough cases left where researchers can happily claim to have found a large effect ranging around 40%.”

4. Effect Size Magnification: No Variable Is as Important as the One You’re Thinking About—While You’re Thinking About It SAGE Publications (Current Directions in Psychological Science) · 11 October 2024

   “In genetics these are the exception, not the rule (O’Connor, 2021), and we expect fewer to exist in psychology.”

5. Empirical effect size comparison studies (2020 overview) Emil Kirkegaard (personal blog compiling published studies) · 4 March 2020

   “The analysis reveals that the 25th, 50th, and 75th percentiles corresponded to correlation coefficients values of 0.12, 0.25, and 0.42 and to Hedges’ g values of 0.15, 0.38, and 0.69, respectively.”

6. Empirical effect size comparison studies (2020 overview) Emil Kirkegaard (personal blog compiling published studies) · 4 March 2020

   “Based on the results, it is suggested that Cohen’s correlation guidelines are too exigent, as <3% of correlations in the literature were found to be as large as r = 0.50.”

7. Is psychological science progressing? Explained variance in PsycINFO articles during the period 1956 to 2022 PubMed Central (PMC); journal name not shown in fetched text · 21 December 2022

   “For all 1,565 articles, the mean amount of (maximum) explained variance was 42.8 percent with 95 percent confidence interval from 41.7 to 43.9.”

8. Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study PMC (National Library of Medicine), authors of the ABCD study analysis · 23 September 2021

   “In aim one analyses, the median in-sample effect size was .03, and values at the first and third quartiles were .01 and .07.”

9. Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study PMC (National Library of Medicine), authors of the ABCD study analysis · 23 September 2021

   “For example, amongst false discovery rate corrected, between-instrument, mixed effect models (“real-world” analysis), the 90th percentile of effect size was a correlation of .18 and the 95th percentile was .29.”

10. Recalibrating expectations about effect size: A multi-method survey of effect sizes in the ABCD study PMC (National Library of Medicine), authors of the ABCD study analysis · 23 September 2021

    “One recent meta-analysis from six prominent psychology journals found that the average published effect size was r = .19 and that the 90th percentile of effect sizes was only r = .41 [7].”

11. One Hundred Years of Social Psychology Quantitatively Described SAGE Publications / Review of General Psychology (APA Division 1) · December 2003

    “Analyses reveal that social psychological effects typically yield a value of *r* equal to.21 and that, in the typical research literature, effects vary from study to study in ways that produce a standard deviation in *r* of.15.”

Reasoning

E1 calls a correlation of r = .40 or more very large in psychology and says it is rarely found in large samples or replications. E7 reports that under 3% of literature correlations reach r = .50. E10 and E11 put the upper percentiles of effect sizes at modest levels: a 95th percentile of r = .29 in one dataset and a 90th percentile of r = .41 in another. E5 says large single-variable effects in genetics are the exception, not the rule. Against this, E2/E6 report a 75th-percentile correlation of .42 in one compilation, and E3 notes that some studies still claim large effects of around 40%. The evidence therefore supports the claim mainly for psychology and genetics, not across all fields, which fits mostly true.

Cite this claim

Marc Andreessen, “Almost nothing is 40% causative of anything, apart from a very small number of exceptions.” (said 2 September 2023). Verdict: mostly true. Reliability API, claim c_a4e9b737e802, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_a4e9b737e802