Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
82 76 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
61 47 75 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
67 46 85
All types combined: 77 (likely 72 to 82, high confidence), from 84 claims: 63 factual claims, 16 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Nate Silver's claims
37 claims
“This mostly makes a difference for very low-probability events.”
Nate Silver, FiveThirtyEight editor-in-chief, column author 2016 Election Update: Why Our Model Is More Bullish Than Others On Trump (FiveThirtyEight)
The t-distribution in FiveThirtyEight's model mostly makes a difference for very low-probability events.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judged as of 2016. The claim is that using a fat-tailed t-distribution rather than a normal distribution in FiveThirtyEight's election model changed results mainly for very low-probability outcomes, i.e. in the tails. True: FiveThirtyEight's methodology documents, together with standard statistical properties, show the t-distribution's effect is concentrated in tail probabilities (e.g. a candidate with a small chance of winning a state) and is small for outcomes near the middle. Mostly true: the effect is concentrated in the tails but also has some non-trivial effect on moderate probabilities. Mixed: the effect is spread fairly evenly between tail and moderate probabilities. Mostly false: the t-distribution materially changed moderate probabilities as well. False: the model did not use a t-distribution, or the effect was mainly on moderate probabilities.
Evidence
- What’s Wrong With 538? HuffPost · 7 November 2016
“I am not questioning any specific assumption that 538 makes, from their state-by-state correlations to their use of a t-distribution to create “fat tails” in their probability distributions (i.e., higher likelihood of otherwise obscure events).”
- What’s Wrong With 538? HuffPost · 7 November 2016
“Perhaps Nate Silver can say that he’s gotten all of these improbable outcomes because of his use of the t-distribution rather than a normal distribution to accentuate the tails of the model.”
- Accurate Prediction of Electoral Outcomes arXiv (academic preprint by outside authors)
“If we study the histogram in Figure 24, it is not clear that fat-tailed distributions are relevant here, unless we believe there are non-trivial probabilities of a landslide.”
- Accurate Prediction of Electoral Outcomes arXiv (academic preprint by outside authors)
“The probabilities provided by the Student T regression as well as the Hierarchical Model were very similar to the CAPM Model.”
Reasoning
An independent 2016 HuffPost piece confirms that FiveThirtyEight used a t-distribution to create "fat tails" in its probability distributions, which it describes as a "higher likelihood of otherwise obscure events" [E1]. The same piece describes the t-distribution as a way to "accentuate the tails of the model" [E2]. That matches the claim that the effect falls mainly on low-probability outcomes. No FiveThirtyEight methodology document is in the evidence, and nothing measures how much the choice changed moderate probabilities, so the claim cannot be fully confirmed. The arXiv preprint [E3, E4] studies a different model, so it is not used.
Cite this claim
Nate Silver, “The t-distribution in FiveThirtyEight's model mostly makes a difference for very low-probability events.” (said 2016). Verdict: mostly true. Reliability API, claim c_4c6e5be817ea, method 0.1.0+28da144b. https://epistacklabs.com/claims/c_4c6e5be817ea
Claims without a verdict
Is a verdict here wrong, or do you speak for Nate Silver? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.