Looking back at his 2014, 2015 and 2016 predictions, Scott Alexander said he is probably generally a little underconfident in medium-probability predictions.

This claim is about what Scott Alexander wrote. True: his 2016-12-31 calibration post, reviewing his predictions for 2014 to 2016, states that he is probably generally a little underconfident in medium-probability predictions. Mostly true: he says this but frames it as covering only some of those years, or with slightly different wording that keeps the meaning. Mostly false: he describes his calibration in a materially different way, for example overconfidence, or underconfidence at high rather than medium probabilities. False: the post contains no such statement.

Set before the research began.

1. E1 Scott Alexander's Self-Evaluations - Econlib Econlib (Liberty Fund) · 6 April 2017

   “Since he assigns probabilities to each prediction, Scott can graph his ex ante subjective probabilities against the ex post objective probabilities.”

2. E2 Scott Alexander's Self-Evaluations - Econlib Econlib (Liberty Fund) · 6 April 2017

   “Of 70% predictions, I got 13 right and 3 wrong, for a score of 81%”

3. E3 Scott Alexander's Self-Evaluations - Econlib Econlib (Liberty Fund) · 6 April 2017

   “It looks like he is underestimating the probabilities.”

The criteria turn on whether Scott Alexander's 2016-12-31 calibration post says he is probably a little underconfident in medium-probability predictions. None of the evidence quotes that sentence or the post itself. A later Econlib post describes Alexander's calibration graphs (E1), quotes one result, 81% correct on 70% predictions (E2), and says Alexander appears to be underestimating probabilities (E3). That fits underconfidence, but the quoted result is not tied to a stated year and does not show what the post said about medium probabilities across 2014–2016.