Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 72 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
70 62 78 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 2 promises resolved, 5 needed for a score.
All types combined: 75 (likely 69 to 80, high confidence), from 79 claims: 35 factual claims, 44 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Scott Alexander's claims
39 claims
“They won the CSET-Foretell forecasting competition by an absolutely obscene margin”
Scott Alexander, ACX author 14 March 2022 Mantic Monday 3/14/22 (Astral Codex Ten)
A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.
Resolved true The record shows it happened or was so.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2022-03-14. TRUE: official CSET-Foretell (later INFER) results or leaderboard records show that the team in question (Samotsvety) finished first in the competition, and its relative Brier score was about twice as good as the second-best team's (roughly 1.7x to 2.3x better, or a gap described in the official records as about double). MOSTLY TRUE: the team did win by a large margin, but the margin was clearly smaller or measured differently than 'about twice as good' (for example around 1.4x, or on a different scoring metric). MIXED: the team won by only a modest margin, or won only one season or category of a larger competition. MOSTLY FALSE: the team placed near the top but did not win, or won only a minor subsection that is presented as the whole competition. FALSE: the team did not take part, or finished well below first.
Evidence
- Forecasting Newsletter: December 2021 — LessWrong LessWrong (newsletter by NunoSempere) · 10 January 2022
“As in the first season, Samotsvety Forecasting, a team made up of Eli Lifland, Misha Yagudin, and myself, completely demolished the competition. We were around twice as good as the next-best team in terms of the relative Brier score.”
- How a ragtag band of internet friends became the best at forecasting world events Vox · 13 February 2024
“In 2021, the last full year Samotsvety participated, their score in the Infer tournament was -2.43, compared to -1.039 for the next-best team. They were more than twice as good as the nearest competition.”
- Samotsvety Forecasting Samotsvety Forecasting (team's own site) · 10 September 2022
“1st place in 2021 with a relative score of -3.259 (vs. -0.889 by the 2nd best team and vs. -0.267 by “Pro Forecasters”).”
Reasoning
In a newsletter dated before the claim, E1 says Samotsvety "completely demolished the competition" in the season and was "around twice as good as the next-best team" on relative Brier score. Its author is a team member, though not the subject. E2, an independent Vox report, puts Samotsvety's 2021 Infer relative score at -2.43 against -1.039 for the next-best team, a ratio of about 2.3x, and calls the team "more than twice as good." E3, the team's own track-record page, also lists 1st place in 2021, but with different figures (-3.259 vs -0.889), which would make the gap even larger. No official CSET-Foretell or INFER leaderboard was provided, and the sources give different exact figures, but all of them agree that Samotsvety won by roughly twice the next-best team's score or more.
Cite this claim
Scott Alexander, “A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.” (said 14 March 2022). Verdict: resolved true. Reliability API, claim c_f02e2f0644f8, method 0.1.0+8bea457b. https://epistacklabs.com/claims/c_f02e2f0644f8
Claims without a verdict
Is a verdict here wrong, or do you speak for Scott Alexander? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.