Scott Alexander

Factual · Science and research methods

“They won the CSET-Foretell forecasting competition by an absolutely obscene margin”

Scott Alexander, ACX author · 14 March 2022 · Mantic Monday 3/14/22 (Astral Codex Ten)

A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.

Resolved true The record shows it happened or was so.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2022-03-14. TRUE: official CSET-Foretell (later INFER) results or leaderboard records show that the team in question (Samotsvety) finished first in the competition, and its relative Brier score was about twice as good as the second-best team's (roughly 1.7x to 2.3x better, or a gap described in the official records as about double). MOSTLY TRUE: the team did win by a large margin, but the margin was clearly smaller or measured differently than 'about twice as good' (for example around 1.4x, or on a different scoring metric). MIXED: the team won by only a modest margin, or won only one season or category of a larger competition. MOSTLY FALSE: the team placed near the top but did not win, or won only a minor subsection that is presented as the whole competition. FALSE: the team did not take part, or finished well below first.

Evidence

1. Forecasting Newsletter: December 2021 — LessWrong LessWrong (newsletter by NunoSempere) · 10 January 2022

   “As in the first season, Samotsvety Forecasting, a team made up of Eli Lifland, Misha Yagudin, and myself, completely demolished the competition. We were around twice as good as the next-best team in terms of the relative Brier score.”

2. How a ragtag band of internet friends became the best at forecasting world events Vox · 13 February 2024

   “In 2021, the last full year Samotsvety participated, their score in the Infer tournament was -2.43, compared to -1.039 for the next-best team. They were more than twice as good as the nearest competition.”

3. Samotsvety Forecasting Samotsvety Forecasting (team's own site) · 10 September 2022

   “1st place in 2021 with a relative score of -3.259 (vs. -0.889 by the 2nd best team and vs. -0.267 by “Pro Forecasters”).”

Reasoning

In a newsletter dated before the claim, E1 says Samotsvety "completely demolished the competition" in the season and was "around twice as good as the next-best team" on relative Brier score. Its author is a team member, though not the subject. E2, an independent Vox report, puts Samotsvety's 2021 Infer relative score at -2.43 against -1.039 for the next-best team, a ratio of about 2.3x, and calls the team "more than twice as good." E3, the team's own track-record page, also lists 1st place in 2021, but with different figures (-3.259 vs -0.889), which would make the gap even larger. No official CSET-Foretell or INFER leaderboard was provided, and the sources give different exact figures, but all of them agree that Samotsvety won by roughly twice the next-best team's score or more.

Cite this claim

Scott Alexander, “A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.” (said 14 March 2022). Verdict: resolved true. Reliability API, claim c_f02e2f0644f8, method 0.1.0+8bea457b. https://epistacklabs.com/claims/c_f02e2f0644f8