A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2022-03-14. TRUE: official CSET-Foretell (later INFER) results or leaderboard records show that the team in question (Samotsvety) finished first in the competition, and its relative Brier score was about twice as good as the second-best team's (roughly 1.7x to 2.3x better, or a gap described in the official records as about double). MOSTLY TRUE: the team did win by a large margin, but the margin was clearly smaller or measured differently than 'about twice as good' (for example around 1.4x, or on a different scoring metric). MIXED: the team won by only a modest margin, or won only one season or category of a larger competition. MOSTLY FALSE: the team placed near the top but did not win, or won only a minor subsection that is presented as the whole competition. FALSE: the team did not take part, or finished well below first.
Set before the research began.
Evidence (3)
-
E1 Forecasting Newsletter: December 2021 — LessWrong LessWrong (newsletter by NunoSempere) · 10 January 2022
“As in the first season, Samotsvety Forecasting, a team made up of Eli Lifland, Misha Yagudin, and myself, completely demolished the competition. We were around twice as good as the next-best team in terms of the relative Brier score.”
-
E2 How a ragtag band of internet friends became the best at forecasting world events Vox · 13 February 2024
“In 2021, the last full year Samotsvety participated, their score in the Infer tournament was -2.43, compared to -1.039 for the next-best team. They were more than twice as good as the nearest competition.”
-
E3 Samotsvety Forecasting Samotsvety Forecasting (team's own site) · 10 September 2022
“1st place in 2021 with a relative score of -3.259 (vs. -0.889 by the 2nd best team and vs. -0.267 by “Pro Forecasters”).”
Reasoning
In a newsletter dated before the claim, E1 says Samotsvety "completely demolished the competition" in the season and was "around twice as good as the next-best team" on relative Brier score. Its author is a team member, though not the subject. E2, an independent Vox report, puts Samotsvety's 2021 Infer relative score at -2.43 against -1.039 for the next-best team, a ratio of about 2.3x, and calls the team "more than twice as good." E3, the team's own track-record page, also lists 1st place in 2021, but with different figures (-3.259 vs -0.889), which would make the gap even larger. No official CSET-Foretell or INFER leaderboard was provided, and the sources give different exact figures, but all of them agree that Samotsvety won by roughly twice the next-best team's score or more.
Cite this claim
Scott Alexander, “A team described as some of the best superforecasters in the world won the CSET-Foretell forecasting competition by a large margin, described as around twice as good as the next-best team in relative Brier score.” (said 14 March 2022). Verdict: resolved true. Reliability API, claim c_f02e2f0644f8, method 0.1.0+8bea457b, dataset release v1.1.2. https://epistacklabs.com/claims/c_f02e2f0644f8
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Scott Alexander
-
Mostly true
“Some of it is certainly genetic – estimates of the genetic contribution to political association range from 0.4 to 0.6.”
Estimates of the genetic contribution to political association range from 0.4 to 0.6.
-
Mostly true
“Heritability of one’s attitudes toward gay rights range from 0.3 to 0.5, which hilariously is a little more heritable than homosexuality itself.”
-
Resolved true
“they found that people’s unconscious partisan biases were half again as strong as their unconscious racial biases”
-
Mostly true
“MNX is a real-money prediction market on AI.”
-
Resolved true
“Friend of the blog Goodheart Labs is working on Common Notes, aka “Twitter Community Notes, but for everything”.”
Goodheart Labs is working on Common Notes, described as Twitter Community Notes but for everything.
Verdicts are made automatically; how verdicts are made.