Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
81 76 87 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 2 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
65 45 83
All types combined: 80 (likely 75 to 85, high confidence), from 84 claims: 78 factual claims, 6 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Demis Hassabis's claims
42 claims
“Not much progress made for a decade.”
Demis Hassabis, Nobel lecture slides December 2024 Accelerating scientific discovery with AI (Nobel lecture slides) (NobelPrize.org)
Little progress was made in CASP protein structure prediction accuracy for a decade before AlphaFold.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge against published CASP results (predictioncenter.org and CASP assessment papers) for roughly CASP7–CASP12 (about 2006–2016), before AlphaFold's entry in CASP13 (2018). True: top GDT_TS scores on free-modelling/hard targets were largely flat over about a decade. Mostly true: progress was small and slow, with modest gains (e.g. some improvement around CASP11–12 from contact prediction) that the claim's rough wording understates. Mixed: the sources disagree substantially, or real progress came in a substantial part of that decade. Mostly false: CASP accuracy rose substantially over that decade before AlphaFold. False: the records show large, steady gains throughout.
Evidence
- Critical Assessment of Methods of Protein Structure Prediction (CASP) – Progress and New directions in Round XI Proteins (CASP11 assessment review; NIHPA Author Manuscript) · 1 June 2016
“free modeling – FM, now often effective for small proteins, but largely stuck for the last decade until new contact prediction methods partially came to the rescue this round”
- AlphaFold @ CASP13: “What just happened?” moalquraishi.wordpress.com (Mohammed AlQuraishi, Harvard Medical School) · 9 December 2018
“Historically progress in CASP has ebbed and flowed, with a ten year period of almost absolute stagnation, finally broken by the advances seen at CASP11 and 12, which were substantial.”
- Critical Assessment of Methods of Protein Structure Prediction (CASP) – Progress and New directions in Round XI Proteins (CASP11 assessment review; NIHPA Author Manuscript) · 1 June 2016
“Although overall model accuracy has not improved much when the increased size of sequence and structure databases is taken into account (Figures 5 and 6), examination of improvement in accuracy without correcting for database changes shows impressive improvement (Figure 7).”
- Critical Assessment of Methods of Protein Structure Prediction (CASP) – Progress and New directions in Round XI Proteins (CASP11 assessment review; NIHPA Author Manuscript) · 1 June 2016
“Results here also suggest no substantial progress in overall model accuracy.”
Reasoning
The CASP11 assessment review describes free-modelling accuracy as "largely stuck for the last decade" until new contact prediction methods partly helped in CASP11 (E1). It also reports no substantial progress in overall model accuracy (E4), and little improvement once database growth is accounted for, though without that correction it reports impressive improvement (E3). An independent academic account likewise describes about ten years of near-stagnation, but says it was broken by substantial advances at CASP11 and CASP12 before AlphaFold entered (E2). The claim therefore broadly holds, but its wording leaves out the real gains made around CASP11–12, which matches the "mostly true" criterion.
Cite this claim
Demis Hassabis, “Little progress was made in CASP protein structure prediction accuracy for a decade before AlphaFold.” (said December 2024). Verdict: mostly true. Reliability API, claim c_1192e90f1487, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_1192e90f1487
Claims without a verdict
Is a verdict here wrong, or do you speak for Demis Hassabis? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.