Downstream capabilities such as MMLU or math performance sometimes don't follow from predictable improvements in training loss.
Not researched Checkable, but outside the sample that was researched.
Factual: Statements about the present or the past, checked against the record: did they hold up?
Cite this claim
Demis Hassabis, “Downstream capabilities such as MMLU or math performance sometimes don't follow from predictable improvements in training loss.” (said 28 February 2024). Verdict: not researched. Reliability API, claim c_6b11ca89cb67, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_6b11ca89cb67
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Demis Hassabis
-
Mostly true
“what’s important about them is their 3D structure.”
A protein's 3D structure is what determines, or partially determines, its function.
-
Resolved true
“So over 3 million scientists are now using AlphaFold.”
-
Resolved true
“Only about 10% of drugs actually get through all the clinical stages.”
Only about 10% of drugs get through all the clinical stages.
-
Mostly true
“The AlphaFold Server is empowering non-commercial researchers globally to harness this technology, accelerating their ability to formulate and test new hypotheses.”
-
Mostly true
“Our AlphaProteo model can design novel, high-strength protein binders that target diverse molecules – including those associated with cancer and diabetes.”
Verdicts are made automatically; how verdicts are made.