Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
81 76 87 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 2 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
65 45 83
All types combined: 80 (likely 75 to 85, high confidence), from 84 claims: 78 factual claims, 6 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Demis Hassabis's claims
37 claims
“I think we helped pioneer RLHF and other things like that which can obviously be used for performance but also for safety.”
Demis Hassabis, CEO of Google DeepMind 28 February 2024 Demis Hassabis — Scaling, superhuman AIs, AlphaZero atop LLMs, AlphaFold (Dwarkesh Podcast)
Hassabis said he thinks Google DeepMind helped pioneer RLHF.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
The claim is that Google DeepMind (including its predecessor DeepMind) was among those who helped pioneer reinforcement learning from human feedback, not that it was the sole originator. True if DeepMind researchers made foundational early contributions to RLHF, for example as co-authors of the 2017 paper 'Deep reinforcement learning from human preferences' (Christiano et al.) or of other seminal early RLHF work. Mostly true if DeepMind's contribution was real and early but secondary to another lab's. Mixed if DeepMind's early role was minor or disputed. Mostly false if DeepMind's RLHF work came only after the technique was established elsewhere. False if DeepMind had no part in developing RLHF.
Evidence
- Deep Reinforcement Learning from Human Preferences arXiv (Christiano, Leike, Brown, Martic, Legg, Amodei)
“Jan Leike DeepMind leike@google.com Tom B Brown nottombrown@gmail.com Miljan Martic DeepMind miljanm@google.com Shane Legg DeepMind legg@google.com”
- Learning through human feedback DeepMind · 12 June 2017 Own record
“This research was done as part of an ongoing collaboration between Jan Leike, Miljan Martic, and Shane Legg at DeepMind and Paul Christiano, Dario Amodei, and Tom Brown at OpenAI.”
- A Tiny History of RLHF | RLHF and Post-Training Book by Nathan Lambert RLHF and Post-Training Book (Nathan Lambert)
“One of the first papers with an approach similar to modern RLHF was TAMER. TAMER: Training an Agent Manually via Evaluative Reinforcement proposed an approach in which humans iteratively scored an agent’s actions to learn a reward model, which was used to learn the action policy [4].”
- A Survey of Reinforcement Learning from Human Feedback arXiv
“Comparisons were first used for direct policy learning (Akrour et al., 2011; Cheng et al., 2011), but were later extended to the reward-learning setting (Wirth et al., 2016; Christiano et al., 2017).”
Reasoning
The 2017 paper 'Deep Reinforcement Learning from Human Preferences' lists three DeepMind-affiliated co-authors: Jan Leike, Miljan Martic and Shane Legg (E1). DeepMind's 2017 blog post describes the work as a collaboration between DeepMind and OpenAI (E2). The criteria name co-authorship of this paper as an example of a foundational early contribution. Earlier preference-based methods existed (E3, E4), but the claim is only that DeepMind helped pioneer RLHF, not that it was the sole originator, so the claim resolves true.
Cite this claim
Demis Hassabis, “Hassabis said he thinks Google DeepMind helped pioneer RLHF.” (said 28 February 2024). Verdict: resolved true. Reliability API, claim c_9e8fdd0460e3, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_9e8fdd0460e3
Claims without a verdict
Is a verdict here wrong, or do you speak for Demis Hassabis? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.