Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
67 49 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 79 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Rodney Brooks's claims
39 claims
“In essence ChatGPT is picking the most likely word to follow all those that precede it, whether it is the words in a human input prompt, or what it has already generated itself.”
Rodney Brooks, blog author 1 January 2023 Predictions Scorecard, 2023 January 01 (rodneybrooks.com)
In essence, ChatGPT picks the most likely next word given the preceding words, whether from the user's prompt or its own generated output.
Mostly true True in substance, wrong in a detail.
What would count as true or false
True: technical documentation from OpenAI or authoritative technical accounts show that ChatGPT generates text autoregressively, predicting the next token from all preceding tokens (the prompt plus its own earlier output), so that the description is accurate in substance. Mostly true: the autoregressive next-token mechanism is confirmed, and the only differences are minor technical details the claim glosses over: tokens rather than words, and sampling (temperature) rather than always taking the single most likely option, with RLHF shaping the probabilities. Mixed: the next-token mechanism is right, but a substantial part of how it works is materially misdescribed. Mostly false: the description misrepresents the core generation process in a significant way. False: ChatGPT does not generate text by predicting successive tokens conditioned on prior context.
Evidence
- What Is ChatGPT Doing … and Why Does It Work? Stephen Wolfram (writings.stephenwolfram.com) · 14 February 2023
“The first thing to explain is that what ChatGPT is always fundamentally trying to do is to produce a “reasonable continuation” of whatever text it’s got so far, where by “reasonable” we mean “what one might expect someone to write after seeing what people have written on billions of webpages, etc.””
- What Is ChatGPT Doing … and Why Does It Work? Stephen Wolfram (writings.stephenwolfram.com) · 14 February 2023
“Because when ChatGPT is going to generate a new token, it always “reads” (i.e. takes as input) the whole sequence of tokens that come before it, including tokens that ChatGPT itself has “written” previously.”
- What is LLM Temperature? | IBM IBM · 16 December 2024
“LLMs generate text by predicting the next word (or rather, the next token) according to a probability distribution.”
- Introducing ChatGPT - OpenAI OpenAI · 30 November 2022
“We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup.”
Reasoning
Independent technical accounts confirm that ChatGPT generates text autoregressively: it produces a continuation of the text so far (E1), and each new token is generated from the whole preceding sequence, including its own earlier output (E2). The claim glosses over some minor technical details. The unit is a token rather than a word, and the next token is predicted according to a probability distribution (E3). The probabilities are also shaped by RLHF training (E4). These are the minor differences the criteria list for mostly true.
Cite this claim
Rodney Brooks, “In essence, ChatGPT picks the most likely next word given the preceding words, whether from the user's prompt or its own generated output.” (said 1 January 2023). Verdict: mostly true. Reliability API, claim c_cded391652d6, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_cded391652d6
Claims without a verdict
Is a verdict here wrong, or do you speak for Rodney Brooks? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.