Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 70 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
60 42 78
All types combined: 74 (likely 68 to 80, high confidence), from 79 claims: 71 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Eliezer Yudkowsky's claims
8 claims
“It's still not very good at long multiplication (unless you give it the right instructions, in which case suddenly it can do it).”
Eliezer Yudkowsky, interviewee on Bankless podcast February 2023 Full Transcript: Eliezer Yudkowsky on the Bankless podcast (LessWrong)
ChatGPT is not very good at long multiplication unless given the right instructions, in which case it can do it.
Mixed Partly true and partly false in roughly equal measure.
What would count as true or false
Judge as of February 2023, against ChatGPT versions available then (GPT-3.5 based; GPT-4 was not yet released). The claim has two parts: (a) with plain prompts, ChatGPT often gets multi-digit long multiplication wrong; (b) with suitable instructions (for example step-by-step or chain-of-thought prompting that breaks the problem into digit-level steps), it can do the task reliably or mostly correctly. True: tests or reports from around that time show both parts. Mostly true: both parts hold broadly, but prompted performance is clearly better without being fully reliable, or holds only up to a moderate number of digits. Mixed: only one of the two parts is supported. Mostly false: ChatGPT was in fact fairly good at long multiplication without special prompting, or instructions gave only marginal improvement, though some basis for the claim exists. False: evidence shows ChatGPT reliably did long multiplication unprompted, or that no prompting allowed it to do so.
Evidence
- Why Can't Little Chatty Do Math? Computational Complexity blog · 9 February 2023
“Despite OpenAI's claim that ChatGPT has improved mathematical capabilities, we don't get far multiplying large numbers.”
- Why Can't Little Chatty Do Math? Computational Complexity blog · 9 February 2023
“ChatGPT can't multiply but it does know how to write a program to multiply.”
- Analyzing ChatGPT's Mathematical Deficiencies ACL Anthology (ROCLING 2023)
“Forthearithmetictask,weobservethatalthoughtheaccuracyofmultiplicationforfourdigitsorhigheris0,theAPEscoresarearound20%.”
- Analyzing ChatGPT's Mathematical Deficiencies ACL Anthology (ROCLING 2023)
“Foreachsample,wepromptedChatGPTwiththeinstruction,”Respondwithonlytheanswertothefollowingquestion”
Reasoning
Part (a) is supported. A February 2023 blog post reports that "we don't get far multiplying large numbers" with ChatGPT (E1). An independent paper found 0 accuracy on multiplication of four or more digits when ChatGPT was asked for only the answer (E3, E4). Part (b) is not supported: E2 says only that ChatGPT knows how to write a program to multiply, which is not the same as doing the multiplication itself with step-by-step instructions. No item tests chain-of-thought or digit-level prompting, so only one of the two parts is supported.
Cite this claim
Eliezer Yudkowsky, “ChatGPT is not very good at long multiplication unless given the right instructions, in which case it can do it.” (said February 2023). Verdict: mixed. Reliability API, claim c_94bff25d6c5c, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_94bff25d6c5c
Claims without a verdict
Is a verdict here wrong, or do you speak for Eliezer Yudkowsky? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.