ChatGPT is not very good at long multiplication unless given the right instructions, in which case it can do it.
Mixed Partly true and partly false in roughly equal measure.
Factual: Statements about the present or the past, checked against the record: did they hold up?
What would count as true or false
Judge as of February 2023, against ChatGPT versions available then (GPT-3.5 based; GPT-4 was not yet released). The claim has two parts: (a) with plain prompts, ChatGPT often gets multi-digit long multiplication wrong; (b) with suitable instructions (for example step-by-step or chain-of-thought prompting that breaks the problem into digit-level steps), it can do the task reliably or mostly correctly. True: tests or reports from around that time show both parts. Mostly true: both parts hold broadly, but prompted performance is clearly better without being fully reliable, or holds only up to a moderate number of digits. Mixed: only one of the two parts is supported. Mostly false: ChatGPT was in fact fairly good at long multiplication without special prompting, or instructions gave only marginal improvement, though some basis for the claim exists. False: evidence shows ChatGPT reliably did long multiplication unprompted, or that no prompting allowed it to do so.
Set before the research began.
Evidence (4)
-
E1 Why Can't Little Chatty Do Math? Computational Complexity blog · 9 February 2023
“Despite OpenAI's claim that ChatGPT has improved mathematical capabilities, we don't get far multiplying large numbers.”
-
E2 Why Can't Little Chatty Do Math? Computational Complexity blog · 9 February 2023
“ChatGPT can't multiply but it does know how to write a program to multiply.”
-
E3 Analyzing ChatGPT's Mathematical Deficiencies ACL Anthology (ROCLING 2023)
“Forthearithmetictask,weobservethatalthoughtheaccuracyofmultiplicationforfourdigitsorhigheris0,theAPEscoresarearound20%.”
-
E4 Analyzing ChatGPT's Mathematical Deficiencies ACL Anthology (ROCLING 2023)
“Foreachsample,wepromptedChatGPTwiththeinstruction,”Respondwithonlytheanswertothefollowingquestion”
Reasoning
Part (a) is supported. A February 2023 blog post reports that "we don't get far multiplying large numbers" with ChatGPT (E1). An independent paper found 0 accuracy on multiplication of four or more digits when ChatGPT was asked for only the answer (E3, E4). Part (b) is not supported: E2 says only that ChatGPT knows how to write a program to multiply, which is not the same as doing the multiplication itself with step-by-step instructions. No item tests chain-of-thought or digit-level prompting, so only one of the two parts is supported.
Cite this claim
Eliezer Yudkowsky, “ChatGPT is not very good at long multiplication unless given the right instructions, in which case it can do it.” (said February 2023). Verdict: mixed. Reliability API, claim c_94bff25d6c5c, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_94bff25d6c5c
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Eliezer Yudkowsky
-
Resolved true
“We learned that that's false, but we did not learn that you can have a thing that can redesign a tree and build an AI much smarter than itself, that builds an AI much smarter than itself, and that…”
-
Mostly true
“I was a passionate noticer of the human brain not being homogenous, and was therefore unpleasantly surprised by transformers scaling as well as they did.”
-
Mostly true
“First of all, you realize in principle that the task of minimizing losses on predicting human text does not stop when you’re as smart as a human, right?”
-
Mostly true
“Transformers are the main thing like that.”
-
Mostly true
“Because the basic paradigm of machine learning that we are presently operating under is that you can have the loss function, but only for things you can evaluate.”
Verdicts are made automatically; how verdicts are made.