ChatGPT regularly makes things up, and a fair bit of what it says is untrue.
Mostly true True in substance, wrong in a detail.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2022-12-25 for ChatGPT as released in November 2022. True: contemporaneous evidence (OpenAI's own launch notes acknowledging plausible-sounding but incorrect answers, documented examples, or early evaluations) shows that ChatGPT regularly produced fabricated or false statements, at a non-trivial rate in ordinary use. Mostly true: fabrication was common, but only in certain domains (e.g., citations, facts, math), so 'a fair bit' somewhat overstates the overall rate. Mixed: evidence shows errors happened but is split on whether they were regular or a meaningful share of output. Mostly false: errors were real but rare, so the claim generalises from isolated cases. False: evidence shows ChatGPT seldom produced false statements.
Set before the research began.
Evidence (4)
-
E1 Introducing ChatGPT OpenAI · 30 November 2022
“ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”
-
E2 AI-generated answers temporarily banned on coding Q&A site Stack Overflow The Verge · 5 December 2022
“The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce,”
-
E3 AI Platforms like ChatGPT Are Easy to Use but Also Potentially Dangerous Scientific American · 19 December 2022
“Disturbingly, both the journal reference and the statistics were invented.”
-
E4 ChatGPT: How Customer Service Can Benefit From a GPT-3 Chatbot BotsCrew · 27 December 2022
“Indeed, ChatGPT's responses may be occasionally wrong.”
Reasoning
In its launch post, OpenAI acknowledged that ChatGPT "sometimes writes plausible-sounding but incorrect or nonsensical answers" (E1). Stack Overflow moderators said its coding answers had "a high rate of being incorrect" (E2). Scientific American reported a documented case of an invented journal reference and statistic (E3). Together these show regular fabrication at a non-trivial rate, but the strongest rate evidence comes from specific domains (coding answers and citations). The other sources call the errors "sometimes" (E1) or "occasionally" (E4, from 27 December 2022, two days after the judge-as-of date), so "a fair bit" of overall output is not directly measured.
Cite this claim
Gary Marcus, “ChatGPT regularly makes things up, and a fair bit of what it says is untrue.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_21aa4d048691, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_21aa4d048691
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mostly true
“Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”
Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.
-
Mostly true
“Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”
-
Mostly true
“Despite all the hype, agents didn’t turn out to be reliable.”
-
Mixed
“but the systems don’t have the knowledge graph to do that.”
-
Resolved true
“Hasn’t worked so far. That’s for sure.”
Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.
Verdicts are made automatically; how verdicts are made.