ChatGPT is frequently retuned based on feedback, but weeks later Marcus still keeps being sent tweets showing its blunders.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge said mixed; the verdict shown is the first judge's.
What would count as true or false
Judged as of 2022-12-25. The claim has two parts: (a) ChatGPT was updated or retuned based on feedback during December 2022; (b) weeks after its 30 November 2022 launch, people were still circulating tweets about ChatGPT errors and sending them to Marcus. True: both hold. Evidence can include OpenAI release notes or update announcements, plus tweets from mid-to-late December 2022 that show ChatGPT blunders, such as those embedded in Marcus's post. Mostly true: both hold, but retuning was less frequent than 'frequently' suggests, for example only a single update. Mixed: only one part is supported. False: there was no retuning and no such tweets.
Set before the research began.
Evidence (4)
-
E1 ChatGPT update: OpenAI details changes to popular chatbot Tech Monitor · 16 December 2022
“OpenAI has published the first major update to its viral chatbot ChatGPT since it launched last month, including performance enhancements and daily usage limits.”
-
E2 "I think this is the most disruptive technology" Exploring Sentiments of ChatGPT Early Adopters using Twitter Data arXiv · 7 December 2022
“The negative sentiments for Q&A Testing topics are evolved due to the wrong, incorrect, and sometimes invalid answers for their questions.”
-
E3 OpenAI's new ChatGPT bot: 10 dangerous things it's capable of BleepingComputer · 6 December 2022
“Note, both these answers sound plausible. Both are wrong. pic.twitter.com/5ZMWkBZ6Kp”
-
E4 GitHub - giuven95/chatgpt-failures: Failure archive for ChatGPT and similar models GitHub (user giuven95) · 2 January 2023
“Failed to reproduce on the Dec 15 version”
Reasoning
E1 and E4 show that ChatGPT was updated in mid-December 2022, and E2 and E3 show that tweets about its wrong answers were circulating in early December. None of the evidence shows such tweets in the weeks after launch or being sent to Marcus, so part (b) of the criteria is not settled.
Cite this claim
Gary Marcus, “ChatGPT is frequently retuned based on feedback, but weeks later Marcus still keeps being sent tweets showing its blunders.” (said 25 December 2022). Verdict: unresolvable. Reliability API, claim c_50e734d4a581, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_50e734d4a581
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mostly true
“Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”
Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.
-
Mostly true
“Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”
-
Mostly true
“Despite all the hype, agents didn’t turn out to be reliable.”
-
Mixed
“but the systems don’t have the knowledge graph to do that.”
-
Resolved true
“Hasn’t worked so far. That’s for sure.”
Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.
Verdicts are made automatically; how verdicts are made.