Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“Although ChatGPT can write about anything, it is also easily confused.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
ChatGPT can write about anything but is easily confused.
Mostly true True in substance, wrong in a detail.
The second judge said resolved true; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2022-12-25, using evidence about ChatGPT as released in late 2022. The claim has two parts: (a) ChatGPT will produce text on essentially any topic it is prompted with; (b) it is easily confused, meaning ordinary prompts readily lead it into contradictions, factual errors, nonsensical reasoning or mistakes caused by misreading the prompt. True: contemporaneous evidence (documented examples, evaluations, or reporting from Dec 2022 to early 2023) shows both: it generates text across a very wide range of topics, and it commonly makes such errors without adversarial effort. 'Anything' is rhetorical, so refusals on some topics do not defeat (a). Mostly true: both parts hold broadly, but the confusion was shown only on certain kinds of task, or the topic range was noticeably limited by refusals. Mixed: only one of the two parts is supported. Mostly false: there are occasional documented errors, but evidence shows ChatGPT was generally robust and not readily confused, so the claim overstates. False: evidence shows ChatGPT was not easily confused in ordinary use, and it could not write across a broad range of topics.
Evidence
- Indiana Chamber of Commerce · 8 February 2023
“In the simplest terms, ChatGPT can take a request or query from a person and write a comprehensive document on just about any topic in a seemingly endless array of formats.”
- What is ChatGPT And How Can You Use It? Search Engine Journal · 26 December 2022
“ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers.”
- What is ChatGPT And How Can You Use It? Search Engine Journal · 26 December 2022
“Stack Overflow was flooded with user responses generated from ChatGPT that appeared to be correct, but a great many were wrong answers.”
- Indiana Chamber of Commerce · 8 February 2023
“it can even debug computer code, admit mistakes, challenge incorrect premises and reject inappropriate requests.”
Reasoning
E1, an independent source, says ChatGPT could write a comprehensive document on "just about any topic" in many formats, which supports part (a). On part (b), E2 (Dec 26, 2022) says ChatGPT sometimes gave plausible-sounding but incorrect or nonsensical answers. E3 reports that many ChatGPT-generated answers posted on Stack Overflow appeared correct but were wrong. The documented errors are described as occurring "sometimes" [E2], and E4 notes it could admit mistakes and challenge incorrect premises, so "easily confused" holds broadly but not fully.
Cite this claim
Gary Marcus, “ChatGPT can write about anything but is easily confused.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_d372ea32a20f, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_d372ea32a20f
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.