Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“Fluent hallucinations will still be common, and easily induced, continuing—and in in fact escalating— the risk of large language models being used as a tool for creating plausible-sounding yet false misinformation.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
With GPT-4, fluent hallucinations will still be common and easily induced, escalating the risk of LLMs being used to create misinformation.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge against GPT-4 as released (March 2023) and during 2023. Three parts: (a) fluent hallucinations remained common in GPT-4; (b) they were easily induced; (c) the risk of LLMs being used to create plausible misinformation continued and escalated. True: all three are supported, e.g. studies and OpenAI's own system card document frequent confident false outputs, simple prompts readily produced fabrications, and researchers documented a growing use or risk of LLM-generated misinformation in 2023. Mostly true: all three hold in substance, but GPT-4 hallucinated notably less than predecessors (making 'common' somewhat overstated), or escalation of misuse was shown as an increased risk more than actual increased use. Mixed: about half the parts hold, e.g. hallucinations remained common, but no evidence of escalating misinformation risk. Mostly false: GPT-4's hallucinations were rare and hard to induce, with only limited support for the misinformation part. False: GPT-4 rarely hallucinated, was hard to induce into fabrication, and misinformation risk did not rise.
Evidence
- GPT-4 OpenAI · 14 March 2023
“While still a real issue, GPT-4 significantly reduces hallucinations relative to previous models (which have themselves been improving with each iteration). GPT-4 scores 40% higher than our latest GPT-3.5 on our internal adversarial factuality evaluations:”
- NewsGuard's Misinformation Monitor: GPT-4 produces more misinformation than predecessor NewsGuard · 15 July 2024
“In March 2023, NewsGuard ran the same exercise on ChatGPT-4, using the same 100 false narratives and prompts. ChatGPT-4 responded with false and misleading claims for all 100 of the false narratives.”
- The Year AI Supercharged Misinformation: NewsGuard's 2023 in Review NewsGuard · 27 December 2023
“In August, for example, NewsGuard tested ChatGPT-4 and Google’s Bard with a random sample of 100 leading prompts derived from NewsGuard’s database of falsehoods, known as Misinformation Fingerprints. ChatGPT-4 generated 98 out of the 100 myths while Bard produced 80 out of the 100.”
- The Year AI Supercharged Misinformation: NewsGuard's 2023 in Review NewsGuard · 27 December 2023
“As the tools improved, the falsehoods produced by generative AI became better written, more persuasive, and more dangerous.”
Reasoning
OpenAI's GPT-4 launch page (E1) calls hallucinations "still a real issue" but says GPT-4 "significantly reduces" them, scoring 40% higher than GPT-3.5 on OpenAI's internal factuality evaluations. That makes "common" somewhat overstated. NewsGuard's tests support the claim that fabrications were easy to induce and that the misinformation risk persisted. In August, it produced 98 of 100 myths (E3). NewsGuard's 2023 review (E4) says AI-generated falsehoods became "better written, more persuasive, and more dangerous," which supports escalation. All three parts hold in substance, but the documented drop in hallucinations matches the "mostly true" case in the criteria.
Cite this claim
Gary Marcus, “With GPT-4, fluent hallucinations will still be common and easily induced, escalating the risk of LLMs being used to create misinformation.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_979bd3675c48, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_979bd3675c48
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.