Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“It will remain turbocharged pastiche generator, and a fine tool for brainstorming, and for first drafts, but not trustworthy general intelligence.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4 will remain a pastiche generator and useful for brainstorming and first drafts, but not trustworthy general intelligence.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge GPT-4 as released by OpenAI (March 2023) and as assessed during its first year or so of use. The claim's testable parts are: (a) GPT-4 is useful for brainstorming and first drafts; (b) it is not trustworthy general intelligence, meaning it still makes notable errors, hallucinations or reasoning failures that make it unreliable without human checking, and it is not broadly regarded as AGI. 'Pastiche generator' is a characterisation; judge it only as consistent with GPT-4 producing text by recombining patterns from its training data. TRUE: both (a) and (b) hold according to independent evaluations, OpenAI's own GPT-4 technical report or system card, and widespread reporting. MOSTLY TRUE: (b) holds, but the 'pastiche' characterisation is notably disputed, for example by credible evidence of novel generalisation, while it is still not trustworthy general intelligence. MIXED: GPT-4 is unreliable in some major uses but credibly regarded as trustworthy and general in others. MOSTLY FALSE: there is wide expert agreement that GPT-4 is general intelligence, with only minor reliability caveats. FALSE: there is broad consensus that GPT-4 is trustworthy general intelligence.
Evidence
- GPT-4 | OpenAI OpenAI · 14 March 2023
“Despite its capabilities, GPT‑4 has similar limitations as earlier GPT models. Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).”
- GPT-4 Can’t Reason (Position Paper) arXiv (ar5iv mirror) · 26 June 2023
“Based on the results of that analysis, this paper argues that, despite the occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.”
- Sparks of Artificial General Intelligence: Early experiments with GPT-4 arXiv (ar5iv mirror)
“These examples suggest that GPT-4 has not only learned some general principles and patterns of different domains and styles but can also synthesize them in creative and novel ways.”
- Sparks of Artificial General Intelligence: Early experiments with GPT-4 arXiv (ar5iv mirror)
“Given the breadth and depth of GPT-4’s capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system.”
Reasoning
OpenAI's GPT-4 launch announcement says the model "is not fully reliable" and still hallucinates facts and makes reasoning errors (E1). A position paper argues that GPT-4 is "utterly incapable of reasoning" (E2). Together these support part (b): GPT-4 is not trustworthy general intelligence. The Microsoft 'Sparks' paper argues GPT-4 could reasonably be viewed as an "early (yet still incomplete)" AGI (E4), which is one research group's view and not evidence of broad consensus that it is AGI. But the same paper says its examples suggest GPT-4 can synthesize learned patterns "in creative and novel ways" (E3), which disputes the 'pastiche' description. No item directly shows part (a), that GPT-4 is useful for brainstorming and first drafts.
Cite this claim
Gary Marcus, “GPT-4 will remain a pastiche generator and useful for brainstorming and first drafts, but not trustworthy general intelligence.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_f7e1127f94c5, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_f7e1127f94c5
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.