Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“GPT-4 is going to be a monster.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4 will be a very large, powerful system (a 'monster').
Mostly true True in substance, wrong in a detail.
What would count as true or false
This vague prediction means GPT-4 would be a very large, very capable system. Judge it at GPT-4's release (March 2023). An undisclosed parameter count does not by itself settle it. TRUE: on release, GPT-4 was widely recognized as a major jump in capability over GPT-3.5 and prior systems, shown by large benchmark and professional-exam gains in OpenAI's technical report and confirmed by independent evaluations and broad coverage. MOSTLY TRUE: GPT-4 was clearly a significant step up, but some assessments called the gains moderate in certain areas. MIXED: assessments were split between seeing it as a major leap and seeing it as an incremental upgrade. MOSTLY FALSE: GPT-4 was generally seen as an incremental improvement with modest gains. FALSE: GPT-4 was widely regarded as underwhelming or no meaningful advance over GPT-3.5.
Evidence
- GPT-4 OpenAI · 14 March 2023
“For example, it passes a simulated bar exam with a score around the top 10% of test takers; in contrast, GPT‑3.5’s score was around the bottom 10%.”
- OpenAI Says GPT-4 Is Better in Nearly Every Way. What Matters More Is Millions Will Use It SingularityHub · 19 March 2023
“It also outperformed the upgraded GPT-3 (called GPT 3.5) on a slew of standardized tests, placing among the top 10 percent of human test-takers on the bar licensing exam for lawyers and scoring either a 4 or a 5 on 13 out of 15 college-level advanced placement (AP) exams for high school students.”
- Why GPT-4 is notable but not groundbreaking University of Sydney (originally published in The Conversation) · 16 March 2023
“One could argue GPT-4 represents only an incremental improvement over its predecessors in many practical scenarios. Results showed human judges preferred GPT-4 outputs over the most advanced variant of GPT-3.5 only about 61% of the time.”
- OpenAI Says GPT-4 Is Better in Nearly Every Way. What Matters More Is Millions Will Use It SingularityHub · 19 March 2023
“While GPT-4 may not to be the step change some predicted, the scale of its deployment almost certainly will be.”
Reasoning
OpenAI's launch report (E1) shows large exam gains over GPT-3.5, such as a simulated bar exam score near the top 10% of test takers, compared with near the bottom 10% for GPT-3.5. Launch-week coverage (E2) confirmed it outperformed GPT-3.5 on many standardized tests. Some independent assessments, however, called the gains moderate: E3 described GPT-4 as an incremental improvement in many practical scenarios, citing a 61% human preference rate over GPT-3.5, and E4 said it may not be the step change some predicted. This fits the 'mostly true' criterion: a clear, significant step up, with some assessments calling the gains moderate.
Cite this claim
Gary Marcus, “GPT-4 will be a very large, powerful system (a 'monster').” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_8c4cd6808d99, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_8c4cd6808d99
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.