Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
42 claims
“Everything must be taken on faith (and yet there already have been reports of contamination in the training data).”
Gary Marcus, Substack author 24 March 2023 The Sparks of AGI? Or the End of Science? (Marcus on AI)
There have already been reports of contamination in GPT-4's training data.
Resolved true The record shows it happened or was so.
What would count as true or false
This is a factual claim judged as of 2023-03-24. It is true if, on or before 2023-03-24, at least one public report (from researchers, journalists or credible commentators) gave evidence or a credible argument that GPT-4's training data was contaminated with benchmark or test material. Examples include GPT-4 solving coding problems published before its training cutoff but not newer ones, or memorizing test sets. OpenAI's own technical report acknowledging partial contamination also counts. Mostly true: such reports existed but were informal or tentative (for example a widely shared analysis on social media) rather than formal findings. Mixed: the only reports before that date were about earlier models, or about GPT-4's performance generally, with only a loose link to contamination. Mostly false: contamination concerns were raised only as speculation, with no specific report about GPT-4 dated before 2023-03-24. False: there is positive evidence that no contamination concern about GPT-4 had been raised by that date, for example the first such reports are dated later.
Evidence
- GPT-4 and professional benchmarks: the wrong answer to the wrong question normaltech.ai (Narayanan and Kapoor newsletter) · 20 March 2023
“Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021.”
- GPT-4 and professional benchmarks: the wrong answer to the wrong question normaltech.ai (Narayanan and Kapoor newsletter) · 20 March 2023
“In fact, we can definitively show that it has memorized problems in its training set: when prompted with the title of a Codeforces problem, GPT-4 includes a link to the exact contest where the problem appears”
- GPT-4 Technical Report OpenAI (arXiv)
“A minority of the problems in the exams were seen by the model during training; for each exam we run a variant with these questions removed and report the lower score of the two.”
- GPT-4 Technical Report OpenAI (arXiv)
“During our contamination check we discovered that portions of BIG-bench [48] were inadvertently mixed into the training set, and we excluded it from our reported results.”
Reasoning
Narayanan and Kapoor's newsletter post dated 2023-03-20, four days before the claim, cites Horace He's observation that GPT-4 solved 10 of 10 easy Codeforces problems from before 2021 but 0 of 10 recent ones, and ties this to GPT-4's September 2021 training cutoff (E1). In the same post, the authors say their own testing shows GPT-4 has memorized problems from its training set, because it gives links to the exact contests when prompted with problem titles (E2). This is a specific public report from credible researchers, made before 2023-03-24, of benchmark contamination in GPT-4's training data. The excerpts from OpenAI's technical report (E3, E4) also describe contamination, but the version cited has no date, so the verdict does not rely on them.
Cite this claim
Gary Marcus, “There have already been reports of contamination in GPT-4's training data.” (said 24 March 2023). Verdict: resolved true. Reliability API, claim c_09399c7c89dc, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_09399c7c89dc
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.