There have already been reports of contamination in GPT-4's training data.

This is a factual claim judged as of 2023-03-24. It is true if, on or before 2023-03-24, at least one public report (from researchers, journalists or credible commentators) gave evidence or a credible argument that GPT-4's training data was contaminated with benchmark or test material. Examples include GPT-4 solving coding problems published before its training cutoff but not newer ones, or memorizing test sets. OpenAI's own technical report acknowledging partial contamination also counts. Mostly true: such reports existed but were informal or tentative (for example a widely shared analysis on social media) rather than formal findings. Mixed: the only reports before that date were about earlier models, or about GPT-4's performance generally, with only a loose link to contamination. Mostly false: contamination concerns were raised only as speculation, with no specific report about GPT-4 dated before 2023-03-24. False: there is positive evidence that no contamination concern about GPT-4 had been raised by that date, for example the first such reports are dated later.

Set before the research began.

1. E1 GPT-4 and professional benchmarks: the wrong answer to the wrong question normaltech.ai (Narayanan and Kapoor newsletter) · 20 March 2023

   “Surprisingly, Horace He pointed out that GPT-4 solved 10/10 pre-2021 problems and 0/10 recent problems in the easy category. The training data cutoff for GPT-4 is September 2021.”

2. E2 GPT-4 and professional benchmarks: the wrong answer to the wrong question normaltech.ai (Narayanan and Kapoor newsletter) · 20 March 2023

   “In fact, we can definitively show that it has memorized problems in its training set: when prompted with the title of a Codeforces problem, GPT-4 includes a link to the exact contest where the problem appears”

3. E3 GPT-4 Technical Report OpenAI (arXiv)

   “A minority of the problems in the exams were seen by the model during training; for each exam we run a variant with these questions removed and report the lower score of the two.”

4. E4 GPT-4 Technical Report OpenAI (arXiv)

   “During our contamination check we discovered that portions of BIG-bench [48] were inadvertently mixed into the training set, and we excluded it from our reported results.”

Narayanan and Kapoor's newsletter post dated 2023-03-20, four days before the claim, cites Horace He's observation that GPT-4 solved 10 of 10 easy Codeforces problems from before 2021 but 0 of 10 recent ones, and ties this to GPT-4's September 2021 training cutoff (E1). In the same post, the authors say their own testing shows GPT-4 has memorized problems from its training set, because it gives links to the exact contests when prompted with problem titles (E2). This is a specific public report from credible researchers, made before 2023-03-24, of benchmark contamination in GPT-4's training data. The excerpts from OpenAI's technical report (E3, E4) also describe contamination, but the version cited has no date, so the verdict does not rely on them.