Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“OpenAI has begun to incorporate user experiments into the training corpus, killing the scientific community’s ability to test the single most critical question: the ability of these models to generalize to new test cases.”
Gary Marcus, Substack author, relaying Ernie Davis 24 March 2023 The Sparks of AGI? Or the End of Science? (Marcus on AI)
Citing Ernie Davis, Marcus said OpenAI had begun to incorporate user experiments into its models' training corpus.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judged as of 2023-03-24. True: OpenAI documentation, policies or credible reporting show that by that date OpenAI was using user conversations or prompts from ChatGPT or its API (including users' test prompts) as training data for its models. Mostly true: OpenAI was using user data to train or fine-tune models (e.g. ChatGPT conversations under its data-usage policy), but there is no specific evidence that the 'experiments' testing generalisation were included. Mixed: OpenAI kept the right to train on user data and some use is documented, but how far it went before March 2023 is unclear. Mostly false: user data was used only for narrow purposes, such as safety review or feedback ratings, and not in the training corpus, with the claim overstating this. False: OpenAI's policies and practice show that user inputs were not used for training at that time.
Evidence
- Addressing criticism, OpenAI will no longer use customer data to train its models by default | TechCrunch TechCrunch · 1 March 2023
“Developers have long taken issue with OpenAI’s (now-deprecated) data processing policy, which they claim posed a privacy risk and allowed the company to profit off of their data.”
- ChatGPT is back in Italy – What changes have been made and what do users need to know? DSN Group · 5 May 2023
“Until recently, all conversations taking place between private users and the bot were subject to being used for training the algorithm, which opened the possibility of user inputs being used by the machine for future public responses.”
- Italy lifts ban on ChatGPT after data privacy improvements Deutsche Welle (DW) · 29 April 2023
“The country's data protection watchdog said its developers did not have a legal basis to justify the storage and collection of users' personal data in order to train the site's algorithms.”
- Sam Altman: OpenAI won't tap into customer APIs CNBC · 5 May 2023
“We don't train on any API data at all, we haven't for a while,” Altman told CNBC.”
Reasoning
E2 reports that until spring 2023, all conversations between private users and ChatGPT could be used to train the model. E3 reports that Italy's regulator objected that OpenAI had no legal basis to collect users' personal data to train its algorithms. E1 notes that developers had criticised OpenAI's earlier data-processing policy, which they said let the company profit from their data. Together these indicate that, before spring 2023, OpenAI's policies allowed user conversations to be used for training, and that this drew criticism from developers [E1] and Italy's regulator [E3]. However, no item shows specifically that users' experiments testing generalisation went into the training corpus, and E4 has Altman saying API data had not been used for training "for a while".
Cite this claim
Gary Marcus, “Citing Ernie Davis, Marcus said OpenAI had begun to incorporate user experiments into its models' training corpus.” (said 24 March 2023). Verdict: mostly true. Reliability API, claim c_5fcf16ad3613, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_5fcf16ad3613
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.