Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“The first is that the two giant OpenAI and Microsoft papers have been about a model about which absolutely nothing has been revealed, not the architecture, nor the training set.”
Gary Marcus, Substack author 24 March 2023 The Sparks of AGI? Or the End of Science? (Marcus on AI)
The OpenAI and Microsoft GPT-4 papers reveal nothing about the model's architecture or training set.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2023-03-24, against OpenAI's GPT-4 Technical Report (March 2023) and Microsoft's 'Sparks of Artificial General Intelligence' paper (March 2023). True: neither paper discloses substantive information about GPT-4's architecture (for example model size, layer structure, mixture-of-experts) or its training set (data sources, composition, size), beyond generic statements. Mostly true: the papers give only minimal, generic descriptions, for example 'a Transformer-style model pre-trained on publicly available data and data licensed from third-party providers' and fine-tuned with RLHF, which make 'absolutely nothing' a slight overstatement while the substance holds. Mixed: one paper discloses meaningful architecture or training-data details and the other does not. Mostly false: the papers disclose several meaningful details about the architecture or training data. False: the papers give substantial specifics about both.
Evidence
- OpenAI's GPT-4 Is Closed Source and Shrouded in Secrecy Vice · 16 March 2023
“Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method.”
- OpenAI's GPT-4 Is Closed Source and Shrouded in Secrecy Vice · 16 March 2023
“This means that OpenAI did not disclose what it used to train the model or how it trained the model, including the energy costs and hardware used for it, making GPT-4 the company's most secretive release thus far.”
- OpenAI never disclosed GPT-4's size or architecture CenterConsulting · 14 March 2023
“Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”
Reasoning
E1 and E3 quote the GPT-4 Technical Report as saying it "contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method." E2 independently reports that OpenAI did not disclose what it used to train GPT-4 or how it trained it. The words "no further details" suggest the report gave some minimal, generic description, so "absolutely nothing" slightly overstates it, though the substance holds. None of the evidence covers Microsoft's Sparks paper, so this verdict rests mainly on the OpenAI report.
Cite this claim
Gary Marcus, “The OpenAI and Microsoft GPT-4 papers reveal nothing about the model's architecture or training set.” (said 24 March 2023). Verdict: mostly true. Reliability API, claim c_45e5958ed6a9, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_45e5958ed6a9
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.