Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
213 claims · page 3 of 3
“@OpenAI is SoTA on some QA tests, but no match for humans on story understanding.”
Gary Marcus, writing on X 16 February 2019 Post by @GaryMarcus on X, 2019-02-16 (X)
OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.
Unresolvable The public record could not settle it.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2019-02-16 for OpenAI's GPT-2, released that week. The claim has two parts: (a) GPT-2 achieved state-of-the-art results on some question-answering or reading-comprehension benchmarks; (b) it was well short of human performance on story understanding. True: both hold. That would be shown by OpenAI's GPT-2 paper or blog reporting SOTA on at least one QA-type test (e.g., a zero-shot reading-comprehension or commonsense benchmark), together with evidence such as Marcus's tests, benchmark scores or contemporaneous evaluations that GPT-2 fell clearly below humans on story or narrative comprehension. Mostly true: both broadly hold, but the SOTA results were on language-modelling or related tasks rather than strictly QA, or held only in a narrow setting such as zero-shot. Mixed: one part clearly holds and the other does not. Mostly false: GPT-2 was not SOTA on any QA test, and the human-gap point rests only on anecdote. False: both parts are contradicted.
Evidence
- Better language models and their implications OpenAI · 14 February 2019
“On other language tasks like question answering, reading comprehension, summarization, and translation, we are able to get surprising results without any fine-tuning of our models, simply by prompting the trained model in the right way (see below for examples of how we do this), though we do still fall short of state-of-the-art for specialized systems.”
- Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“GPT-2 achieves new state of the art results of 93.3% on common nouns and 89.1% on named entities.”
- Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“On reading comprehension the performance of GPT-2 is competitive with supervised baselines in a zero-shot setting.”
- Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“As seen in Figure 2 performance steadily improves as model size is increased and closes the majority of the gap to human performance on this test.”
Reasoning
E2 reports state-of-the-art results on common nouns and named entities but does not show they are on a question-answering test, and E1 says GPT-2 fell short of state of the art on question answering. E4 indicates some remaining gap to human performance on an unnamed test but does not show it concerns story understanding or that the gap was large, so neither part of the claim is clearly settled.
Cite this claim
Gary Marcus, “OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.” (said 16 February 2019). Verdict: unresolvable. Reliability API, claim c_d9cf9266e3b9, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_d9cf9266e3b9
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.