OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2019-02-16 for OpenAI's GPT-2, released that week. The claim has two parts: (a) GPT-2 achieved state-of-the-art results on some question-answering or reading-comprehension benchmarks; (b) it was well short of human performance on story understanding. True: both hold. That would be shown by OpenAI's GPT-2 paper or blog reporting SOTA on at least one QA-type test (e.g., a zero-shot reading-comprehension or commonsense benchmark), together with evidence such as Marcus's tests, benchmark scores or contemporaneous evaluations that GPT-2 fell clearly below humans on story or narrative comprehension. Mostly true: both broadly hold, but the SOTA results were on language-modelling or related tasks rather than strictly QA, or held only in a narrow setting such as zero-shot. Mixed: one part clearly holds and the other does not. Mostly false: GPT-2 was not SOTA on any QA test, and the human-gap point rests only on anecdote. False: both parts are contradicted.
Set before the research began.
Evidence (4)
-
E1 Better language models and their implications OpenAI · 14 February 2019
“On other language tasks like question answering, reading comprehension, summarization, and translation, we are able to get surprising results without any fine-tuning of our models, simply by prompting the trained model in the right way (see below for examples of how we do this), though we do still fall short of state-of-the-art for specialized systems.”
-
E2 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“GPT-2 achieves new state of the art results of 93.3% on common nouns and 89.1% on named entities.”
-
E3 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“On reading comprehension the performance of GPT-2 is competitive with supervised baselines in a zero-shot setting.”
-
E4 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)
“As seen in Figure 2 performance steadily improves as model size is increased and closes the majority of the gap to human performance on this test.”
Reasoning
E2 reports state-of-the-art results on common nouns and named entities but does not show they are on a question-answering test, and E1 says GPT-2 fell short of state of the art on question answering. E4 indicates some remaining gap to human performance on an unnamed test but does not show it concerns story understanding or that the gap was large, so neither part of the claim is clearly settled.
Cite this claim
Gary Marcus, “OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.” (said 16 February 2019). Verdict: unresolvable. Reliability API, claim c_d9cf9266e3b9, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_d9cf9266e3b9
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mostly true
“Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”
Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.
-
Mostly true
“Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”
-
Mostly true
“Despite all the hype, agents didn’t turn out to be reliable.”
-
Mixed
“but the systems don’t have the knowledge graph to do that.”
-
Resolved true
“Hasn’t worked so far. That’s for sure.”
Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.
Verdicts are made automatically; how verdicts are made.