ClaimsGary Marcus

“@OpenAI is SoTA on some QA tests, but no match for humans on story understanding.”

Gary Marcus, writing on X 16 February 2019 Post by @GaryMarcus on X, 2019-02-16 (X)

Factual AI capabilities

OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.

Unresolvable The public record could not settle it.

Factual: Statements about the present or the past, checked against the record: did they hold up?

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2019-02-16 for OpenAI's GPT-2, released that week. The claim has two parts: (a) GPT-2 achieved state-of-the-art results on some question-answering or reading-comprehension benchmarks; (b) it was well short of human performance on story understanding. True: both hold. That would be shown by OpenAI's GPT-2 paper or blog reporting SOTA on at least one QA-type test (e.g., a zero-shot reading-comprehension or commonsense benchmark), together with evidence such as Marcus's tests, benchmark scores or contemporaneous evaluations that GPT-2 fell clearly below humans on story or narrative comprehension. Mostly true: both broadly hold, but the SOTA results were on language-modelling or related tasks rather than strictly QA, or held only in a narrow setting such as zero-shot. Mixed: one part clearly holds and the other does not. Mostly false: GPT-2 was not SOTA on any QA test, and the human-gap point rests only on anecdote. False: both parts are contradicted.

Set before the research began.

Evidence (4)

  1. E1 Better language models and their implications OpenAI · 14 February 2019

    “On other language tasks like question answering, reading comprehension, summarization, and translation, we are able to get surprising results without any fine-tuning of our models, simply by prompting the trained model in the right way (see below for examples of how we do this), though we do still fall short of state-of-the-art for specialized systems.”
  2. E2 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)

    “GPT-2 achieves new state of the art results of 93.3% on common nouns and 89.1% on named entities.”
  3. E3 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)

    “On reading comprehension the performance of GPT-2 is competitive with supervised baselines in a zero-shot setting.”
  4. E4 Language Models are Unsupervised Multitask Learners OpenAI (Radford, Wu, Child, Luan, Amodei, Sutskever)

    “As seen in Figure 2 performance steadily improves as model size is increased and closes the majority of the gap to human performance on this test.”

Reasoning

E2 reports state-of-the-art results on common nouns and named entities but does not show they are on a question-answering test, and E1 says GPT-2 fell short of state of the art on question answering. E4 indicates some remaining gap to human performance on an unnamed test but does not show it concerns story understanding or that the gap was large, so neither part of the claim is clearly settled.

Cite this claim

Gary Marcus, “OpenAI's new system is state of the art on some question-answering tests but is no match for humans on story understanding.” (said 16 February 2019). Verdict: unresolvable. Reliability API, claim c_d9cf9266e3b9, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_d9cf9266e3b9

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Gary Marcus, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Gary Marcus

  1. Mostly true Factual January 2026
    “Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”

    Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.

  2. Mostly true Factual January 2026
    “Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”

    Many would agree the AI field has reached a point of diminishing returns from scaling, and faith in scaling as a route to AGI has dissipated.

  3. Mostly true Factual January 2026
    “Despite all the hype, agents didn’t turn out to be reliable.”

    AI agents did not turn out to be reliable in 2025.

  4. Mixed Factual 2025
    “but the systems don’t have the knowledge graph to do that.”

    Current AI systems lack the knowledge graph needed to handle easy facts like those on the first page of Wikipedia.

  5. Resolved true Factual 2025
    “Hasn’t worked so far. That’s for sure.”

    Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.

All of them

Verdicts are made automatically; how verdicts are made.