ClaimsGary Marcus

“The lack of reliability will continue to haunt generative AI.”

Gary Marcus, Substack author 1 January 2025 25 AI Predictions for 2025, from Marcus on AI (Marcus on AI)

Prediction AI capabilities

Lack of reliability will continue to afflict generative AI in 2025.

Resolved true The record shows it happened or was so.

Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?

What would count as true or false

The claim is that unreliability would remain a significant, widely recognised problem for generative AI during 2025. Unreliability here means hallucinations, factual errors, and inconsistent or unpredictable outputs. True: through 2025, credible sources document ongoing reliability problems in leading generative AI systems. Examples include hallucination benchmarks showing non-trivial error rates, prominent publicised failures such as fabricated legal citations or erroneous outputs, and industry or academic reports naming reliability as a key barrier. Mostly true: such problems persist but are widely reported as substantially reduced. Mixed: the evidence is split, with major systems widely treated as reliable for most uses but notable failures still occurring. Mostly false: credible evidence shows reliability largely stopped being a significant concern for leading systems, with only isolated incidents. False: by 2025 leading generative AI is broadly documented as reliable, and reliability is no longer a recognised problem.

Set before the research began.

Evidence (4)

  1. E1 AI chatbots unable to accurately summarise news, BBC finds BBC News · 11 February 2025

    “It found 51% of all AI answers to questions about the news were judged to have significant issues of some form.”
  2. E2 AI chatbots fail at accurate news, major study reveals Deutsche Welle (DW) · 21 October 2025

    “the study found that almost half of all answers had at least one significant issue, while 31% contained serious sourcing problems and 20% contained major factual errors.”
  3. E3 How Strong Is AI When Hallucinations Haunt? G2 · 2 April 2025

    “On average, reviews mentioning hallucinations dropped from 35% in March 2025 to 8.3% in October 2025, according to G2 review data.”
  4. E4 AI Hallucinations May Soon Be History Inside Higher Ed · 1 October 2025

    “While more advanced models may reduce the frequency of obvious factual mistakes, the issue persists in more subtle forms.”

Reasoning

Independent studies from 2025 document ongoing reliability problems in leading chatbots. The BBC found that 51% of AI answers about the news had significant issues (E1). A major study reported by DW in October 2025 found that almost half of answers had at least one significant issue, and 20% contained major factual errors (E2). G2 review data shows mentions of hallucinations falling over 2025 (E3), and Inside Higher Ed says the problem persists in subtler forms (E4). Taken together, unreliability remained a widely recognised problem through 2025, which meets the criteria for true.

Cite this claim

Gary Marcus, “Lack of reliability will continue to afflict generative AI in 2025.” (said 1 January 2025). Verdict: resolved true. Reliability API, claim c_4eeff44d9f40, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_4eeff44d9f40

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Gary Marcus, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More prediction claims by Gary Marcus

  1. Mostly true Prediction 27 July 2025
    “Some of its advice will be head-scratchingly bad.”

    Some of GPT-5's advice will be very bad.

  2. Mostly true Prediction 27 July 2025
    “GPT-5 will surely be better, a lot better than GPT-4.”

    GPT-5 will be considerably better than GPT-4.

  3. Resolved true Prediction 27 July 2025
    “GPT-5 really is about to drop, no foolin’.”

    OpenAI will release GPT-5 imminently.

  4. Resolved true Prediction 27 July 2025
    “It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”

    GPT-5 will still make a significant number of obvious errors that are hard to predict.

  5. Mostly true Prediction 1 January 2025
    “Neurosymbolic AI will become much more prominent.”

    Neurosymbolic AI will become much more prominent in 2025.

All of them

Verdicts are made automatically; how verdicts are made.