ClaimsGary Marcus

“People building things like virtual assistants and agents will find that they cannot reliably enough map user language onto user intentions.”

Gary Marcus, author 27 July 2025 What to Expect When You're Expecting … GPT-5 (Marcus on AI) archived copy

Prediction AI capabilities

Developers building virtual assistants and agents on GPT-5 will find they cannot map user language onto user intentions reliably enough.

Unresolvable The public record could not settle it.

Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?

What would count as true or false

Judge over about a year after GPT-5's August 2025 release (to August 2026). True: credible evidence (developer reports, evaluations of GPT-5-based agents, benchmarks of agentic task success, industry surveys) shows that assistants and agents built on GPT-5 commonly misread user intent often enough that developers report it as a significant barrier to reliable deployment. Mostly true: intent-mapping failures are widely documented as a significant problem, but some substantial deployments report acceptable reliability. Mixed: the evidence is roughly split between reports of reliable intent understanding and reports of significant failures. Mostly false: most developers and evaluations find GPT-5 maps user language to intent reliably enough for production, with failures only in edge cases. False: GPT-5-based assistants and agents are broadly found to interpret user intentions reliably, and misinterpretation is not a notable barrier. Failures caused mainly by other factors (tool errors, cost, latency) do not count towards the claim.

Set before the research began.

Evidence (8)

  1. E1 Agentic AI in Industry: Adoption Level and Deployment Barriers arXiv · 31 January 2026

    “Greater task ownership also amplifies friction already visible in LLM-based tools: systems may omit aspects of developer intent, vary across equivalent prompts, hallucinate, or lose relevant information as context grows [13, 7].”
  2. E2 Agentic AI in Industry: Adoption Level and Deployment Barriers arXiv · 31 January 2026

    “Human verification was therefore used to check both technical output and semantic intent.”
  3. E3 Agentic AI in Industry: Adoption Level and Deployment Barriers arXiv · 31 January 2026

    “Across the cases, four previously identified barriers recurred: context management, performance on proprietary content, non-determinism and qualification, and data confidentiality.”
  4. E4 Agentic AI in Industry: Adoption Level and Deployment Barriers arXiv · 31 January 2026

    “Narrow IDE- or CLI-based agentic processes remained ad hoc, requiring context and intent to be re-established for each use.”
  5. E5 Zendesk AI + GPT-5: Setting the pace for the next generation of support Zendesk · 7 August 2025

    “GPT-5 clarified vague customer input more effectively, enabling better routing and increasing coverage of automated flows in over 65% of conversations.”
  6. E6 Zendesk AI + GPT-5: Setting the pace for the next generation of support Zendesk · 7 August 2025

    “GPT-5 operates in conjunction with our intent classification and reasoning pipeline.”
  7. E7 Zendesk reports 30% faster response, 95% reliability after GPT-5 integration VentureBeat · 8 September 2025

    “The company has found that, for most customers, agents powered by GPT-5 can solve more than 50% of tickets — and in some cases, even 80 to 90% of them.”
  8. E8 Learning #4. AI agents: Promising, but not yet mainstream – 2025 Stack Overflow Developer Survey: A TL;DR for Leaders Stack Overflow · 26 October 2026

    “87% of survey respondents are concerned about the accuracy of AI agents.”

Reasoning

The industry study (E1–E4) covers agentic AI in general, not GPT-5 specifically. It notes that systems may leave out parts of developer intent and that intent had to be re-established or checked by people. Even so, misreading intent is not among the four barriers that recurred across its cases (E3). Zendesk reports that GPT-5 handled vague customer input better and resolved many tickets (E5, E7), but these are claims about one deployment, and intent there is also handled by a separate classification pipeline (E6). The survey (E8) shows only general concern about agent accuracy. None of this shows how often GPT-5-based assistants and agents broadly misread user intent, so the criteria cannot be settled either way.

Cite this claim

Gary Marcus, “Developers building virtual assistants and agents on GPT-5 will find they cannot map user language onto user intentions reliably enough.” (said 27 July 2025). Verdict: unresolvable. Reliability API, claim c_f0642744cc4a, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_f0642744cc4a

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Gary Marcus, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More prediction claims by Gary Marcus

  1. Mostly true Prediction 27 July 2025
    “Some of its advice will be head-scratchingly bad.”

    Some of GPT-5's advice will be very bad.

  2. Mostly true Prediction 27 July 2025
    “GPT-5 will surely be better, a lot better than GPT-4.”

    GPT-5 will be considerably better than GPT-4.

  3. Resolved true Prediction 27 July 2025
    “GPT-5 really is about to drop, no foolin’.”

    OpenAI will release GPT-5 imminently.

  4. Resolved true Prediction 27 July 2025
    “It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”

    GPT-5 will still make a significant number of obvious errors that are hard to predict.

  5. Mostly true Prediction 1 January 2025
    “Neurosymbolic AI will become much more prominent.”

    Neurosymbolic AI will become much more prominent in 2025.

All of them

Verdicts are made automatically; how verdicts are made.