ClaimsGary Marcus

“Without external aids it won’t be able beat Meta’s Cicero in Diplomacy; it won’t be able to drive a car reliably;”

Gary Marcus, author 27 July 2025 What to Expect When You're Expecting … GPT-5 (Marcus on AI) archived copy

Prediction AI capabilities

Without external aids, GPT-5 won't beat Meta's Cicero at Diplomacy or drive a car reliably.

Unresolvable The public record could not settle it.

Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?

What would count as true or false

The claim has two parts, both about GPT-5 without external aids (tools or specialised scaffolding): (1) it will not beat Meta's Cicero at Diplomacy; (2) it will not drive a car reliably. True: there is no credible demonstration of GPT-5 alone doing either. Mostly true: neither has been done, apart from borderline demonstrations that use limited aids. Mixed: one part is credibly contradicted and the other holds. Mostly false: one part is clearly contradicted and the other is near-contradicted. False: both are credibly demonstrated by GPT-5 alone.

Set before the research began.

Evidence (11)

  1. E1 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026

    “and GPT-5.6 Sol achieved 6% in all three attempts.”
  2. E2 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026

    “Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course.”
  3. E3 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026

    “with nothing between it and the car except for openpilot’s low-level actuation (comma.ai, 2026).”
  4. E4 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)

    “Grok 4.6 and GPT-5.6 Sol failed to pass the first corner.”
  5. E5 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)

    “The model drives it via three Model Context Protocol tools (Anthropic, 2024): observe, set_motion and stop_now.”
  6. E6 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)

    “we retrofitted a comma four running a modified version of the openpilot software”
  7. E7 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)

    “Its second attempt, however, finished the entire course and parked in the blue finish zone.”
  8. E8 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (DrivingBench authors) · 24 September 2026

    “Grok 4.6 in its three attempts achieved 8%, 11%, and 10%; and GPT-5.6 Sol achieved 6% in all three attempts.”
  9. E9 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (DrivingBench authors) · 24 September 2026

    “Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course.”
  10. E10 Report · DrivingBench DrivingBench · 1 January 2026

    “Below you can see the 3 MCP tools we give our agents (in their native harness - Codex/Claude Code/Cursor).”
  11. E11 Diplomacy and CICERO Meta AI · 1 January 2026

    “CICERO achieved more than 2x the average score of its opponents.”

Reasoning

The only driving evidence is a DrivingBench preprint dated September–October 2026 (E1, E4, E8). It tested GPT-5.6 Sol rather than GPT-5, and that model reached 6% of the course and failed to pass the first corner. The test also gave every model MCP tools and openpilot actuation (E3, E5, E6, E10), so it does not measure GPT-5 driving without external aids by the end of 2025. For Diplomacy, the evidence describes only Cicero's own results (E11) and nothing on how GPT-5 did against it. The evidence does not show either part for GPT-5 without external aids, so the claim is unresolvable.

Cite this claim

Gary Marcus, “Without external aids, GPT-5 won't beat Meta's Cicero at Diplomacy or drive a car reliably.” (said 27 July 2025). Verdict: unresolvable. Reliability API, claim c_fd83b618eccb, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_fd83b618eccb

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Gary Marcus, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More prediction claims by Gary Marcus

  1. Mostly true Prediction 27 July 2025
    “Some of its advice will be head-scratchingly bad.”

    Some of GPT-5's advice will be very bad.

  2. Mostly true Prediction 27 July 2025
    “GPT-5 will surely be better, a lot better than GPT-4.”

    GPT-5 will be considerably better than GPT-4.

  3. Resolved true Prediction 27 July 2025
    “GPT-5 really is about to drop, no foolin’.”

    OpenAI will release GPT-5 imminently.

  4. Resolved true Prediction 27 July 2025
    “It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”

    GPT-5 will still make a significant number of obvious errors that are hard to predict.

  5. Mostly true Prediction 1 January 2025
    “Neurosymbolic AI will become much more prominent.”

    Neurosymbolic AI will become much more prominent in 2025.

All of them

Verdicts are made automatically; how verdicts are made.