Without external aids, GPT-5 won't beat Meta's Cicero at Diplomacy or drive a car reliably.
Unresolvable The public record could not settle it.
Prediction: Statements about events outside the person's control, checked once they fell due: did they come true?
What would count as true or false
The claim has two parts, both about GPT-5 without external aids (tools or specialised scaffolding): (1) it will not beat Meta's Cicero at Diplomacy; (2) it will not drive a car reliably. True: there is no credible demonstration of GPT-5 alone doing either. Mostly true: neither has been done, apart from borderline demonstrations that use limited aids. Mixed: one part is credibly contradicted and the other holds. Mostly false: one part is clearly contradicted and the other is near-contradicted. False: both are credibly demonstrated by GPT-5 alone.
Set before the research began.
Evidence (11)
-
E1 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026
“and GPT-5.6 Sol achieved 6% in all three attempts.”
-
E2 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026
“Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course.”
-
E3 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler, et al.) · 1 October 2026
“with nothing between it and the car except for openpilot’s low-level actuation (comma.ai, 2026).”
-
E4 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)
“Grok 4.6 and GPT-5.6 Sol failed to pass the first corner.”
-
E5 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)
“The model drives it via three Model Context Protocol tools (Anthropic, 2024): observe, set_motion and stop_now.”
-
E6 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)
“we retrofitted a comma four running a modified version of the openpilot software”
-
E7 DRIVINGBENCH: CAN VISION-LANGUAGE MODELS DRIVE A TOYOTA COROLLA? arXiv (preprint; authors Aditya Ramabadran, Simon Mahns, Tobias Gessler and others)
“Its second attempt, however, finished the entire course and parked in the blue finish zone.”
-
E8 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (DrivingBench authors) · 24 September 2026
“Grok 4.6 in its three attempts achieved 8%, 11%, and 10%; and GPT-5.6 Sol achieved 6% in all three attempts.”
-
E9 DrivingBench: Can Vision-Language Models Drive a Toyota Corolla? arXiv (DrivingBench authors) · 24 September 2026
“Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course.”
-
E10 Report · DrivingBench DrivingBench · 1 January 2026
“Below you can see the 3 MCP tools we give our agents (in their native harness - Codex/Claude Code/Cursor).”
-
E11 Diplomacy and CICERO Meta AI · 1 January 2026
“CICERO achieved more than 2x the average score of its opponents.”
Reasoning
The only driving evidence is a DrivingBench preprint dated September–October 2026 (E1, E4, E8). It tested GPT-5.6 Sol rather than GPT-5, and that model reached 6% of the course and failed to pass the first corner. The test also gave every model MCP tools and openpilot actuation (E3, E5, E6, E10), so it does not measure GPT-5 driving without external aids by the end of 2025. For Diplomacy, the evidence describes only Cicero's own results (E11) and nothing on how GPT-5 did against it. The evidence does not show either part for GPT-5 without external aids, so the claim is unresolvable.
Cite this claim
Gary Marcus, “Without external aids, GPT-5 won't beat Meta's Cicero at Diplomacy or drive a car reliably.” (said 27 July 2025). Verdict: unresolvable. Reliability API, claim c_fd83b618eccb, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_fd83b618eccb
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More prediction claims by Gary Marcus
-
Mostly true
“Some of its advice will be head-scratchingly bad.”
-
Mostly true
“GPT-5 will surely be better, a lot better than GPT-4.”
-
Resolved true
“GPT-5 really is about to drop, no foolin’.”
-
Resolved true
“It will still make a significant number of shake-your-head stupid errors, in ways that are hard to fully predict.”
GPT-5 will still make a significant number of obvious errors that are hard to predict.
-
Mostly true
“Neurosymbolic AI will become much more prominent.”
Verdicts are made automatically; how verdicts are made.