Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“but the systems don’t have the knowledge graph to do that.”
Gary Marcus, speaking in conversation 2025 Dr. Ben Goertzel's Conversation with Dr. Gary Marcus at the World AI Summit 2025 (SingularityNET)
Current AI systems lack the knowledge graph needed to handle easy facts like those on the first page of Wikipedia.
Mixed Partly true and partly false in roughly equal measure.
The second judge said mostly false; the verdict shown is the first judge's.
What would count as true or false
Judged as of 2025, about the leading general-purpose AI systems of that time (frontier LLM chatbots). True: evidence shows these systems lacked an explicit, structured knowledge graph (or equivalent store of facts) and that this caused regular failures on easy, well-documented facts like those on the first page of a Wikipedia article. Mostly true: the systems lacked an explicit knowledge graph and made such errors noticeably often, but with small differences in detail, such as being reliable on most simple facts and failing mainly on some types. Mixed: the systems had no explicit knowledge graph, but by 2025 they usually answered easy Wikipedia-level facts correctly (often with retrieval or search), so the claimed practical shortfall only partly holds. Mostly false: the claim stretches a real but narrow weakness (occasional factual hallucination) into a general inability, or ignores widely deployed retrieval or knowledge-graph integration that handled such facts. False: benchmarks or evaluations from 2025 show leading systems answering easy, first-page Wikipedia facts with near-perfect reliability, or show that they did use knowledge graphs for this.
Evidence
- 'Flawed from the start': Accuracy of OpenAI's newest GPT questioned ABC News (Australia) · 20 March 2025
“Then OpenAI ran GPT-4.5 through the quiz, finding it hallucinated 37 per cent of the time.”
- 'Flawed from the start': Accuracy of OpenAI's newest GPT questioned ABC News (Australia) · 20 March 2025
“The next most recent GPT model, GPT-4o, hallucinated 62 per cent of the time.”
- SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge arXiv (Google DeepMind authors) · 4 January 2025
“On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5.”
- 'Flawed from the start': Accuracy of OpenAI's newest GPT questioned ABC News (Australia) · 20 March 2025
“And if you do have a simple query, SimpleQA's error rate shows you're better off using a search engine.”
- SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge arXiv (authors from Google DeepMind's Gemini Factuality team, per acknowledgments) · 4 January 2025
“On this new benchmark, Gemini 2.5 Pro achieves a state-of-the-art F1-score of 55.6, outperforming other frontier models, including GPT-5.”
- SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge arXiv (authors from Google DeepMind's Gemini Factuality team, per acknowledgments) · 4 January 2025
“Enabling tools on SimpleQA Verified results in near perfect performance, emphasizing that SimpleQA Verified should be employed for measuring parametric factuality only.”
- Introducing GPT-4.5 | OpenAI OpenAI · 27 February 2025
“GPT‑4.5 has access to the latest up-to-date information with search, supports file and image uploads, and can use canvas to work on writing and code.”
- When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs arXiv · 1 May 2026
“Our analysis shows that models perform strongly on topics typically covered in widely available encyclopedic and educational sources, such as scientific and historical knowledge, while being less reliable on long-tail and everyday culture-specific topics.”
- Why DO large language models hallucinate? Gary Marcus (Substack newsletter) · 5 May 2025 Own record
“Three paragraphs and a box, and Google AI Overviews couldn’t get it right.”
- Why DO large language models hallucinate? Gary Marcus (Substack newsletter) · 5 May 2025 Own record
“They don’t have a database of records like any proper business would (which would be a strong basis to solve the problem)”
- SimpleQA AI Wiki (aiwiki.ai)
“Challenging: Questions were adversarially collected against GPT-4o and GPT-3.5 responses.”
- SimpleQA AI Wiki (aiwiki.ai)
“As tool-augmented scores diverge sharply from closed-book scores, evaluation suites are increasingly reporting both”
Reasoning
GPT-4.5 and GPT-4o had high hallucination rates on SimpleQA (E1, E2), and Gemini 2.5 Pro scored an F1 of 55.6 without tools (E3). However, SimpleQA's questions were collected adversarially against GPT-4o and GPT-3.5 responses, so the benchmark is not a direct test of easy first-page facts (E11). On SimpleQA Verified, enabling tools such as search produced near-perfect performance (E6), tool-augmented scores diverge sharply from closed-book scores (E12), and GPT-4.5 shipped with search (E7). A later study found models perform strongly on widely documented encyclopedic topics and are weaker on long-tail ones (E8). Together, this means the claimed practical failure on easy Wikipedia-level facts only partly holds, which matches the "mixed" criterion.
Cite this claim
Gary Marcus, “Current AI systems lack the knowledge graph needed to handle easy facts like those on the first page of Wikipedia.” (said 2025). Verdict: mixed. Reliability API, claim c_b78ae3ab6443, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_b78ae3ab6443
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.