Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“I don’t think it’s going to happen in the next five years.”
Gary Marcus, interviewee 20 September 2019 Gary Marcus on Why AI Needs a Reboot (Psychology Today)
Marcus said he did not think machines would reach world-class scientific reasoning within the next five years (by 2024).
Resolved true The record shows it happened or was so.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
The horizon is five years from 2019-09-20. True: by 2024-09-20 no AI system had shown scientific reasoning at a world-class level, meaning independently recognised as matching leading human scientists at forming hypotheses, designing experiments and reaching original scientific conclusions across science in general. Narrow tools such as protein-structure prediction or benchmark scores do not count as world-class scientific reasoning. Mostly true: no such system existed, though some systems matched expert level on narrow, bounded scientific tasks or exams. Mixed: credible, independently reviewed evidence existed by the deadline of AI producing original, expert-level scientific reasoning in at least one field, but nothing general. Mostly false: by the deadline, several independent sources treated AI systems as already reasoning at a world-class scientific level across multiple fields. False: by 2024-09-20 an AI system was widely and independently accepted as reasoning at the level of top human scientists in general. A capability that appeared after 2024-09-20 does not make the claim false.
Evidence
- Sakana AI · 13 August 2024
“But while the current iteration of The AI Scientist demonstrates a strong ability to innovate on top of well-established ideas, such as Diffusion Modeling or Transformers, it is still an open question whether such systems can ultimately propose genuinely paradigm-shifting ideas.”
- Newly Created 'AI Scientist' Is About to Start Churning Out Research ScienceAlert (republished from The Conversation, by Karin Verspoor, RMIT University) · 24 August 2024
“Even the system's own review of its outputs judges the papers weak at best.”
- How Well Does AI Conduct Research? Meet the Researcher Trying to Find Out University of Arizona College of Information Science · 16 January 2025
“The AI models, on the other hand, completed the moderate and most difficult tasks between just 10%-20% of the time.”
- Large Language Models for Automated Open-domain Scientific Hypotheses Discovery arXiv (preprint, authors not named on the fetched page) · 31 January 2026
“During expert evaluation, many hypotheses generated by MOOSE are recognized by social science researchers to be both novel (“not existing in the literature”) and valid (“reflecting reality”).”
Reasoning
E1 is Sakana's own August 2024 announcement of The AI Scientist. In it the company says the system builds on well-established ideas, and that whether such systems can propose paradigm-shifting ideas is still an open question. In E2, an outside academic writing in August 2024 reports that even the system's own automated review judged its papers "weak at best". E3, from January 2025, reports that AI models completed the moderate and most difficult research tasks only 10–20% of the time, which points to no world-class research ability even after the deadline. E4 reports novel and valid hypotheses only in the narrow area of social-science hypothesis generation, which the criteria do not count as general world-class reasoning. None of the evidence shows such a system existing by 2024-09-20.
Cite this claim
Gary Marcus, “Marcus said he did not think machines would reach world-class scientific reasoning within the next five years (by 2024).” (said 20 September 2019). Verdict: resolved true. Reliability API, claim c_af94cc2fd084, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_af94cc2fd084
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.