Marcus said he did not think machines would reach world-class scientific reasoning within the next five years (by 2024).

The horizon is five years from 2019-09-20. True: by 2024-09-20 no AI system had shown scientific reasoning at a world-class level, meaning independently recognised as matching leading human scientists at forming hypotheses, designing experiments and reaching original scientific conclusions across science in general. Narrow tools such as protein-structure prediction or benchmark scores do not count as world-class scientific reasoning. Mostly true: no such system existed, though some systems matched expert level on narrow, bounded scientific tasks or exams. Mixed: credible, independently reviewed evidence existed by the deadline of AI producing original, expert-level scientific reasoning in at least one field, but nothing general. Mostly false: by the deadline, several independent sources treated AI systems as already reasoning at a world-class scientific level across multiple fields. False: by 2024-09-20 an AI system was widely and independently accepted as reasoning at the level of top human scientists in general. A capability that appeared after 2024-09-20 does not make the claim false.

Set before the research began.

1. E1 Sakana AI · 13 August 2024

   “But while the current iteration of The AI Scientist demonstrates a strong ability to innovate on top of well-established ideas, such as Diffusion Modeling or Transformers, it is still an open question whether such systems can ultimately propose genuinely paradigm-shifting ideas.”

2. E2 Newly Created 'AI Scientist' Is About to Start Churning Out Research ScienceAlert (republished from The Conversation, by Karin Verspoor, RMIT University) · 24 August 2024

   “Even the system's own review of its outputs judges the papers weak at best.”

3. E3 How Well Does AI Conduct Research? Meet the Researcher Trying to Find Out University of Arizona College of Information Science · 16 January 2025

   “The AI models, on the other hand, completed the moderate and most difficult tasks between just 10%-20% of the time.”

4. E4 Large Language Models for Automated Open-domain Scientific Hypotheses Discovery arXiv (preprint, authors not named on the fetched page) · 31 January 2026

   “During expert evaluation, many hypotheses generated by MOOSE are recognized by social science researchers to be both novel (“not existing in the literature”) and valid (“reflecting reality”).”

E1 is Sakana's own August 2024 announcement of The AI Scientist. In it the company says the system builds on well-established ideas, and that whether such systems can propose paradigm-shifting ideas is still an open question. In E2, an outside academic writing in August 2024 reports that even the system's own automated review judged its papers "weak at best". E3, from January 2025, reports that AI models completed the moderate and most difficult research tasks only 10–20% of the time, which points to no world-class research ability even after the deadline. E4 reports novel and valid hypotheses only in the narrow area of social-science hypothesis generation, which the criteria do not count as general world-class reasoning. None of the evidence shows such a system existing by 2024-09-20.