Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
55 claims
“LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”
Yann LeCun, Meta chief AI scientist, interviewed 8 March 2024 Transcript for Yann Lecun: Meta AI, Open Source, Limits of LLMs, AGI & the Future of AI (Lex Fridman Podcast)
Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2024-03-08 against the state of large language models at that time. The four capabilities are understanding the physical world, persistent memory, reasoning and planning. True: benchmarks, evaluations and expert assessments from around that date show that the leading LLMs (for example GPT-4, Claude 3, Gemini) either lacked each of the four capabilities or showed only rudimentary, unreliable versions of them, and that they did not really understand the physical world. Mostly true: this holds for most of the four, but for one capability (for example, some forms of reasoning) the evidence shows clearly more than primitive ability. Mixed: about half of the capabilities show substantial, non-primitive performance. Mostly false: most of the capabilities were shown at a substantial level, and only one or two were primitive or absent. False: the evidence shows that LLMs then performed all four capabilities at a clearly non-primitive level. Contested expert opinion alone does not settle the claim; weigh documented evaluation results.
Evidence
- On the Planning Abilities of Large Language Models : A Critical Investigation arXiv (Valmeekam, Marquez, Sreedharan, Kambhampati)
“Our findings reveal that LLMs’ ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of \(\sim\)12% across the domains.”
- PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change NeurIPS Datasets and Benchmarks track (proceedings.neurips.cc)
“Overall, the performance of these LLMs on our benchmark shows that, as of right now, LLMs are pretty ineffective in reasoning about actions and change.”
- GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models IJCAI-24 (International Joint Conference on Artificial Intelligence) · 2024
“At the same time, all models perform below or at the chance level of 50% in the Intuitive Physics tests, while human subjects are on average 80% correct.”
- GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models IJCAI-24 (International Joint Conference on Artificial Intelligence) · 2024
“In comparison, GPT-4 achieves an accuracy of 75.0% on this task.”
Reasoning
Independent evaluations from around the claim date show limited planning: GPT-4 produced executable plans in only about 12% of autonomous instances (E1), and PlanBench found LLMs 'pretty ineffective in reasoning about actions and change' (E2). On GRASP, multimodal LLMs scored at or below chance on intuitive physics, against about 80% for humans (E3), which supports the claim about the physical world. The evidence does not address persistent memory, and E4's 75% GPT-4 figure has no stated task, so the claim is supported for most but not all of the four capabilities.
Cite this claim
Yann LeCun, “Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.” (said 8 March 2024). Verdict: mostly true. Reliability API, claim c_861fee986cde, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_861fee986cde
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.