Current LLMs can do none of understanding the physical world, persistent memory, reasoning and planning, or only in a very primitive way.

Judge as of 2024-03-08 against the state of large language models at that time. The four capabilities are understanding the physical world, persistent memory, reasoning and planning. True: benchmarks, evaluations and expert assessments from around that date show that the leading LLMs (for example GPT-4, Claude 3, Gemini) either lacked each of the four capabilities or showed only rudimentary, unreliable versions of them, and that they did not really understand the physical world. Mostly true: this holds for most of the four, but for one capability (for example, some forms of reasoning) the evidence shows clearly more than primitive ability. Mixed: about half of the capabilities show substantial, non-primitive performance. Mostly false: most of the capabilities were shown at a substantial level, and only one or two were primitive or absent. False: the evidence shows that LLMs then performed all four capabilities at a clearly non-primitive level. Contested expert opinion alone does not settle the claim; weigh documented evaluation results.

Set before the research began.

1. E1 On the Planning Abilities of Large Language Models : A Critical Investigation arXiv (Valmeekam, Marquez, Sreedharan, Kambhampati)

   “Our findings reveal that LLMs’ ability to generate executable plans autonomously is rather limited, with the best model (GPT-4) having an average success rate of \(\sim\)12% across the domains.”

2. E2 PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change NeurIPS Datasets and Benchmarks track (proceedings.neurips.cc)

   “Overall, the performance of these LLMs on our benchmark shows that, as of right now, LLMs are pretty ineffective in reasoning about actions and change.”

3. E3 GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models IJCAI-24 (International Joint Conference on Artificial Intelligence) · 2024

   “At the same time, all models perform below or at the chance level of 50% in the Intuitive Physics tests, while human subjects are on average 80% correct.”

4. E4 GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models IJCAI-24 (International Joint Conference on Artificial Intelligence) · 2024

   “In comparison, GPT-4 achieves an accuracy of 75.0% on this task.”

Independent evaluations from around the claim date show limited planning: GPT-4 produced executable plans in only about 12% of autonomous instances (E1), and PlanBench found LLMs 'pretty ineffective in reasoning about actions and change' (E2). On GRASP, multimodal LLMs scored at or below chance on intuitive physics, against about 80% for humans (E3), which supports the claim about the physical world. The evidence does not address persistent memory, and E4's 75% GPT-4 figure has no stated task, so the claim is supported for most but not all of the four capabilities.