If GPT-4 is architecturally essentially the same as GPT-3, it can be expected to be unable to construct internal models of how the world works and, in consequence, unable to understand things at an abstract level.

Judge against GPT-4 as released (March 2023) and as evaluated in the following months of 2023. True: credible evaluations and research in 2023 widely found that GPT-4 lacked robust internal world models and failed at abstract understanding (e.g. systematic failures on abstraction benchmarks such as ARC, physical/spatial reasoning, planning), and the balance of expert evidence supports both inability claims. Mostly true: GPT-4 showed clear, systematic failures at world modelling and abstraction, but some evidence showed partial or emergent world-model-like competence in some domains, so 'inability' is somewhat overstated. Mixed: the evidence is roughly balanced, with substantial findings both of world-model/abstract competence and of fundamental failures, or one of the two parts (world models, abstraction) is supported and the other is not. Mostly false: the predominant evidence shows GPT-4 could form usable internal models and handle many abstract tasks, with only narrow failures. False: GPT-4 was clearly shown to construct reliable internal world models and understand abstractions at a broadly human level. The claim is hedged ('if so'): its premise is that GPT-4 is a scaled-up LLM, which holds.

Set before the research began.

1. E1 Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks arXiv (authors: Melanie Mitchell, Alessandro B. Palmarini, Arseny Moskvichev, Santa Fe Institute) · 12 September 2023

   “Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.”

2. E2 LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations arXiv (authors: Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, Elias B. Khalil)

   “GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids.”

3. E3 Inherent limitations of GPT-4 regarding spatial information arXiv · 20 November 2023

   “As the map size and the obstacles ratio increase, the precision in finding the shortest path decreases.”

4. E4 GPT-4 Can't Reason (Position Paper) ar5iv (arXiv mirror) · 26 June 2023

   “Based on the results of that analysis, this paper argues that, despite the occasional flashes of analytical brilliance, GPT-4 at present is utterly incapable of reasoning.”

An independent 2023 evaluation concluded that neither version of GPT-4 had developed robust abstraction abilities at humanlike levels [E1]. It solved only 13 of 50 simple ARC tasks when given text encodings [E2]. Its shortest-path precision fell as map size and obstacles increased, which points to weak spatial world modelling [E3], and one position paper argued it could not reason [E4]. The same sources also report partial success (13/50 tasks) and "occasional flashes of analytical brilliance", so calling it a blanket "inability" overstates the case somewhat.