GPT-4 may be better at faking term papers, but if it follows the same playbook as its predecessors, it still won't really understand the world, and its flaws will eventually show.

The core of the hedged claim is that GPT-4 would still lack real understanding of the world and that its flaws would eventually become visible. True: after GPT-4's release, credible research and reporting document its systematic flaws. Examples include hallucinations, reasoning failures, and failures on world-model or physical or causal reasoning tests. GPT-4 must also not be broadly accepted as genuinely understanding the world. Mostly true: flaws are widely documented, but there is a substantial expert debate about whether GPT-4 has some form of understanding. Mixed: the evidence and expert opinion are evenly split on whether it has world understanding, though flaws were shown. Mostly false: few significant flaws surfaced and most experts credit it with substantial understanding. False: GPT-4 was broadly shown to understand the world, with no notable systematic flaws. The 'better at faking term papers' part is hedged ('may') and does not decide the verdict.

Set before the research began.

1. E1 GPT-4 OpenAI · 14 March 2023

   “Most importantly, it still is not fully reliable (it “hallucinates” facts and makes reasoning errors).”

2. E2 Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis PubMed Central (peer-reviewed academic study) · 22 May 2024

   “Hallucination rates stood at 39.6% (55/139) for GPT-3.5, 28.6% (34/119) for GPT-4, and 91.4% (95/104) for Bard (P<.001).”

3. E3 GPT-4 is all talk and not much understanding, says this AI expert National Association of Science Writers (NASW)

   “In a 2023 preprint, Mitchell and her colleagues reported that GPT-4, the most advanced language model that OpenAI has released, performs poorly on visual puzzles that ask it to deal with concepts like “same and different” or “above and below”—tests on which humans excel.”

4. E4 The debate over understanding in AI’s large language models PNAS (Mitchell and Krakauer; mirrored by NSF PAR) · 21 March 2023

   “While LLMs exhibit extraordinary formal linguistic competence—the ability to generate grammatically fluent, humanlike language—they still lack the conceptual understanding needed for humanlike functional language abilities—the ability to robustly understand and use language in the real world (53).”

OpenAI's own GPT-4 release page (E1) says the model still 'hallucinates' facts and makes reasoning errors. A 2023 preprint by Mitchell and colleagues, as reported in E3, found GPT-4 performed poorly on visual puzzles about concepts like same/different and above/below, which are tests humans pass easily. Those systematic flaws match what the prediction said would show. However, E4 is a PNAS paper titled 'The debate over understanding in AI's large language models', which indicates that the question of understanding is debated. That points to real expert disagreement over whether GPT-4 has some understanding, rather than a broad consensus that it has none, which under the criteria makes the verdict mostly true. E2, a 2024 study, reports a 28.6% hallucination rate for GPT-4 when generating systematic-review references [E2].