GPT-4 will definitely seem smarter than its predecessors, but its internal architecture will remain problematic.

Judged at GPT-4's release (March 2023) and the months right after. The prediction has two parts: (a) GPT-4 would seem smarter than its predecessors; (b) its internal architecture would remain problematic, meaning it would still be a large transformer-based LLM that shows the same kinds of architectural failures (hallucination, unreliable reasoning) rather than a fundamentally different design that removes them. True: GPT-4 clearly outperformed GPT-3/3.5 on benchmarks and in user perception, and it remained a transformer LLM that still hallucinated and reasoned unreliably. Mostly true: both parts hold but one is somewhat weaker than stated, e.g. the gains were modest in places. Mixed: only one of the two parts holds. Mostly false: GPT-4 did not clearly seem smarter, and the architectural point holds only weakly. False: GPT-4 did not seem smarter than its predecessors and adopted an architecture that resolved the problems Marcus named.

Set before the research began.

1. E1 GPT-4 Technical Report arXiv (rendered via ar5iv)

   “On a suite of traditional NLP benchmarks, GPT-4 outperforms both previous large language models and most state-of-the-art systems (which often have benchmark-specific training or hand-engineering).”

2. E2 GPT-4 Technical Report arXiv (rendered via ar5iv)

   “GPT-4 is a Transformer-style model [39] pre-trained to predict the next token in a document”

3. E3 OpenAI announces GPT-4, says beats 90% of humans on ... CNBC · 14 March 2023

   “GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed.”

4. E4 GPT-4 Ups The Ante In The AI Arms Race IEEE Spectrum · 18 March 2023

   “In its announcement, OpenAI admitted that GPT-4 suffers from similar problems to its predecessors, in particular its tendency to “hallucinate,” which refers to it confidently stating as fact things that are actually false.”

The GPT-4 Technical Report says GPT-4 outperforms previous large language models on traditional NLP benchmarks (E1). CNBC's release-day coverage reports high exam-percentile results, though it attributes these figures to OpenAI (E3). Together these support part (a), that GPT-4 seemed smarter than its predecessors. On part (b), the report describes GPT-4 as a Transformer-style model pre-trained for next-token prediction (E2), and IEEE Spectrum reports that OpenAI admitted GPT-4 suffers from similar problems to its predecessors, in particular hallucination (E4).