GPT-4 is architecturally essentially the same as GPT-3, according to rumors.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
This is a hedged claim about what rumors said, judged as of 2022-12-25. True: credible rumors or reporting circulating before that date described GPT-4 as essentially the same architecture as GPT-3 (a dense decoder-only transformer, mainly scaled or retrained), and later disclosures (e.g. OpenAI's GPT-4 technical report or credible leaks) do not show a fundamentally different architecture. Mostly true: the rumors said this, but later credible information shows notable but non-fundamental differences, e.g. a mixture-of-experts transformer, which is still a transformer LLM. Mixed: the rumors were split, or later information shows significant architectural changes that make "essentially the same" debatable. Mostly false: hardly any rumors before that date said this, though GPT-4 did turn out to be a transformer LLM. False: rumors at the time described a substantially different architecture and later evidence confirms it was fundamentally different.
Set before the research began.
Evidence (12)
-
E1 ChatGPT is just a taste of a "monster" GPT-4 says Gary Marcus THE DECODER · 26 December 2022
“When it comes to its architecture, Marcus doesn't expect any significant differences and therefore assumes that GPT-4 will have the same weaknesses as GPT-3 or ChatGPT”
-
E2 ChatGPT is just a taste of a "monster" GPT-4 says Gary Marcus THE DECODER · 26 December 2022
“Rumors about GPT-4 have been floating around the web for weeks, and they have two things in common: GPT-4 is supposed to outperform GPT-3 and ChatGPT significantly and be released relatively soon in the spring.”
-
E3 Proprietary models - EcoLogits EcoLogits · 12 October 2026
“The model architecture of GPT-4 was leaked1 and estimated to be a sparse mixture of experts model with a total of 1.8 trillion parameters and 16 experts of 111 billion parameters each.”
-
E4 GPT-4 (2022) LifeArchitect.ai (Alan D. Thompson)
“‘Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.’”
-
E5 GPT-4 (2022) Lifearchitect (Alan D. Thompson) · March 2023
“Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”
-
E6 GPT-4 (2022) Lifearchitect (Alan D. Thompson) · 28 October 2023
“OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model.”
-
E7 GPT-4 (2022) Lifearchitect (Alan D. Thompson) · 12 December 2022
“GPT3: 175b params for $4.6M. GPT4: 100t params for $2.6B. This trend line can't go up much further.”
-
E8 GPT-4 (2022) Lifearchitect (Alan D. Thompson) · 24 August 2021
““From talking to OpenAI, GPT-4 will be about 100 trillion parameters…””
-
E9 Rumor: GPT-4 to be 500x larger, available in a few years OpenAI Community forum (user discussion thread) · 13 September 2021
“Altman said in the interview that contrary to popular belief, GPT-4 will not be any bigger than GPT-3 but will use more compute resources.”
-
E10 GPT-4 architecture, datasets, costs and more leaked THE DECODER · 28 June 2023
“GPT-4 is rumored to be based on eight models, each with 220 billion parameters, which are linked in the Mixture of Experts (MoE) architecture.”
-
E11 GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoE SemiAnalysis · 10 July 2023
“A dense transformer is the model architecture that OpenAI GPT-3, Google PaLM, Meta LLAMA, TII Falcon, MosaicML MPT, etc use.”
-
E12 GPT‑4 - OpenAI OpenAI · 14 March 2023
“GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while less capable than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks.”
Reasoning
The pre-claim rumours in the evidence (E2, E7, E8, E9) concern GPT-4's size, compute, performance and timing, and none states that GPT-4 would share GPT-3's architecture. The criteria turn on what the rumours said about architecture, so the evidence does not settle the claim. Later reports of a mixture-of-experts design (E3, E6, E10) are described as leaked, estimated or rumoured.
Cite this claim
Gary Marcus, “GPT-4 is architecturally essentially the same as GPT-3, according to rumors.” (said 25 December 2022). Verdict: unresolvable. Reliability API, claim c_9f21b3b05a25, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_9f21b3b05a25
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mostly true
“Reviews by Joanna Stern and Marques Brownle of one early prototype were damning;”
Reviews by Joanna Stern and Marques Brownlee of one early humanoid robot prototype were damning.
-
Mostly true
“Many would agree we have reached a point of diminishing returns for scaling; faith in scaling as a route to AGI has dissipated.”
-
Mostly true
“Despite all the hype, agents didn’t turn out to be reliable.”
-
Mixed
“but the systems don’t have the knowledge graph to do that.”
-
Resolved true
“Hasn’t worked so far. That’s for sure.”
Bolting extra components onto an LLM and scaling it up has so far not produced human-level AGI.
Verdicts are made automatically; how verdicts are made.