All the big AI players are converging on systems roughly equivalent to GPT-4, all flawed in essentially the same ways.
Unresolvable The public record could not settle it.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judged as of 31 March 2024. True if the leading frontier models from the major developers then (e.g. OpenAI GPT-4, Anthropic Claude 3, Google Gemini, Meta Llama, Mistral) scored roughly level with GPT-4 on major benchmarks and leaderboards (within a few points, none clearly far ahead), and all showed the same main failure modes (hallucination, reasoning errors, unreliability). Mostly true if most big players were at about GPT-4 level with one modest exception, or the shared flaws differ in degree. Mixed if several major players were well below or clearly above GPT-4, or the flaws differ substantially. Mostly false if the frontier was clearly spreading out rather than converging, with only a narrow cluster near GPT-4. False if one or more major models were clearly and widely recognised as far beyond GPT-4, or the systems' flaws were clearly different.
Set before the research began.
Evidence (4)
-
E1 Claude takes the top spot in AI chatbot ranking — finally knocking GPT-4 down to second place Tom's Guide · 27 March 2024
“| Rank | Model | Elo | Votes | |---|---|---|---| | 1 | Claude-3 Opus | 1253 | 33250 | | 1 | GPT-4-1106-Preview | 1251 | 54141 | | 1 | GPT-4-0125-preview | 1248 | 34825 | | 4 | Gemini Pro | 1203 | 12476 |”
-
E2 GPT-4 loses its position as "best" LLM to Claude-3 in LMSYS benchmark TechSpot · 27 March 2024
“However, OpenAI’s LLM was nudged off the top spot yesterday when Anthropic’s Claude 3 Opus beat GPT-4 by a slim margin, 1253 to 1251.”
-
E3 Claude 3 Opus, Sonnet & Haiku: Prices and Benchmark Results Deepnoodle Atlas · 4 March 2024
“Anthropic's model card reports Opus scoring 86.8% on MMLU (5-shot) against GPT-4's 86.4%, 50.4% on the GPQA Diamond set against 35.7%, 95.0% on GSM8K grade-school math against 92.0%, and 84.9% on HumanEval coding problems against 67.0%.”
-
E4 LLM Benchmarks March 2024 TIMETOACT GROUP
“The model itself doesn't perform as well in our benchmarks as it's described in their PR. It beats neither GPT-3.5 nor Gemini Pro.”
Reasoning
The evidence shows Claude 3 Opus and GPT-4 close together on one chatbot ranking, with Gemini Pro about 50 Elo points lower [E1][E2], and vendor-reported benchmark gaps of different sizes [E3]. It does not cover Meta Llama or Mistral, does not name the model in E4, and does not address whether the systems share the same failure modes, so it cannot settle the claim under the criteria.
Cite this claim
Gary Marcus, “All the big AI players are converging on systems roughly equivalent to GPT-4, all flawed in essentially the same ways.” (said 31 March 2024). Verdict: unresolvable. Reliability API, claim c_ccede3bd6e4d, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_ccede3bd6e4d
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Gary Marcus
-
Mixed
“companies like Anthropic are ending their all you can eat buffet, which will make many customers think twice.”
-
Mixed
“But everybody is building essentially the same technical solution with essentially the same data, so there is no moat.”
-
Mostly true
“Still more reports have coming out questioning the RoI for corporate customers.”
-
Mostly true
“The company has not returned any profits, and is not expecting to do so until at least 2029, but has a burn rate in the billions, and still lacks a clear technical moat.”
-
Resolved true
“There is very little moat.”
As of end 2024, AI model makers have very little moat, with many similar models and a price war.
Verdicts are made automatically; how verdicts are made.