Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“Although GPT-4 will definitely seem smarter than its predecessors, its internal architecture remains problematic.”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4 will definitely seem smarter than its predecessors, but its internal architecture will remain problematic.
Resolved true The record shows it happened or was so.
What would count as true or false
Judged at GPT-4's release (March 2023) and the months right after. The prediction has two parts: (a) GPT-4 would seem smarter than its predecessors; (b) its internal architecture would remain problematic, meaning it would still be a large transformer-based LLM that shows the same kinds of architectural failures (hallucination, unreliable reasoning) rather than a fundamentally different design that removes them. True: GPT-4 clearly outperformed GPT-3/3.5 on benchmarks and in user perception, and it remained a transformer LLM that still hallucinated and reasoned unreliably. Mostly true: both parts hold but one is somewhat weaker than stated, e.g. the gains were modest in places. Mixed: only one of the two parts holds. Mostly false: GPT-4 did not clearly seem smarter, and the architectural point holds only weakly. False: GPT-4 did not seem smarter than its predecessors and adopted an architecture that resolved the problems Marcus named.
Evidence
- GPT-4 Technical Report arXiv (rendered via ar5iv)
“On a suite of traditional NLP benchmarks, GPT-4 outperforms both previous large language models and most state-of-the-art systems (which often have benchmark-specific training or hand-engineering).”
- GPT-4 Technical Report arXiv (rendered via ar5iv)
“GPT-4 is a Transformer-style model [39] pre-trained to predict the next token in a document”
- OpenAI announces GPT-4, says beats 90% of humans on ... CNBC · 14 March 2023
“GPT-4 performed at the 90th percentile on a simulated bar exam, the 93rd percentile on an SAT reading exam, and the 89th percentile on the SAT Math exam, OpenAI claimed.”
- GPT-4 Ups The Ante In The AI Arms Race IEEE Spectrum · 18 March 2023
“In its announcement, OpenAI admitted that GPT-4 suffers from similar problems to its predecessors, in particular its tendency to “hallucinate,” which refers to it confidently stating as fact things that are actually false.”
Reasoning
The GPT-4 Technical Report says GPT-4 outperforms previous large language models on traditional NLP benchmarks (E1). CNBC's release-day coverage reports high exam-percentile results, though it attributes these figures to OpenAI (E3). Together these support part (a), that GPT-4 seemed smarter than its predecessors. On part (b), the report describes GPT-4 as a Transformer-style model pre-trained for next-token prediction (E2), and IEEE Spectrum reports that OpenAI admitted GPT-4 suffers from similar problems to its predecessors, in particular hallucination (E4).
Cite this claim
Gary Marcus, “GPT-4 will definitely seem smarter than its predecessors, but its internal architecture will remain problematic.” (said 25 December 2022). Verdict: resolved true. Reliability API, claim c_5f10f8218780, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_5f10f8218780
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.