Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 70 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
60 42 78
All types combined: 74 (likely 68 to 80, high confidence), from 79 claims: 71 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Eliezer Yudkowsky's claims
46 claims
“Transformers are the main thing like that.”
Eliezer Yudkowsky, interviewee 30 March 2023 Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript)
Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2023-03-30. True: ML literature and expert accounts treat the Transformer architecture as the main architectural innovation that gave a large, non-linear jump in performance over earlier approaches such as RNNs/LSTMs. Mostly true: Transformers are widely recognised as the main such change, though other factors (scale, compute, data) are often credited as equally or more important. Mixed: experts are split, with Transformers seen as one of several comparable innovations. Mostly false: Transformers improved results, but the gains are mainly put down to scale rather than to the architecture, so 'main thing' overstates their role. False: the evidence shows Transformers gave no notable jump over earlier architectures.
Evidence
- Attention is All You Need Google Research (NIPS 2017 paper by Vaswani et al.) · 2017
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.”
- NeurIPS Poster Algorithmic progress in language models NeurIPS 2024 · 26 September 2024
“Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period.”
- "The History of Deep Learning" Rework.com
“The design dropped recurrence, which made it easier to train in parallel, and it scaled into today's large language models; see the Transformer architecture for how it works.”
Reasoning
E1, the original Transformer paper, presents an architecture that drops recurrence and reports results superior in quality to earlier models, with faster, more parallel training. E3, a secondary history, describes the Transformer as the design that scaled into today's large language models. E2 names the Transformer as a notable new architecture but finds that increased compute contributed even more to performance gains. E2 dates from 2024, and its quote does not state the period it analyses. Together the sources treat the Transformer as a notable architectural advance while crediting compute as contributing even more, which fits "mostly true".
Cite this claim
Eliezer Yudkowsky, “Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_283307975060, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_283307975060
Claims without a verdict
Is a verdict here wrong, or do you speak for Eliezer Yudkowsky? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.