Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.
Mostly true True in substance, wrong in a detail.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2023-03-30. True: ML literature and expert accounts treat the Transformer architecture as the main architectural innovation that gave a large, non-linear jump in performance over earlier approaches such as RNNs/LSTMs. Mostly true: Transformers are widely recognised as the main such change, though other factors (scale, compute, data) are often credited as equally or more important. Mixed: experts are split, with Transformers seen as one of several comparable innovations. Mostly false: Transformers improved results, but the gains are mainly put down to scale rather than to the architecture, so 'main thing' overstates their role. False: the evidence shows Transformers gave no notable jump over earlier architectures.
Set before the research began.
Evidence (3)
-
E1 Attention is All You Need Google Research (NIPS 2017 paper by Vaswani et al.) · 2017
“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.”
-
E2 NeurIPS Poster Algorithmic progress in language models NeurIPS 2024 · 26 September 2024
“Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period.”
-
E3 "The History of Deep Learning" Rework.com
“The design dropped recurrence, which made it easier to train in parallel, and it scaled into today's large language models; see the Transformer architecture for how it works.”
Reasoning
E1, the original Transformer paper, presents an architecture that drops recurrence and reports results superior in quality to earlier models, with faster, more parallel training. E3, a secondary history, describes the Transformer as the design that scaled into today's large language models. E2 names the Transformer as a notable new architecture but finds that increased compute contributed even more to performance gains. E2 dates from 2024, and its quote does not state the period it analyses. Together the sources treat the Transformer as a notable architectural advance while crediting compute as contributing even more, which fits "mostly true".
Cite this claim
Eliezer Yudkowsky, “Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_283307975060, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_283307975060
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Eliezer Yudkowsky
-
Resolved true
“We learned that that's false, but we did not learn that you can have a thing that can redesign a tree and build an AI much smarter than itself, that builds an AI much smarter than itself, and that…”
-
Mostly true
“I was a passionate noticer of the human brain not being homogenous, and was therefore unpleasantly surprised by transformers scaling as well as they did.”
-
Mostly true
“First of all, you realize in principle that the task of minimizing losses on predicting human text does not stop when you’re as smart as a human, right?”
-
Mostly true
“Because the basic paradigm of machine learning that we are presently operating under is that you can have the loss function, but only for things you can evaluate.”
-
Mostly true
“So 60 years later, we’re making progress on a bunch of that, thankfully not yet improved themselves.”
Verdicts are made automatically; how verdicts are made.