ClaimsEliezer Yudkowsky

“Transformers are the main thing like that.”

Eliezer Yudkowsky, interviewee 30 March 2023 Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript) archived copy

Factual AI capabilities

Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.

Mostly true True in substance, wrong in a detail.

Factual: Statements about the present or the past, checked against the record: did they hold up?

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2023-03-30. True: ML literature and expert accounts treat the Transformer architecture as the main architectural innovation that gave a large, non-linear jump in performance over earlier approaches such as RNNs/LSTMs. Mostly true: Transformers are widely recognised as the main such change, though other factors (scale, compute, data) are often credited as equally or more important. Mixed: experts are split, with Transformers seen as one of several comparable innovations. Mostly false: Transformers improved results, but the gains are mainly put down to scale rather than to the architecture, so 'main thing' overstates their role. False: the evidence shows Transformers gave no notable jump over earlier architectures.

Set before the research began.

Evidence (3)

  1. E1 Attention is All You Need Google Research (NIPS 2017 paper by Vaswani et al.) · 2017

    “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.”
  2. E2 NeurIPS Poster Algorithmic progress in language models NeurIPS 2024 · 26 September 2024

    “Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period.”
  3. E3 "The History of Deep Learning" Rework.com

    “The design dropped recurrence, which made it easier to train in parallel, and it scaled into today's large language models; see the Transformer architecture for how it works.”

Reasoning

E1, the original Transformer paper, presents an architecture that drops recurrence and reports results superior in quality to earlier models, with faster, more parallel training. E3, a secondary history, describes the Transformer as the design that scaled into today's large language models. E2 names the Transformer as a notable new architecture but finds that increased compute contributed even more to performance gains. E2 dates from 2024, and its quote does not state the period it analyses. Together the sources treat the Transformer as a notable architectural advance while crediting compute as contributing even more, which fits "mostly true".

Cite this claim

Eliezer Yudkowsky, “Transformers were the main architectural change that gave a non-linear jump in performance, compared with earlier approaches such as RNNs.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_283307975060, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_283307975060

This claim is in dataset release v1.1.2, with the same verdict.

Reply to this claim

For Eliezer Yudkowsky, or someone speaking for them: a correction or counter-evidence. Replies are read before anything is shown; an approved reply shows your name, role, date, statement and link. Your email address is never shown and is deleted 30 days after the decision (privacy).

More factual claims by Eliezer Yudkowsky

  1. Resolved true Factual September 2025
    “We learned that that's false, but we did not learn that you can have a thing that can redesign a tree and build an AI much smarter than itself, that builds an AI much smarter than itself, and that…”

    No AI has yet been shown to redesign a tree and build a much smarter AI that builds a still smarter AI; that scenario has not occurred.

  2. Mostly true Factual 25 March 2024
    “I was a passionate noticer of the human brain not being homogenous, and was therefore unpleasantly surprised by transformers scaling as well as they did.”

    Yudkowsky was unpleasantly surprised by how well transformers scaled, having believed the human brain is not homogeneous.

  3. Mostly true Factual 6 April 2023
    “First of all, you realize in principle that the task of minimizing losses on predicting human text does not stop when you’re as smart as a human, right?”

    In principle, the task of minimizing loss on predicting human text does not stop being improvable once a system is as smart as a human.

  4. Mostly true Factual 30 March 2023
    “Because the basic paradigm of machine learning that we are presently operating under is that you can have the loss function, but only for things you can evaluate.”

    The basic paradigm of machine learning currently in use is that a loss function can be defined only for things that can be evaluated.

  5. Mostly true Factual 30 March 2023
    “So 60 years later, we’re making progress on a bunch of that, thankfully not yet improved themselves.”

    Sixty years later, AI has made progress on a number of the earlier problems discussed, such as vision, though AI systems have not yet improved themselves.

All of them

Verdicts are made automatically; how verdicts are made.