Marc Andreessen

Factual · AI capabilities

“this is the invention of this algorithm called The Transformer, which is the form of the neural net that it turns out actually works in a sort of generalized way.”

Marc Andreessen, venture capitalist, interviewee · 12 June 2023 · Marc Andreessen on Why AI Will Save the World (EconTalk)

The Transformer algorithm, a neural net form that works in a sort of generalized way, was a recent theoretical breakthrough.

Mostly true True in substance, wrong in a detail.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

True: the Transformer architecture (introduced in Vaswani et al., 'Attention Is All You Need', 2017) is a neural-network form that proved broadly general across tasks, and it can fairly be called a recent breakthrough as of June 2023. Mostly true: the substance is right but a detail is loose, for example it is an architecture rather than an 'algorithm', or it was an empirical rather than a 'theoretical' breakthrough, or six years counts as only somewhat recent. Mixed: the generality claim holds but the breakthrough was materially misdescribed, or the reverse. Mostly false: the Transformer was not new or not a breakthrough, or it had been shown to work only in narrow areas. False: the Transformer does not exist as described or it is credited to the wrong thing entirely.

Evidence

1. Attention Is All You Need NeurIPS Proceedings (31st Conference on Neural Information Processing Systems, NIPS 2017) · 2017

   “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring”

2. Attention Is All You Need - Wikipedia Wikipedia · 4 December 2023

   “These convinced the team that the Transformer is a general-purpose language model, and not just good for translation.”

3. Transformer (deep learning) - Wikipedia Wikipedia

   “They are used in large-scale natural language processing, computer vision (vision transformers), reinforcement learning,[5] audio,[6] multimodal learning, robotics,[7] and playing chess.[8]”

4. Transformer (deep learning) - Wikipedia Wikipedia

   “The predecessors of transformers were developed as an improvement over previous architectures for machine translation,[3][4] but have found many applications since.”

Reasoning

E1, the 2017 NeurIPS paper, introduces the Transformer as "a new simple network architecture" built on attention, so it is an architecture rather than an "algorithm," and it was shown to work through experiments rather than as a theoretical result. E2 says early tests convinced the authors it was general-purpose and not only good for translation. E3 and E4 list its later uses in language processing, computer vision, reinforcement learning, audio, robotics and other areas, which supports the claim that it works in a general way. The core claim holds, but the description is loose on the points the criteria name: architecture versus algorithm, theoretical versus empirical, and a six-year-old result being called recent. That makes it mostly true.

Cite this claim

Marc Andreessen, “The Transformer algorithm, a neural net form that works in a sort of generalized way, was a recent theoretical breakthrough.” (said 12 June 2023). Verdict: mostly true. Reliability API, claim c_0bb5342c2eac, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_0bb5342c2eac