Eliezer Yudkowsky

Factual · AI capabilities

“So 60 years later, we’re making progress on a bunch of that, thankfully not yet improved themselves.”

Eliezer Yudkowsky, interviewee · 30 March 2023 · Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript)

Sixty years later, AI has made progress on a number of the earlier problems discussed, such as vision, though AI systems have not yet improved themselves.

Mostly true True in substance, wrong in a detail.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2023-03-30. '60 years later' is a rough figure for the time since the early AI agenda (e.g., the 1956 Dartmouth proposal), so roughly 55–70 years counts. The claim has two parts: (a) by 2023, AI had made real progress on several problems from that early agenda, such as vision and language; (b) no AI system had yet improved itself in the sense of autonomous, recursive self-improvement. True: the evidence supports both parts. Mostly true: both parts hold with a minor slip, such as the year count being somewhat off. Mixed: one part is clearly supported and the other is contested, for example there were documented cases widely regarded as AI systems substantially improving their own design. Mostly false: the progress was real but much narrower than stated, or self-improvement had clearly occurred in a meaningful sense. False: there had been no meaningful progress on these problems, or AI systems had clearly and recursively improved themselves.

Evidence

1. A PROPOSAL FOR THE DARTMOUTH SUMMER RESEARCH PROJECT ON ARTIFICIAL INTELLIGENCE J. McCarthy, M. L. Minsky, N. Rochester, C.E. Shannon (proposal authors) · 31 August 1955

   “An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves.”

2. Microsoft Researchers' Algorithm Sets ImageNet Challenge Milestone - Microsoft Research Microsoft Research · 10 February 2015

   “the researchers say their system achieved a 4.94 percent error rate on the 1000-class ImageNet 2012 classification dataset”

3. Technical Performance Stanford Institute for Human-Centered AI (HAI), AI Index · 1 January 2023

   “Recently released models challenge that trend; BEiT-3, PaLI, and Gato, among others, are single AI systems increasingly capable of navigating multiple tasks (for example, vision, language).”

4. Technical Performance Stanford Institute for Human-Centered AI (HAI), AI Index · 1 January 2023

   “Nvidia used an AI reinforcement learning agent to improve the design of the chips that power AI systems. Similarly, Google recently used one of its language models, PaLM, to suggest ways to improve the very same model.”

Reasoning

E1 shows that the 1955 Dartmouth proposal set out goals including language, abstraction and machines that "improve themselves". That is about 67 years before 2023, which falls within the accepted range. E2 and E3 show clear progress on vision and language by 2023, including a 4.94 percent error rate on ImageNet classification [E2] and multi-task models that handle both [E3]. E4 reports that AI was used to help improve chip design and the PaLM model. Those are human-directed, AI-assisted improvements, not autonomous recursive self-improvement, so they mostly fit the claim that AI had not yet improved itself, but they add some nuance and do not directly confirm that no such case existed.

Cite this claim

Eliezer Yudkowsky, “Sixty years later, AI has made progress on a number of the earlier problems discussed, such as vision, though AI systems have not yet improved themselves.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_d20bbbdfc55e, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_d20bbbdfc55e