Yann LeCun

Prediction · AI timelines and forecasting

“The next frontier for Deep Learning is natural language understanding.”

Yann LeCun, Facebook AI director, interviewed · 18 February 2015 · Facebook AI Director Yann LeCun on His Quest to Unleash Deep Learning (IEEE Spectrum)

Natural language understanding will be the next frontier for Deep Learning.

Mostly true True in substance, wrong in a detail.

What would count as true or false

Judge over the years after February 2015 (about 2015–2020). True: natural language understanding became the main area of major new deep-learning advances and research attention in that period. Signs include landmark work (attention and Transformers, BERT, GPT-series language models), benchmarks such as GLUE and SuperGLUE, and contemporary reviews describing NLP or NLU as where deep learning made its next big breakthroughs. Mostly true: NLU was one of the leading frontiers, roughly equal with another area. Mixed: NLU advanced substantially, but other areas (for example deep reinforcement learning or image generation) were seen as at least as much the 'next frontier'. Mostly false: NLU progress was secondary, and another area was clearly the next frontier. False: deep learning made little progress in, or paid little attention to, NLU in that period.

Evidence

1. Last Words: Computational Linguistics and Deep Learning - Nautilus Nautilus (reprinting a Computational Linguistics "Last Words" column) · December 2015

   “So far, problems in higher-level language processing have not seen the dramatic error rate reductions from deep learning that have been seen in speech recognition and in object recognition in vision.”

2. Last Words: Computational Linguistics and Deep Learning - Nautilus Nautilus (reprinting a Computational Linguistics "Last Words" column) · December 2015

   “The next big step for Deep Learning is natural language understanding, which aims to give machines the power to understand not just individual words but entire sentences and paragraphs.”

3. Attention Is All You Need arXiv (Google authors)

   “We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”

4. Mastering the game of Go without human knowledge - Nature Nature (DeepMind authors) · 19 October 2017

   “Here we introduce an algorithm based solely on reinforcement learning, without human data, guidance or domain knowledge beyond game rules.”

5. Last Words: Computational Linguistics and Deep Learning - Nautilus Nautilus · 16 March 2017

   “The next big step for Deep Learning is natural language understanding, which aims to give machines the power to understand not just individual words but entire sentences and paragraphs.”

6. Last Words: Computational Linguistics and Deep Learning - Nautilus Nautilus · 16 March 2017

   “Deep Learning waves have lapped at the shores of computational linguistics for several years now, but 2015 seems like the year when the full force of the tsunami hit the major Natural Language Processing (NLP) conferences.”

7. Last Words: Computational Linguistics and Deep Learning - Nautilus Nautilus · 16 March 2017

   “So far, problems in higher-level language processing have not seen the dramatic error rate reductions from deep learning that have been seen in speech recognition and in object recognition in vision.”

8. Attention Is All You Need arXiv (Google authors)

   “Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU.”

9. The state of AI in 2019: Breakthroughs in machine learning, natural language processing, games, and knowledge graphs - ZDNET ZDNET · 8 July 2019

   “As a demonstration of how quickly progress is being made in NLP, they go on to add, the state-of-the art has increased from a GLUE score of 69 to 88 over 13 months. The human baseline level is 87. Progress was so much faster than anticipated that a new benchmark SuperGLUE has already been introduced.”

10. The state of AI in 2019: Breakthroughs in machine learning, natural language processing, games, and knowledge graphs - ZDNET ZDNET · 8 July 2019

    “Key areas covered are reinforcement learning, applications in games and future directions, natural language processing breakthroughs, deep learning in medicine, and AutoML.”

11. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding arXiv (ar5iv rendering of Devlin et al., Google) · 3 April 2018

    “It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).”

12. NeurIPS Poster Language Models are Few-Shot Learners NeurIPS · 1 October 2020

    “We demonstrate that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even becoming competitive with prior state-of-the-art fine-tuning approaches.”

Reasoning

The evidence includes the landmark language work the criteria name: the Transformer (E3, E8), BERT's state-of-the-art results on GLUE, MultiNLI and SQuAD (E11), and the NeurIPS 2020 paper "Language Models are Few-Shot Learners", which reports that scaling up language models greatly improves few-shot performance (E12). ZDNET in 2019 reported that GLUE progress was so much faster than anticipated that a new benchmark, SuperGLUE, had already been introduced (E9), and a column said 2015 seemed to be the year when 'the full force of the tsunami' of deep learning hit the major NLP conferences (E6). However, the same 2019 ZDNET review lists NLP breakthroughs alongside reinforcement learning and games (E10), and AlphaGo Zero (E4) was a major reinforcement-learning landmark. So the evidence shows NLU as one of the leading frontiers, but it does not show that NLU clearly dominated every other area.

Cite this claim

Yann LeCun, “Natural language understanding will be the next frontier for Deep Learning.” (said 18 February 2015). Verdict: mostly true. Reliability API, claim c_a3663ae32bf0, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_a3663ae32bf0