LLMs, including BERT-style bidirectional ones, exploit self-supervised learning, and LeCun has been a very strong advocate of self-supervised learning for many years.

The claim has two parts: (a) LLMs, both autoregressive and BERT-style bidirectional, are trained with self-supervised learning (e.g., next-token prediction or masked-token prediction on unlabeled text); (b) LeCun had been a very strong public advocate of self-supervised learning for many years before March 2024 (several years at least, e.g., from the mid-2010s or earlier). TRUE: both parts hold. MOSTLY TRUE: both hold in substance with a minor slip (e.g., his advocacy is documented for somewhat fewer years than the wording suggests, but still several years). MIXED: one part holds and the other clearly does not. MOSTLY FALSE: (a) holds only for some model types, and (b) is weak or recent. FALSE: standard LLM pretraining is shown not to be self-supervised, and LeCun is not shown to have advocated it before.

Set before the research began.

1. E1 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding arXiv (ar5iv rendering) · 3 April 2018

   “BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers.”

2. E2 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding arXiv (ar5iv rendering) · 3 April 2018

   “The training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction likelihood.”

3. E3 Improving Language Understanding by Generative Pre-Training OpenAI (Radford, Narasimhan, Salimans, Sutskever)

   “First, we use a language modeling objective on the unlabeled data to learn the initial parameters of a neural network model.”

4. E4 AI chip evolution will soon take new direction says AI pioneer - Embedded Embedded.com (sister site of EE Times) · 19 February 2019

   ““In my opinion, the future of AI is self-supervised learning,” said Yann LeCun, who is considered the father of convolutional neural nets (CNNs) now used widely in computer vision and other systems.”

The BERT paper says BERT is pre-trained on unlabeled text using a masked language modelling objective (E1, E2). The original GPT paper describes a language modelling objective on unlabeled data (E3). Together these support part (a) for both bidirectional and autoregressive models. For part (b), a 2019 Embedded.com article quotes LeCun saying "the future of AI is self-supervised learning" (E4), which shows public advocacy about five years before March 2024. That is several years, but the evidence does not show advocacy going back to the mid-2010s or earlier, so the verdict is mostly true rather than true.