Masked autoencoders are less successful with images and other continuous domains than with discrete domains such as language.

Judged as of June 2023. True: the published literature as of mid-2023 (e.g. comparisons of masked autoencoding / masked prediction in vision versus language, such as MAE, BEiT and BERT results and surveys) shows that masked reconstruction pretraining has generally given weaker or less dominant results on images and other continuous signals (for example, needing fine-tuning to be competitive, weaker linear-probe performance, lagging behind joint-embedding or contrastive methods) than it has in discrete domains such as language, where it became the dominant method. Mostly true: this holds for images, but evidence for 'continuous domains in general' (audio, video and so on) is thin or mixed, or the gap holds only on some measures (for example, linear probing but not fine-tuning). Mixed: the literature is roughly evenly split on whether masked AEs underperform in vision compared with language. Mostly false: masked AEs in vision were by then broadly competitive with the best alternatives, and the claim rests only on narrow measures. False: the evidence shows masked AEs were as successful, or more successful, in images and continuous domains as in language.

Set before the research began.

1. E1 Masked Autoencoders Are Scalable Vision Learners ar5iv (arXiv:2111.06377, He et al.)

   “However, despite significant interest in this idea following the success of BERT, progress of autoencoding methods in vision lags behind NLP.”

2. E2 Masked Autoencoders Are Scalable Vision Learners ar5iv (arXiv:2111.06377, He et al.)

   “On the other hand, it still lags behind contrastive methods under this protocol: e.g., MoCo v3 Chen2021a has 77.6% linear probing accuracy for the ViT-L (Figure 9).”

3. E3 Masked Autoencoders Are Scalable Vision Learners ar5iv (arXiv:2111.06377, He et al.)

   “MoCo v3 has higher linear probing accuracy; however, all of its partial fine-tuning results are worse than MAE.”

4. E4 A Survey on Masked Autoencoder for Visual Self-supervised Learning International Joint Conference on Artificial Intelligence (IJCAI-23 proceedings) · 2023

   “By contrast, early attempts at generative methods in vision have been outperformed by their discriminative counterparts (like contrastive learning).”

The MAE paper says autoencoding methods in vision lagged behind NLP after BERT (E1). It also reports that MAE still trails contrastive methods such as MoCo v3 on linear probing (E2), but beats them under partial fine-tuning (E3). The IJCAI-23 survey notes that early generative methods in vision were outperformed by discriminative ones such as contrastive learning (E4). So the claim holds for images on some measures, while the evidence on fine-tuning and on continuous domains beyond images is mixed or thin. Under the criteria, that fits "mostly true".