Training an encoder to reconstruct images does not lead it to produce good generic image features, though the encoder architecture is good.

Judge against evidence as of March 2024 on encoders trained by image reconstruction (for example, masked autoencoders and denoising or generative objectives). 'Good generic features' means features that perform well on many downstream tasks, especially when the features are frozen (for example, linear probing), compared with other pretraining methods. TRUE: the literature shows that reconstruction-trained encoders with good architectures (such as ViTs) produce clearly weaker generic frozen features than joint-embedding or supervised alternatives, so the architecture is not the cause. MOSTLY TRUE: frozen features from reconstruction are notably weaker, but become competitive after fine-tuning. MIXED: results depend heavily on the evaluation protocol, with reconstruction competitive on a substantial share of generic evaluations. MOSTLY FALSE: reconstruction-trained features are broadly competitive, with only narrow weaknesses. FALSE: reconstruction-based pretraining produced generic features that were state of the art or on a par with the best methods across standard evaluations.

Set before the research began.

1. E1 Masked Autoencoders Are Scalable Vision Learners arXiv (ar5iv render); authors Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

   “Our MAE with ViT-L has 75.8% linear probing accuracy. This is substantially better than previous masking-based methods. On the other hand, it still lags behind contrastive methods under this protocol: e.g., MoCo v3 Chen2021a has 77.6% linear probing accuracy for the ViT-L (Figure 9).”

2. E2 Masked Autoencoders Are Scalable Vision Learners arXiv (ar5iv render); authors Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick

   “MoCo v3 has higher linear probing accuracy; however, all of its partial fine-tuning results are worse than MAE.”

3. E3 Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget arXiv (ar5iv render); Masked Autoencoder Contrastive Tuning (MAE–CT) authors

   “The difference in structure can also be seen by evaluating the embedding directly via linear probing or \(k\)-NN classification where ID performs significantly better than MIM.”

4. E4 Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget arXiv (ar5iv render); Masked Autoencoder Contrastive Tuning (MAE–CT) authors

   “With ViT-H/16 MAE–CT achieves a new state-of-the-art in linear probing of 82.2%.”

The MAE paper reports that its reconstruction-trained ViT-L reaches 75.8% linear-probing accuracy, behind the contrastive MoCo v3 at 77.6% (E1). The same paper reports that MAE beats MoCo v3 once top blocks are fine-tuned (E2). The MAE–CT paper says instance-discrimination embeddings do significantly better than masked image modeling under linear probing and k-NN (E3), and its MAE–CT model, which adds contrastive tuning to MAE, reports a new state-of-the-art linear-probe result of 82.2% with ViT-H/16 (E4). Together, the evidence shows that frozen features from reconstruction training are weaker but competitive after fine-tuning, which is the "mostly true" case in the criteria.