In the next decade, a major challenge of computer vision, machine learning and AI will be devising methods that learn good feature hierarchies from unlabeled and labeled data in an integrated fashion.

Judge over 2010–2020. True: during that decade, learning good hierarchical features or representations from unlabeled and labeled data together was a widely recognised major research challenge in computer vision, machine learning or AI. Signs include a large research effort on unsupervised, self-supervised and semi-supervised representation learning, prominent venues such as ICLR, and surveys or leading researchers naming it a central open problem. Mostly true: it was a recognised major challenge, but mainly in a form slightly different from 'integrated' labeled-plus-unlabeled learning (for example pure self-supervised pretraining followed by fine-tuning). Mixed: it was an important research topic, but it was widely seen as secondary to purely supervised learning for much of the decade. Mostly false: the field largely set the problem aside as a minor concern. False: contemporary evidence shows the field regarded it as solved or unimportant throughout the decade.

Set before the research began.

1. E1 Representation Learning: A Review and New Perspectives ar5iv (arXiv mirror)

   “Future work is hopefully going to help identify which of these elements matter most, how to generalize them across a large variety of tasks and architectures, and in particular contexts where most examples are unlabeled, i.e., including an unsupervised component in the training criterion.”

2. E2 Representation Learning: A Review and New Perspectives ar5iv (arXiv mirror)

   “Researchers report than in such conditions, layerwise unsupervised pre-training brought little or no improvement over pure supervised learning from scratch when training for long enough.”

3. E3 Semi-Supervised and Unsupervised Deep Visual Learning: A Survey ar5iv (arXiv mirror)

   “Built upon different assumptions, these paradigms are often developed independently, whilst sharing the same aim to learn more powerful representations and models using unlabeled data.”

4. E4 Semi-Supervised and Unsupervised Deep Visual Learning: A Survey ar5iv (arXiv mirror)

   “SimCLR [12] and SimCLRv2 [72] are follow-up works introducing self-supervised contrastive learning for task-agnostic unsupervised pre-training, followed by supervised or semi-supervised fine-tuning with label supervision as the downstream task.”

The representation-learning review (E1) names learning in contexts where most examples are unlabeled, with an unsupervised component in training, as key future work. A later survey (E3) describes several semi-supervised and unsupervised paradigms, all aimed at learning better representations from unlabeled data, which indicates a large research effort. Some of that effort, such as SimCLR and SimCLRv2 (E4), took the form of self-supervised pre-training followed by supervised or semi-supervised fine-tuning, rather than integrated labeled-plus-unlabeled learning. E2 notes that in certain conditions, and when training for long enough, layerwise unsupervised pre-training brought little or no improvement over supervised training from scratch, but the overall evidence best fits the mostly-true criterion.