A new XLM model outperforms BERT on all GLUE tasks when trained on the same data.

True if the XLM-based model reported around June 2019 (Facebook's XLM update / leaderboard entry) scored higher than BERT on every GLUE task when trained on the same data as BERT, according to the reported results. Mostly true if it beat BERT on all tasks but the 'same data' condition holds only roughly (for example a minor difference in the training corpus), or the comparison was against a particular BERT variant. Mixed if it beat BERT on most but not all GLUE tasks. Mostly false if it beat BERT only on the average score or a minority of tasks, or was trained on substantially more data. False if it did not outperform BERT on GLUE.

Set before the research began.

1. E1 GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining. GitHub (facebookresearch/XLM repository README) · 2 February 2019

   “Overall, our model achieves a better performance than the original BERT on all GLUE tasks (cf. table above for comparison).”

2. E2 GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining. GitHub (facebookresearch/XLM repository README) · 2 February 2019

   “Our XLM PyTorch English model is trained on the same data than the pretrained BERT TensorFlow model (Wikipedia + Toronto Book Corpus). Our implementation does not use the next-sentence prediction task and has only 12 layers but higher capacity (665M parameters).”

The facebookresearch/XLM README states that the English XLM model performs better than the original BERT on all GLUE tasks, and points to its comparison table (E2). The same README says the model was trained on the same data as the pretrained BERT model, Wikipedia plus the Toronto Book Corpus (E3). It also notes architectural differences: no next-sentence prediction and a higher capacity of 665M parameters. The criteria do not count capacity against the claim, so the reported results meet the claim's conditions.