Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
78 72 83 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
64 43 83 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 0 promises resolved, 5 needed for a score.
All types combined: 77 (likely 71 to 82, high confidence), from 84 claims: 79 factual claims, 5 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Yann LeCun's claims
23 claims
“New model from XLM outperforms BERT on all GLUE tasks, trained on same data.”
Yann LeCun, on X 21 June 2019 Post by @ylecun on X, 2019-06-21 (X)
A new XLM model outperforms BERT on all GLUE tasks when trained on the same data.
Resolved true The record shows it happened or was so.
What would count as true or false
True if the XLM-based model reported around June 2019 (Facebook's XLM update / leaderboard entry) scored higher than BERT on every GLUE task when trained on the same data as BERT, according to the reported results. Mostly true if it beat BERT on all tasks but the 'same data' condition holds only roughly (for example a minor difference in the training corpus), or the comparison was against a particular BERT variant. Mixed if it beat BERT on most but not all GLUE tasks. Mostly false if it beat BERT only on the average score or a minority of tasks, or was trained on substantially more data. False if it did not outperform BERT on GLUE.
Evidence
- GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining. GitHub (facebookresearch/XLM repository README) · 2 February 2019
“Overall, our model achieves a better performance than the original BERT on all GLUE tasks (cf. table above for comparison).”
- GitHub - facebookresearch/XLM: PyTorch original implementation of Cross-lingual Language Model Pretraining. GitHub (facebookresearch/XLM repository README) · 2 February 2019
“Our XLM PyTorch English model is trained on the same data than the pretrained BERT TensorFlow model (Wikipedia + Toronto Book Corpus). Our implementation does not use the next-sentence prediction task and has only 12 layers but higher capacity (665M parameters).”
Reasoning
The facebookresearch/XLM README states that the English XLM model performs better than the original BERT on all GLUE tasks, and points to its comparison table (E2). The same README says the model was trained on the same data as the pretrained BERT model, Wikipedia plus the Toronto Book Corpus (E3). It also notes architectural differences: no next-sentence prediction and a higher capacity of 665M parameters. The criteria do not count capacity against the claim, so the reported results meet the claim's conditions.
Cite this claim
Yann LeCun, “A new XLM model outperforms BERT on all GLUE tasks when trained on the same data.” (said 21 June 2019). Verdict: resolved true. Reliability API, claim c_31caad607678, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_31caad607678
Claims without a verdict
- Unresolvable 110
- Not yet due 3
- Not researched 309
- Not checkable 669
Is a verdict here wrong, or do you speak for Yann LeCun? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.