Each NLL iteration involves considerably more work because pushing up every incorrect answer is expensive when the second term's derivative has no analytical form.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judged as of 2006, as a technical claim about energy-based learning. True: the standard analysis (for example, LeCun et al. 2006, 'A Tutorial on Energy-Based Learning') shows that the gradient of the NLL loss has a second term, an expectation over all possible answers weighted by their Gibbs probabilities, which in general has no closed form and needs costly computation or approximation (such as sampling or summing over the answer space), so each NLL iteration costs considerably more than losses that push up only one or a few contrastive answers. Mostly true: the reasoning is correct, but with a minor imprecision, for example the cost depends on the model, and some structured models allow tractable computation. Mixed: the extra cost holds only in some settings that make up roughly half of the cases discussed. Mostly false: the extra cost is real only in narrow cases, and the general claim overstates it. False: NLL iterations are not generally more expensive for this reason.
Set before the research began.
Evidence (3)
-
E1 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning) · 1 January 2012
“This operation is intractable for many interesting models.”
-
E2 18 Confronting the Partition Function deeplearningbook.org (Goodfellow, Bengio and Courville, Deep Learning) · 1 January 2012
“Each of these approximations can be understood as making the negative phase computationally cheaper but also making it push down in the wrong locations.”
-
E3 To Relieve Your Headache of Training an MRF, Take AdVIL ICLR 2020 (Li, Du, Xu, Welling, Zhu, Zhang) · 2020
“Unfortunately, both the NLL in Eqn. (1) and its gradient in Eqn. (2) are intractable in general for two reasons.”
Reasoning
The Deep Learning textbook's chapter on the partition function says the operation it discusses is intractable for many interesting models (E1), and that approximations make the negative phase computationally cheaper (E2); an ICLR 2020 paper states that the NLL and its gradient are intractable in general (E3). They also say that the usual approximations exist to make the negative phase cheaper to compute (E2). The ICLR 2020 paper says the NLL and its gradient are intractable in general (E3). The claim applies only when the second term has no analytical form, so it already allows for the tractable cases, and the evidence supports the standard analysis that exact NLL updates cost considerably more.
Cite this claim
Yann LeCun, “Each NLL iteration involves considerably more work because pushing up every incorrect answer is expensive when the second term's derivative has no analytical form.” (said 2006). Verdict: resolved true. Reliability API, claim c_c68de5ffea08, method 0.1.0+c876030c, dataset release v1.1.2. https://epistacklabs.com/claims/c_c68de5ffea08
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Yann LeCun
-
Mostly true
“For starters, transformers are equivariant to permutations.”
Transformers are equivariant to permutations, as ConvNets are equivariant to translations.
-
Mostly true
“If it were the case, we would have AI systems that could teach themselves to drive a car in 20 hours of practice, like any 17 year-old.”
-
Mostly true
“The architecture is good, the architecture of the encoder is good, but the fact that you train the system to reconstruct images does not lead it to produce to long, good generic features of images.”
-
Mostly true
“LLMs can do none of those or they can only do them in a very primitive way and they don’t really understand the physical world.”
-
Mostly true
“No, there’s one thing that autoregressive LLMs or that LLMs in general, not just the autoregressive one, but including the bird style bidirectional ones, are exploiting and its self supervised…”
Verdicts are made automatically; how verdicts are made.