The hinge loss systematically chooses the most offending incorrect answer as the one whose energy is pulled up.

Judged against the definition of hinge loss in the energy-based learning framework (LeCun et al. 2006, 'A Tutorial on Energy-Based Learning') and standard ML theory. True if the hinge loss, as defined there, depends only on the energy of the correct answer and the energy of the most offending incorrect answer (the lowest-energy incorrect answer), so that its gradient pushes up only that answer's energy. Mostly true if this holds with a minor caveat, such as the margin condition: no push-up happens when the margin is already met. Mixed if it holds only for some formulations of hinge loss and not others. Mostly false if the hinge loss as defined pushes up several incorrect answers or another answer, with only a narrow special case matching the claim. False if the hinge loss does not single out the most offending incorrect answer.

Set before the research began.

1. E1 Deep Learning atcold.github.io (NYU Deep Learning course notes, 2020) · 13 April 2020 Own record

   “This loss enforces that the difference between the correct answer and the most offending incorrect answer be at least $m$.”

2. E2 A survey and taxonomy of loss functions in machine learning arXiv (ar5iv rendering)

   “This loss enforces that the difference between the correct answer and the most offending incorrect answer be at least \(m\) Taskar et al. 2003; Altun et al. 2003. Individual energies are not required to take a specific value because the hinge loss depends on energy differences.”

3. E3 Deep Learning atcold.github.io (NYU Deep Learning course notes, 2020) · 13 April 2020 Own record

   “If you were to use this for classification, you would have $x_1$ be the score of the correct answer and $x_2$ be the score of the highest scoring incorrect answer in the mini-batch.”

4. E4 Deep Learning atcold.github.io (NYU Deep Learning course notes, 2020) · 13 April 2020 Own record

   “For EBMs, this loss function pushes down on desired categories and pushes up on non-desired categories.”

An independent survey of loss functions (E2), citing Taskar et al. and Altun et al., says the hinge loss requires the energy gap between the correct answer and the most offending incorrect answer to be at least a margin m. The loss therefore depends only on that pair. The NYU course notes (E1, E3) describe the same pairing for hinge-type losses. The definition carries a margin condition: nothing is pushed up once the margin is met. E4 says a loss pushes up on non-desired categories in general; it does not say which incorrect answers are chosen or identify that loss as the hinge loss the claim describes.