Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 70 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
60 42 78
All types combined: 74 (likely 68 to 80, high confidence), from 79 claims: 71 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Eliezer Yudkowsky's claims
46 claims
“Because the basic paradigm of machine learning that we are presently operating under is that you can have the loss function, but only for things you can evaluate.”
Eliezer Yudkowsky, interviewee 30 March 2023 Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript)
The basic paradigm of machine learning currently in use is that a loss function can be defined only for things that can be evaluated.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge as of 2023-03-30. True: authoritative ML sources (textbooks, surveys, expert commentary) show that mainstream ML training at the time, i.e. gradient-based optimisation of a loss computed against labels, rewards or ratings, required a computable or evaluable signal for whatever was optimised, so that properties that cannot be measured or evaluated cannot be directly trained on. Mostly true: the core point holds but the claim slightly overstates it, e.g. it ignores proxy, self-supervised or learned-reward methods that only partly get around it. Mixed: experts are split on whether this describes 'the basic paradigm' in any meaningful way. Mostly false: the claim is technically related but badly distorts practice, e.g. widespread methods at the time did train effectively on targets that could not be evaluated. False: the evidence shows ML at the time routinely defined and optimised losses on things that could not be evaluated.
Evidence
- Neural Network Training and Non-Differentiable Objective Functions arXiv (Czech Technical University in Prague thesis, Faculty of Electrical Engineering) · May 2023
“Training deep networks by gradient descent on the user-defined objective is not possible when the objective is non-differentiable. Deep learning methods use a proxy loss function as a workaround.”
- Lecture 3, Part 2: Training a Classifier University of Toronto, CSC311 course notes (Roger Grosse)
“We can’t just use the classification error itself, because the gradient is zero almost everywhere! Instead, we’ll define a surrogate loss function, i.e. an alternative loss function which is easier to optimize.”
- Survey on Self-Supervised Learning: Auxiliary Pretext Tasks and Contrastive Learning Methods in Imaging Entropy (MDPI), via PMC · 14 April 2022
“The early methods of SSL are based on auxiliary pretext tasks as a way to learn representations using pseudo-labels, or labels that were created automatically based on the dataset’s attributes.”
- Neural Network Training and Non-Differentiable Objective Functions arXiv (Czech Technical University in Prague thesis, Faculty of Electrical Engineering) · May 2023
“Note that methods for training on non-differentiable losses, such as actor-critic [185] and learning surrogates [20] are not directly applicable to recall@k.”
Reasoning
University course notes (E2) and a 2023 thesis (E1) say gradient-based training needs a computable, optimisable loss. When the true objective can't be optimised directly, a surrogate or proxy loss is used instead, which fits the core of the claim. However, E4 names actor-critic and learned-surrogate methods for training on non-differentiable objectives, and E3 describes self-supervised learning with labels created automatically from the data. These are ways of partly working around the limit, so the claim overstates it slightly. That matches the 'mostly true' criterion.
Cite this claim
Eliezer Yudkowsky, “The basic paradigm of machine learning currently in use is that a loss function can be defined only for things that can be evaluated.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_b860c50af4d2, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_b860c50af4d2
Claims without a verdict
Is a verdict here wrong, or do you speak for Eliezer Yudkowsky? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.