Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
42 claims
“change the environment, and lots of models start to break:”
Gary Marcus, posting on X 12 May 2020 Post by @GaryMarcus on X, 2020-05-12 (X)
When the environment changes, as during the pandemic, many AI models start to break.
Resolved true The record shows it happened or was so.
What would count as true or false
Judge as of May 2020. TRUE: reporting or research from around then documents many AI or machine learning models (several across different domains, such as demand forecasting, inventory, fraud detection, recommendation or credit models) degrading or failing because behaviour or data shifted during the COVID-19 pandemic, or more generally documents that distribution shift commonly breaks models. MOSTLY TRUE: documented breakdowns exist across several cases, but 'lots' somewhat overstates how widespread they were. MIXED: there are only a few documented cases alongside reports that many models coped. MOSTLY FALSE: only isolated anecdotes exist, presented as a general phenomenon. FALSE: evidence shows that models largely kept working through the pandemic's changes.
Evidence
- The algorithms big companies use to manage their supply chains don’t work during pandemics The Verge · 27 April 2020
““That’s what we’re seeing over and over,” Beal says. “A lot of these systems haven’t caught up.””
- The algorithms big companies use to manage their supply chains don’t work during pandemics The Verge · 27 April 2020
“The models can still be used. “It’s the data that you input that has to be changed,” Nagurney says.”
- What to Do Now That Your Demand Forecast Is Wrong Bain & Company · 7 April 2020
“Due to the crisis, most demand forecast models in place today are no longer accurate.”
- Concept Drift and the Impact of COVID-19 on Data Science Iguazio · 30 April 2020
“Concept drift is also impacting models designed to predict fraud across various industries. For example, previously models were trained to view the purchase of one-way flight tickets as a strong indicator of airline fraud.”
Reasoning
Bain's April 2020 analysis says most demand forecast models in use were no longer accurate because of the crisis (E3). The Verge quotes Beal saying "That's what we're seeing over and over" and that "a lot of these systems haven't caught up" in an article on supply-chain algorithms during the pandemic (E1). Together these show model degradation in several domains around May 2020. E2 notes the models could still be used once the input data changed, but that describes how to fix them rather than showing they kept working.
Cite this claim
Gary Marcus, “When the environment changes, as during the pandemic, many AI models start to break.” (said 12 May 2020). Verdict: resolved true. Reliability API, claim c_67ea09215561, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_67ea09215561
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.