Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
42 claims
“system has no idea that a unicorn has one horn, blithely asserts it has four, etc.”
Gary Marcus, writing on X 16 February 2019 Post by @GaryMarcus on X, 2019-02-16 (X)
The system has no idea that a unicorn has one horn and asserts that it has four.
Resolved true The record shows it happened or was so.
What would count as true or false
Judged as of 2019-02-16. The 'system' is OpenAI's GPT-2 and its published unicorn sample text. True: that output describes unicorns as having four horns (for example 'four-horned'), even though unicorns are defined as having one horn. Mostly true: the output contradicts the one-horn idea but the wording or number differs slightly from 'four'. Mostly false: the four-horn statement exists but the claim misattributes it, for example it came from a different system or from text written by humans. False: no such assertion appears in the system's output.
Evidence
- Humans Who Are Not Concentrating Are Not General Intelligences — LessWrong LessWrong (post by sarahconstantin) · 25 February 2019
“These four-horned, silver-white unicorns were previously unknown to science.”
- Humans Who Are Not Concentrating Are Not General Intelligences — LessWrong LessWrong (post by sarahconstantin) · 25 February 2019
“Also, you just said they had *one* horn, so why are you saying they have *four* horns in the next sentence?”
- Improving Coherence and Consistency in Neural Sequence Models with Dual-System,Neuro-Symbolic Reasoning arXiv (ar5iv rendering)
“In another example, GPT-2 states that unicorns have “four horns,” directly after implying that unicorns only have one horn.”
Reasoning
E1 quotes the GPT-2 unicorn sample text directly: "These four-horned, silver-white unicorns were previously unknown to science." E2, from the same independent post, notes that the sample's earlier wording implies one horn, which the four-horn sentence then contradicts. E3, a later academic paper, also attributes the "four horns" statement to GPT-2, directly after it implies one horn. The output therefore describes unicorns as four-horned, which meets the criterion for true.
Cite this claim
Gary Marcus, “The system has no idea that a unicorn has one horn and asserts that it has four.” (said 16 February 2019). Verdict: resolved true. Reliability API, claim c_64af14cb1503, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_64af14cb1503
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.