The system has no idea that a unicorn has one horn and asserts that it has four.

Judged as of 2019-02-16. The 'system' is OpenAI's GPT-2 and its published unicorn sample text. True: that output describes unicorns as having four horns (for example 'four-horned'), even though unicorns are defined as having one horn. Mostly true: the output contradicts the one-horn idea but the wording or number differs slightly from 'four'. Mostly false: the four-horn statement exists but the claim misattributes it, for example it came from a different system or from text written by humans. False: no such assertion appears in the system's output.

Set before the research began.

1. E1 Humans Who Are Not Concentrating Are Not General Intelligences — LessWrong LessWrong (post by sarahconstantin) · 25 February 2019

   “These four-horned, silver-white unicorns were previously unknown to science.”

2. E2 Humans Who Are Not Concentrating Are Not General Intelligences — LessWrong LessWrong (post by sarahconstantin) · 25 February 2019

   “Also, you just said they had *one* horn, so why are you saying they have *four* horns in the next sentence?”

3. E3 Improving Coherence and Consistency in Neural Sequence Models with Dual-System,Neuro-Symbolic Reasoning arXiv (ar5iv rendering)

   “In another example, GPT-2 states that unicorns have “four horns,” directly after implying that unicorns only have one horn.”

E1 quotes the GPT-2 unicorn sample text directly: "These four-horned, silver-white unicorns were previously unknown to science." E2, from the same independent post, notes that the sample's earlier wording implies one horn, which the four-horn sentence then contradicts. E3, a later academic paper, also attributes the "four horns" statement to GPT-2, directly after it implies one horn. The output therefore describes unicorns as four-horned, which meets the criterion for true.