Anthropic's Claude sometimes gives wrong answers.
Resolved true The record shows it happened or was so.
Factual: Statements about the present or the past, checked against the record: did they hold up?
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 29 Jun 2026. True if documented evidence shows that Claude models sometimes give factually incorrect answers, for example Anthropic's model cards or system cards reporting hallucination or error rates, Anthropic's product disclaimers that Claude can make mistakes, or benchmark results below 100% accuracy. Mostly true if errors are documented only in narrow areas. False only if the evidence showed Claude to be reliably error-free, which would also rule out the intermediate verdicts.
Set before the research began.
Evidence (3)
-
E1 Asking a chatbot about your ballot? Read this first The San Francisco Standard · 5 October 2026
“Claude, Anthropic’s chatbot, incorrectly stated that Nithya Raman lost the primary race for Los Angeles mayor.”
-
E2 Consumer Terms of Service - Claude Anthropic · 28 October 2026 Own record
“Outputs may not always be accurate and may contain material inaccuracies even if they appear accurate because of their level of detail or specificity.”
-
E3 Claude’s Constitution Anthropic · 28 October 2026 Own record
“Training models is a difficult task, and Claude’s behavior might not always reflect the constitution’s ideals.”
Reasoning
E1 is an independent news report. It documents that Claude gave a factually incorrect answer, saying Nithya Raman lost the Los Angeles mayoral primary. E2, Anthropic's consumer terms, states that Claude's outputs may contain material inaccuracies, which is the kind of product disclaimer the criteria name. E3 says Claude's behavior may not always match its intended ideals. E1 and E2 are dated after 29 Jun 2026, but they describe an ongoing property of Claude rather than a new event, and nothing in the evidence suggests Claude was ever free of errors, so the claim resolves true.
Cite this claim
Daniela Amodei, “Anthropic's Claude sometimes gives wrong answers.” (said 29 June 2026). Verdict: resolved true. Reliability API, claim c_04a67572ecb6, method 0.1.0+28da144b, dataset release v1.1.2. https://epistacklabs.com/claims/c_04a67572ecb6
This claim is in dataset release v1.1.2, with the same verdict.
Reply to this claim
More factual claims by Daniela Amodei
-
Mostly true
“In the lab, Claude designed new protein binders from scratch.”
In laboratory work, Anthropic's Claude designed new protein binders from scratch.
-
Mostly true
“including open-source code much of the internet runs on, before attackers could exploit them.”
-
Resolved true
“They also showed that more than 6,000 women had come in for prenatal care, but only 1,762 had delivered with a healthcare worker present, a discrepancy that the team is investigating and addressing.”
-
Resolved true
“Those dashboards flagged a spike of 25,000 pneumonia cases, so the team can anticipate and plan for future surges.”
-
Resolved true
“We’re very open about the fact that these models sometimes make things up or get confused.”
Anthropic is open about the fact that its AI models sometimes make things up or get confused.
Verdicts are made automatically; how verdicts are made.