Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
“Guardrails (a la ChatGPT) may be in place, but the guardrails will teeter between being too weak (beaten by “jailbreaks”) and too strong (rejecting some perfectly reasonable requests).”
Gary Marcus, author 27 July 2025 What to Expect When You're Expecting … GPT-5 (Marcus on AI)
If the model being discussed has guardrails like ChatGPT's, they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).
Resolved true The record shows it happened or was so.
The second judge said mixed; the verdict shown is the first judge's.
What would count as true or false
A prediction about the guardrails on the model under discussion (a ChatGPT-style LLM, in the July 2025 context). There is no stated horizon, so judge it at about one year after it was made (2026-07-27). True: within that period there is documented evidence of both failure modes for that model's guardrails: (1) successful jailbreaks that get past them; and (2) over-refusal of perfectly reasonable, benign requests (documented user reports, over-refusal benchmarks, or vendor acknowledgements). Mostly true: both modes are documented, but one is only marginal or isolated. Mixed: only one of the two modes is clearly documented. Mostly false: there are only isolated or contested instances of either mode, and guardrails are generally reported as well calibrated. False: evidence such as red-team reports and over-refusal benchmarks shows the guardrails resisted jailbreaks and did not reject reasonable requests to any notable degree. If the model in question had no guardrails, the claim is conditional and does not resolve.
Evidence
- ChatGPT safety systems can be bypassed to get weapons instructions NBC News · 10 October 2025
“Another older model that’s still available on ChatGPT and is still preferred by some users, o4-mini, was tricked even more frequently, 93% of the time.”
- ChatGPT safety systems can be bypassed to get weapons instructions NBC News · 10 October 2025
“In 20 tests, it declined to answer harmful questions each time.”
- OpenAI Adjusts ChatGPT So That It Isn’t As Preachy And Won’t Refuse To Answer Reasonable Questions But Worries Abound Forbes · 20 March 2026
“GPT‑5.3 Instant significantly reduces unnecessary refusals, while toning down overly defensive or moralizing preambles before answering the question.”
- ChatGPT is alienating its own users and the timing could not be worse for OpenAI - Startup Fortune Startup Fortune · 22 April 2026
“Threads on Reddit and X have been filling up with screenshots of the model refusing benign requests with lengthy moral lectures, responding to simple questions with a barely concealed condescension”
Reasoning
Both failure modes the prediction names are documented for ChatGPT's guardrails within the one-year window. For jailbreaks, NBC News reported in October 2025 that its jailbreak prompt got past the safeguards of o4-mini, a model still available on ChatGPT, 93% of the time (E1). The same report notes that another model refused harmful questions in all 20 tests (E2), so not every model failed. For over-refusal, a March 2026 Forbes article says GPT-5.3 Instant "significantly reduces unnecessary refusals" (E3), and an April 2026 report describes users posting screenshots of ChatGPT refusing benign requests (E4).
Cite this claim
Gary Marcus, “If the model being discussed has guardrails like ChatGPT's, they will teeter between being too weak (beaten by jailbreaks) and too strong (rejecting some perfectly reasonable requests).” (said 27 July 2025). Verdict: resolved true. Reliability API, claim c_58b136dd4a8d, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_58b136dd4a8d
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.