Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“Its natural language output still won’t be something that one can reliably hook up to downstream programs;”
Gary Marcus, author 25 December 2022 What to Expect When You're Expecting … GPT-4 (Marcus on AI)
GPT-4's natural language output still won't be reliably usable as input to downstream programs such as databases or virtual assistants.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
A prediction about GPT-4 once released (March 2023); judge it at roughly one year after release (2024-03). True: evidence shows GPT-4's natural-language output could not reliably be passed as input to downstream programs without errors, e.g., documented malformed outputs, hallucinated fields or inconsistent formats when connected to databases, APIs, or assistants, such that developers needed validation, retries, or added structured-output tooling to cope. Mostly true: unreliability is documented but substantially mitigated in practice by function calling or JSON mode within that year, though not eliminated. Mixed: the evidence is split, reliable for some downstream uses but not others. Mostly false: GPT-4 output was broadly reported as reliably usable downstream, with only isolated failures. False: it was shown to be reliably usable as downstream input without notable failures.
Evidence
- Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs arXiv · 31 January 2026
“Furthermore, even the most effective text-to-SQL models, i.e. GPT-4, only achieve 54.89% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand.”
- Introducing Structured Outputs in the API OpenAI · 6 August 2024
“Developers have long been working around the limitations of LLMs in this area via open source tooling, prompting, and retrying requests repeatedly to ensure that model outputs match the formats needed to interoperate with their systems.”
- A simple Claude Opus v. GPT-4 structured JSON benchmark 96layers.ai · 10 June 2024
“The overall performance was good. Using Instruction Set 3 as a baseline — which included the full set of instructions and reminders but without any JSON errors added to the template — Claude Opus scored 98% while GPT-4 scored 99.6%.”
- Function calling and other API updates OpenAI · 13 June 2023
“Function calling allows developers to more reliably get structured data back from the model.”
Reasoning
E1 reports GPT-4 reaching only 54.89% execution accuracy when turning natural-language questions into SQL for databases, against a human result of 92.96%. E2 (OpenAI, August 2024) says developers had long relied on tooling, prompting and repeated retries to get outputs in the formats their systems needed. E4 shows OpenAI added function calling in June 2023 so developers could get structured data back "more reliably", and E3 reports GPT-4 at 99.6% on a structured JSON benchmark's baseline instruction set, though that benchmark dates from June 2024, after the judging window. Taken together, the unreliability is documented [E1, E2], and within the year OpenAI introduced function calling to make structured output more reliable [E4], while E2 shows workarounds were still in use; the report treats this as the criteria's "mostly true".
Cite this claim
Gary Marcus, “GPT-4's natural language output still won't be reliably usable as input to downstream programs such as databases or virtual assistants.” (said 25 December 2022). Verdict: mostly true. Reliability API, claim c_2199c4d2e055, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_2199c4d2e055
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.