Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“And current-sized models, already expensive, aren’t reliable enough for many jobs.”
Gary Marcus, author, Marcus on AI 31 March 2024 When Will the GenAI Bubble Burst? (Marcus on AI)
Current-sized LLMs are expensive and not reliable enough for many jobs.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2024-03-31. The claim has two parts: (a) current-sized frontier LLMs as of March 2024 were expensive to train and/or run, and (b) they were not reliable enough for many jobs (for example, error or hallucination rates too high for deployment in a substantial number of occupational tasks). True: contemporaneous evidence (cost estimates for training or inference, industry reports, studies of hallucination and error rates, documented deployment failures or companies limiting LLM use because of accuracy) supports both parts. Mostly true: both parts are broadly supported, but one is somewhat overstated, for example costs were falling fast or reliability was adequate for a large share of tasks while still falling short for many. Mixed: one part is well supported and the other is not. Mostly false: the evidence shows LLMs were already reliable enough for most of the jobs at issue and cheap enough for wide use, with only narrow support for the claim. False: the evidence shows that in March 2024 such models were both inexpensive and reliable across most jobs.
Evidence
- Why the cost of training AI could soon become too much to bear | Fortune Fortune · 4 April 2024
“Although companies like OpenAI and Google don’t disclose the precise costs of training AI models like GPT-4 and Gemini, it’s clearly a fiendishly expensive business—and the bigger and more capable these so-called frontier models get, the more it costs to train them.”
- IT Spending Pulse: As GenAI Investment Grows, Other IT Projects Get Squeezed Boston Consulting Group · 1 July 2024
“The leading challenge is the immaturity of GenAI technology, which was cited as a barrier by 43% of high maturity, 36% of mid maturity, 38% of low maturity, and 50% of companies with little or no maturity.”
- Colorado Lawyer Filed a Motion Citing Hallucinated ChatGPT Cases (2023) – AI Incident 615 MIT AI Risk Repository (AI Incident Database entry) · 1 January 2026
“The model hallucinated several non-existent cases, which the attorney submitted to the court without verification.”
- LLM inference prices have fallen rapidly but unequally across tasks Epoch AI · 12 March 2025
“The inference price of LLMs has fallen dramatically in recent years.”
Reasoning
E1, published a few days after the judging date, describes training frontier models such as GPT-4 and Gemini as "fiendishly expensive," which supports the cost part. E4, a 2025 analysis, reports that LLM inference prices have fallen dramatically in recent years, which may temper the cost part for running models, though it post-dates the judging date and does not state what running the models cost in March 2024. On reliability, E2 reports that companies cited the immaturity of GenAI technology as their leading barrier to adoption. E3 documents a workplace case in which hallucinated legal citations were filed in court. Together these broadly support the reliability part, but they are indirect and anecdotal rather than measured error rates across jobs.
Cite this claim
Gary Marcus, “Current-sized LLMs are expensive and not reliable enough for many jobs.” (said 31 March 2024). Verdict: mostly true. Reliability API, claim c_5c5d8719cdf9, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_5c5d8719cdf9
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.