Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
72 65 79 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
66 54 78 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 71 (likely 64 to 77, high confidence), from 79 claims: 60 factual claims, 19 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Balaji Srinivasan's claims
“The current crop of agents can’t set complex goals, or properly verify outputs.”
Balaji Srinivasan, author of his own blog post 3 August 2025 AI is polytheistic, not monotheistic (balajis.com)
Current AI agents cannot set complex goals or properly verify outputs.
Mostly true True in substance, wrong in a detail.
What would count as true or false
Judge as of 2025-08-03, against the state of publicly available AI agents at that time. The claim says agents could not do two things: set complex goals, and properly verify their own outputs. TRUE: benchmark results, expert assessments or technical reports from mid-2025 show that agents of the time generally could not reliably set complex goals on their own and could not reliably verify their outputs (e.g. low success on long-horizon agentic benchmarks, documented failures at self-verification). MOSTLY TRUE: this holds broadly, but there are minor, narrow exceptions (e.g. verification in specific domains such as running code tests). MIXED: the evidence supports one of the two limitations but shows agents could do the other reasonably well. MOSTLY FALSE: agents of the time demonstrably did both in significant settings, so the blanket 'can't' overstates real but partial limitations. FALSE: agents of the time routinely and reliably did both.
Evidence
- Measuring AI Ability to Complete Long Tasks arXiv (METR authors: Kwa, West et al.)
“We also noted important limitations of current systems, particularly their lower performance on less structured, “messier” tasks”
- Measuring AI Ability to Complete Long Tasks arXiv (METR authors: Kwa, West et al.)
“On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes.”
- How Does Time Horizon Vary Across Domains? METR · 14 July 2025
“The agent fails to click the correct coordinates despite planning detailed and accurate steps in their code comments, indicating strong planning but weak execution capabilities.”
- 4 Phases To Smarter AI Integration For Business Oliver Wyman · 22 July 2025
“A benchmark study by Salesforce AI Research found that AI agents achieve only 58% success on single-turn tasks. Add multi-turn interactions, and performance drops to 35%.”
- Measuring AI Ability to Complete Long Tasks arXiv (METR authors)
“On these tasks, current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes.”
- Measuring AI Ability to Complete Long Tasks arXiv (METR authors)
“We also measure the 80% time horizon of models (Figure 6) and find a similar trend, though horizons are roughly 5x shorter.”
- Measuring AI Ability to Complete Long Tasks arXiv (METR authors)
“We also note several limitations of current systems—notably, performance is much lower on less structured, “messier” tasks (Section 6.2).”
- How Does Time Horizon Vary Across Domains? METR · 14 July 2025
“Agentic GUI tasks (OSWorld) are still ~100 × shorter than coding or math; Tesla self-driving has improved much slower than other domains.”
- On the self-verification limitations of large language models on reasoning and planning tasks ICLR 2025 Proceedings · 1 May 2025
“We observe significant performance collapsewith self-critique and significant performance gains with sound external verification.”
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains arXiv (Sierra Research authors) · 2 May 2024
“Our experiments show that even state-of-the-art function calling agents (like gpt-4o) succeed on \(<50\%\) of the tasks, and are quite inconsistent (pass^8 < 25% in retail).”
- τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment arXiv · 15 January 2025
“gpt-4.1 pass \(\hat{}\) 1 drops from 74%/56% for retail and airline respectively to 34% for telecom.”
- System Card: Claude Opus 4 & Claude Sonnet 4 Anthropic · May 2025
“They are particularly adept at complex computer coding tasks, which they can productively perform autonomously for sustained periods of time.”
Reasoning
An ICLR 2025 paper found that performance collapses when LLMs critique their own work on reasoning and planning tasks, and improves with sound external verification (E9). That supports the self-verification limitation. METR and other benchmarks show partial, unreliable capability on longer and messier tasks: a 50% time horizon of about 50 minutes, 80% horizons roughly 5x shorter, and low or inconsistent success on multi-turn tasks (E1, E2, E6, E7, E4, E10, E11). Together these show limited reliability on longer, less structured and multi-step tasks; they measure task completion rather than goal-setting, so they bear only indirectly on the claim that agents cannot set complex goals. There are narrow exceptions: Anthropic describes sustained autonomous coding (E12), and OSWorld agents planned well but executed poorly (E3). These temper the blanket "can't" without overturning it.
Cite this claim
Balaji Srinivasan, “Current AI agents cannot set complex goals or properly verify outputs.” (said 3 August 2025). Verdict: mostly true. Reliability API, claim c_87e91de51727, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_87e91de51727
Claims without a verdict
- Unresolvable 148
- Not yet due 21
- Own records only 2
- Not researched 227
- Not checkable 1,091
Is a verdict here wrong, or do you speak for Balaji Srinivasan? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.