Tested against published outcomes
Before anyone relies on a verdict, it helps to see the method reach answers that someone else has already published.
How the test works
Each set is a group of claims whose outcome someone else has published. Every claim was run through the ordinary method with the published answer blocked from its research, except the Seoul set, run again on 14 October 2026: there research searched freely, as in a customer’s report, and each verdict whose evidence came from the answer was marked afterwards. Only then was each verdict set beside the published outcome. Where a page repeating the answer still reached the evidence, that claim is left out of the numbers.
Case studies
Elon Musk’s self-driving timelines
24 dated predictions. The outcome comes from Wikipedia’s list, CC BY-SA 4.0. They either happened or did not, and news reports say so.
Seoul AI safety commitments
19 promises, one per company: publish a safety framework by the Paris summit. The outcome comes from METR’s list. Frameworks are published documents with dates. 7 are left out of the numbers because their evidence came from METR’s list, a page that cites it, or another tracker of these commitments. With them counted: right direction 14 of 14, same verdict 11 of 18, unresolvable 4.
White House AI commitments, 2023
56 promises by 16 companies, scored by Wang et al. 2025, CC BY 4.0. It mostly tests whether evidence exists outside the answer key: the reference scores what companies said they did, and the method asks for the deliverable itself.
The numbers
“Right direction”: take the claims where both the published record and the machine gave a clear answer, kept or not kept. How often did they give the same answer?
“Same verdict” is stricter: how often the machine’s answer matched exactly, including how sure it was. “Resolved false” beside “not met” counts; “mostly false” beside “not met” does not.
“Unresolvable”: claims where the machine found too little dated evidence to decide, so it gave no verdict rather than guess.
| Set | Right direction | Same verdict | Unresolvable |
|---|---|---|---|
| Musk | 22 of 22 | 20 of 24 | 0 |
| Seoul | 9 of 9 | 7 of 11 | 2 |
| White House | 18 of 20 | 13 of 56 | 23 |