Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
79 72 85 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
80 72 88 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
Insufficient record: 1 promise resolved, 5 needed for a score.
All types combined: 79 (likely 74 to 84, high confidence), from 102 claims: 64 factual claims, 38 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Gary Marcus's claims
52 claims
“None of the top labs actually is remotely transparent, governance basically doesn’t exist yet, and there are basically no safeguards actually in place.”
Gary Marcus, Substack author 30 March 2023 The Open Letter Controversy (Marcus on AI)
As of March 2023, none of the top AI labs is remotely transparent, AI governance basically doesn't exist, and there are basically no safeguards in place.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
A three-part factual claim, judged as of 2023-03-30 and read with its hedged, rhetorical wording ('remotely', 'basically'). (1) None of the top labs (OpenAI, Google/DeepMind, Anthropic, Meta) was remotely transparent; (2) AI governance basically didn't exist; (3) there were basically no safeguards in place. True: all three hold, meaning top labs withheld key details such as training data, architecture and safety evaluations, there was no binding AI-specific regulation of frontier models in force, and there were no meaningful external or enforceable safeguards. Mostly true: all three are broadly right, but one is overstated (e.g., labs had some internal safety measures such as RLHF or red-teaming, or some non-binding frameworks existed). Mixed: about half the parts hold, e.g., transparency and governance are accurate but substantial safeguards demonstrably existed. Mostly false: most parts are contradicted, e.g., there were substantive disclosures, governance regimes in force, and real safeguards, but some kernel is true. False: the evidence shows top labs were substantially transparent, governance was in force, and safeguards were in place.
Evidence
- GPT-4 Technical Report arXiv (authors at OpenAI)
“Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.”
- With GPT-4, OpenAI opts for secrecy versus disclosure - ZDNET ZDNET · 16 March 2023
“In the GPT-4 technical report published Tuesday, alongside the blog post by OpenAI, the firm states that it is refraining from offering technical details because of competitive and safety considerations.”
- GPT-4 Technical Report arXiv (authors at OpenAI)
“It also describes interventions we made to mitigate potential harms from the deployment of GPT-4, including adversarial testing with domain experts, and a model-assisted safety pipeline.”
- NIST Risk Management Framework Aims to Improve Trustworthiness of Artificial Intelligence National Institute of Standards and Technology (U.S. Department of Commerce) · 26 January 2023
“has released its Artificial Intelligence Risk Management Framework (AI RMF 1.0), a guidance document for voluntary use by organizations designing, developing, deploying or using AI systems”
Reasoning
On transparency, OpenAI's GPT-4 report (E1) and ZDNET's coverage (E2) show that OpenAI withheld its architecture, model size, compute, dataset and training method. That supports the claim for OpenAI, though the evidence does not cover the other labs. On governance, the only framework in the evidence is NIST's AI RMF 1.0, which is voluntary (E4); this fits the criteria's example of a non-binding framework that leaves the claim mostly true. On safeguards, OpenAI described internal measures such as adversarial testing with domain experts and a model-assisted safety pipeline (E3), so 'basically no safeguards' is overstated, matching the criteria's mostly_true case.
Cite this claim
Gary Marcus, “As of March 2023, none of the top AI labs is remotely transparent, AI governance basically doesn't exist, and there are basically no safeguards in place.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_d75f143920c5, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_d75f143920c5
Claims without a verdict
Is a verdict here wrong, or do you speak for Gary Marcus? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.