Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
83 77 88 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
69 49 87 -
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
69 49 87
All types combined: 81 (likely 76 to 86, high confidence), from 87 claims: 77 factual claims, 5 predictions, 5 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Sam Altman's claims
40 claims
“We think it’s important that efforts like ours submit to independent audits before releasing new systems; we will talk about this in more detail later this year.”
Sam Altman, CEO, writing on OpenAI's blog 24 February 2023 Sam Altman: "Planning for AGI and beyond" (LessWrong (mirror of OpenAI post))
OpenAI will discuss in more detail, later in 2023, independent audits of AI systems before release.
Mostly true True in substance, wrong in a detail.
The evidence is the person's or their organisation's own records.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
True: in 2023, after 24 February, OpenAI published or presented a more detailed discussion of independent audits or external evaluation of AI systems before release (e.g. a blog post, policy paper, testimony or framework addressing external/independent audits or red-teaming before deployment). Mostly true: such a discussion was published in 2023 but dealt with external audits only as one part of a broader governance document. Mostly false: it touched on audits only in passing, or the detailed discussion came only after 2023. False: OpenAI's 2023 publications show no further discussion of independent audits.
Evidence
- Governance of superintelligence - OpenAI OpenAI · 22 May 2023 Own record
“any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc.”
- Governance of superintelligence - OpenAI OpenAI · 22 May 2023 Own record
“We think it’s important to allow companies and open-source projects to develop models below a significant capability threshold, without the kind of regulation we describe here (including burdensome mechanisms like licenses or audits).”
- OpenAI’s Approach to Frontier Risk OpenAI · 26 October 2023 Own record
“We evaluate each new major model released for safety, including using red-teaming. For example, before publicly releasing GPT‑4, external red-teamers tested the model for the following frontier risks”
- OpenAI’s Approach to Frontier Risk OpenAI · 26 October 2023 Own record
“This includes undergoing third-party audits of our security program including SOC 2 Type 2.”
Reasoning
OpenAI's May 2023 post "Governance of superintelligence" says that efforts above a capability threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards and place restrictions on deployment [E1]. It also says that audits should not apply to models below that threshold (E1, E2). Its October 2023 "Approach to Frontier Risk" document describes external red-teaming of GPT-4 before public release and third-party audits of OpenAI's security program (E3, E4). These are OpenAI's own 2023 publications, which count as the delivered output. In both, audits and external evaluation are one part of a broader governance or risk document rather than a dedicated, detailed treatment, which the criteria rate as mostly true.
Cite this claim
Sam Altman, “OpenAI will discuss in more detail, later in 2023, independent audits of AI systems before release.” (said 24 February 2023). Verdict: mostly true. Reliability API, claim c_49d4d81bfd92, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_49d4d81bfd92
Claims without a verdict
Is a verdict here wrong, or do you speak for Sam Altman? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.