OpenAI will discuss in more detail, later in 2023, independent audits of AI systems before release.

True: in 2023, after 24 February, OpenAI published or presented a more detailed discussion of independent audits or external evaluation of AI systems before release (e.g. a blog post, policy paper, testimony or framework addressing external/independent audits or red-teaming before deployment). Mostly true: such a discussion was published in 2023 but dealt with external audits only as one part of a broader governance document. Mostly false: it touched on audits only in passing, or the detailed discussion came only after 2023. False: OpenAI's 2023 publications show no further discussion of independent audits.

Set before the research began.

1. E1 Governance of superintelligence - OpenAI OpenAI · 22 May 2023 Own record

   “any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc.”

2. E2 Governance of superintelligence - OpenAI OpenAI · 22 May 2023 Own record

   “We think it’s important to allow companies and open-source projects to develop models below a significant capability threshold, without the kind of regulation we describe here (including burdensome mechanisms like licenses or audits).”

3. E3 OpenAI’s Approach to Frontier Risk OpenAI · 26 October 2023 Own record

   “We evaluate each new major model released for safety, including using red-teaming. For example, before publicly releasing GPT‑4, external red-teamers tested the model for the following frontier risks”

4. E4 OpenAI’s Approach to Frontier Risk OpenAI · 26 October 2023 Own record

   “This includes undergoing third-party audits of our security program including SOC 2 Type 2.”

OpenAI's May 2023 post "Governance of superintelligence" says that efforts above a capability threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards and place restrictions on deployment [E1]. It also says that audits should not apply to models below that threshold (E1, E2). Its October 2023 "Approach to Frontier Risk" document describes external red-teaming of GPT-4 before public release and third-party audits of OpenAI's security program (E3, E4). These are OpenAI's own 2023 publications, which count as the delivered output. In both, audits and external evaluation are one part of a broader governance or risk document rather than a dedicated, detailed treatment, which the criteria rate as mostly true.