Beyond external performance metrics, there are no guardrails for determining what is inside current large AI models.

Judge as of 2023-03-30. True: the evidence (interpretability literature, expert assessments, lab safety documents from around March 2023) shows that, apart from external behavioural or performance evaluations, there were no established, working methods in practice for checking what is going on inside frontier large AI models. Mostly true: interpretability tools existed only as research prototypes that could not meaningfully check the internals of models at GPT-4 scale, so the claim holds in substance and overstates only slightly. Mixed: some internal-inspection methods (e.g. probing, mechanistic interpretability on parts of the model) gave real but limited insight into large models, so 'no other guardrails' is half right. Mostly false: internal-inspection methods were in real use on large models and gave meaningful assurance, even though behavioural evaluation was the main method. False: well-established internal guardrails that do not rely on performance metrics were in routine use on current large models at the time.

Set before the research began.

1. E1 OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023

   “Language models have become more capable and more widely deployed, but we do not understand how they work.”

2. E2 OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023

   “And by automating the interpretation process, OpenAI seeks to overcome the limitations of traditional manual human inspection, which is not scalable for larger neural networks with billions of parameters.”

3. E3 Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching alphaXiv · 25 November 2023

   “Using linear probing and activation patching, we localize five layers that appear especially important for lying.”

4. E4 Probing / Linear Probes Not stated on page (longterm-wiki.vercel.app) · 28 January 2026

   “Azaria and Mitchell (2023) demonstrated that trained classifiers on LLM hidden states achieve 71-83% accuracy at detecting whether statements are true or false”

Reporting from May 2023, about six weeks after the claim, on OpenAI research includes the statement "we do not understand how they work" about language models (E1). The same report describes manual inspection of neural networks as "not scalable for larger neural networks with billions of parameters" (E2). The probing work in E3 (November 2023) and E4 (Azaria and Mitchell, 2023) shows research methods that gave some limited insight into the internals of language models. Both items are from 2023 or later and show research results, not routine checks on frontier models as of March 2023. On that basis, internal-inspection methods were research prototypes, not working guardrails for models at GPT-4 scale, so the claim holds in substance but overstates slightly.