Claims

Factual · AI safety and alignment

“We don’t have any other guardrails.”

Eliezer Yudkowsky, interviewee · 30 March 2023 · Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript)

Beyond external performance metrics, there are no guardrails for determining what is inside current large AI models.

Mostly true True in substance, wrong in a detail.

The second judge would not resolve this claim; the verdict shown is the first judge's.

What would count as true or false

Judge as of 2023-03-30. True: the evidence (interpretability literature, expert assessments, lab safety documents from around March 2023) shows that, apart from external behavioural or performance evaluations, there were no established, working methods in practice for checking what is going on inside frontier large AI models. Mostly true: interpretability tools existed only as research prototypes that could not meaningfully check the internals of models at GPT-4 scale, so the claim holds in substance and overstates only slightly. Mixed: some internal-inspection methods (e.g. probing, mechanistic interpretability on parts of the model) gave real but limited insight into large models, so 'no other guardrails' is half right. Mostly false: internal-inspection methods were in real use on large models and gave meaningful assurance, even though behavioural evaluation was the main method. False: well-established internal guardrails that do not rely on performance metrics were in routine use on current large models at the time.

Evidence

1. OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023

   “Language models have become more capable and more widely deployed, but we do not understand how they work.”

2. OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023

   “And by automating the interpretation process, OpenAI seeks to overcome the limitations of traditional manual human inspection, which is not scalable for larger neural networks with billions of parameters.”

3. Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching alphaXiv · 25 November 2023

   “Using linear probing and activation patching, we localize five layers that appear especially important for lying.”

4. Probing / Linear Probes Not stated on page (longterm-wiki.vercel.app) · 28 January 2026

   “Azaria and Mitchell (2023) demonstrated that trained classifiers on LLM hidden states achieve 71-83% accuracy at detecting whether statements are true or false”

Reasoning

Reporting from May 2023, about six weeks after the claim, on OpenAI research includes the statement "we do not understand how they work" about language models (E1). The same report describes manual inspection of neural networks as "not scalable for larger neural networks with billions of parameters" (E2). The probing work in E3 (November 2023) and E4 (Azaria and Mitchell, 2023) shows research methods that gave some limited insight into the internals of language models. Both items are from 2023 or later and show research results, not routine checks on frontier models as of March 2023. On that basis, internal-inspection methods were research prototypes, not working guardrails for models at GPT-4 scale, so the claim holds in substance but overstates slightly.

Cite this claim

Eliezer Yudkowsky, “Beyond external performance metrics, there are no guardrails for determining what is inside current large AI models.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_7807589fdfd7, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_7807589fdfd7