Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
76 70 82 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
60 42 78
All types combined: 74 (likely 68 to 80, high confidence), from 79 claims: 71 factual claims, 8 predictions. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Eliezer Yudkowsky's claims
46 claims
“We don’t have any other guardrails.”
Eliezer Yudkowsky, interviewee 30 March 2023 Lex Fridman Podcast #368 Eliezer Yudkowsky transcript (Podscript)
Beyond external performance metrics, there are no guardrails for determining what is inside current large AI models.
Mostly true True in substance, wrong in a detail.
The second judge would not resolve this claim; the verdict shown is the first judge's.
What would count as true or false
Judge as of 2023-03-30. True: the evidence (interpretability literature, expert assessments, lab safety documents from around March 2023) shows that, apart from external behavioural or performance evaluations, there were no established, working methods in practice for checking what is going on inside frontier large AI models. Mostly true: interpretability tools existed only as research prototypes that could not meaningfully check the internals of models at GPT-4 scale, so the claim holds in substance and overstates only slightly. Mixed: some internal-inspection methods (e.g. probing, mechanistic interpretability on parts of the model) gave real but limited insight into large models, so 'no other guardrails' is half right. Mostly false: internal-inspection methods were in real use on large models and gave meaningful assurance, even though behavioural evaluation was the main method. False: well-established internal guardrails that do not rely on performance metrics were in routine use on current large models at the time.
Evidence
- OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023
“Language models have become more capable and more widely deployed, but we do not understand how they work.”
- OpenAI peeks into the “black box” of neural networks with new research Ars Technica · 11 May 2023
“And by automating the interpretation process, OpenAI seeks to overcome the limitations of traditional manual human inspection, which is not scalable for larger neural networks with billions of parameters.”
- Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching alphaXiv · 25 November 2023
“Using linear probing and activation patching, we localize five layers that appear especially important for lying.”
- Probing / Linear Probes Not stated on page (longterm-wiki.vercel.app) · 28 January 2026
“Azaria and Mitchell (2023) demonstrated that trained classifiers on LLM hidden states achieve 71-83% accuracy at detecting whether statements are true or false”
Reasoning
Reporting from May 2023, about six weeks after the claim, on OpenAI research includes the statement "we do not understand how they work" about language models (E1). The same report describes manual inspection of neural networks as "not scalable for larger neural networks with billions of parameters" (E2). The probing work in E3 (November 2023) and E4 (Azaria and Mitchell, 2023) shows research methods that gave some limited insight into the internals of language models. Both items are from 2023 or later and show research results, not routine checks on frontier models as of March 2023. On that basis, internal-inspection methods were research prototypes, not working guardrails for models at GPT-4 scale, so the claim holds in substance but overstates slightly.
Cite this claim
Eliezer Yudkowsky, “Beyond external performance metrics, there are no guardrails for determining what is inside current large AI models.” (said 30 March 2023). Verdict: mostly true. Reliability API, claim c_7807589fdfd7, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_7807589fdfd7
Claims without a verdict
Is a verdict here wrong, or do you speak for Eliezer Yudkowsky? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.