People outside the AI field are often surprised and alarmed to learn that AI creators do not understand how their AI systems work.

Judge as of 2025-02-18. The claim has two parts: (a) AI developers do not understand how their systems work (in the sense of mechanistic interpretability); (b) people outside the field are often surprised and alarmed to learn this. True: expert sources agree that the inner workings of frontier models are largely not understood, and surveys, polling or reporting show that laypeople commonly hold the opposite belief or react with surprise or concern on learning it. Mostly true: (a) is well supported and (b) is backed by anecdote or commentary rather than systematic data. Mixed: (a) is supported but evidence on (b) is weak or points both ways. Mostly false: (a) is overstated (e.g. substantial understanding exists), or evidence shows that most of the public already knows and is not surprised. False: evidence shows developers do understand how their models work and the public reaction claimed does not occur.

Set before the research began.

1. E1 Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025

   “Despite the rapid performance improvements of the tech industry’s AI models, we still have relatively little idea how these systems arrive at decisions.”

2. E2 Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025

   “For example, OpenAI recently launched new reasoning AI models, o3 and o4-mini, that perform better on some tasks, but also hallucinate more than its other models. The company doesn’t know why it’s happening.”

3. E3 Anthropic CEO wants to open the black box of AI models by 2027 | TechCrunch TechCrunch · 24 April 2025

   “Anthropic is one of the pioneering companies in mechanistic interpretability, a field that aims to open the black box of AI models and understand why they make the decisions they do.”

An independent TechCrunch report says the industry still has "relatively little idea" how AI systems reach decisions, and that OpenAI does not know why its newer reasoning models hallucinate more [E1, E2]. It also describes mechanistic interpretability as a field still trying to open the black box [E3]. This supports part (a), and nothing in the evidence suggests the inner workings of these models are well understood. No item addresses part (b), whether people outside the field are often surprised or alarmed: there are no surveys, polls or reports of public reactions. Under the criteria, (a) supported with weak evidence on (b) resolves as mixed.