AI alignment between human wants and machine behavior will remain a critical unsolved problem, with GPT-4 unable to reliably follow shared human values.

Judge against GPT-4 as released (March 2023) and the state of the field during 2023. True: through 2023 alignment remained widely recognised as an unsolved, critical problem, and GPT-4 demonstrably failed to reliably follow human values and intentions, e.g. jailbreaks, harmful outputs, documented misbehaviour in OpenAI's system card or independent red-teaming. Mostly true: both parts hold, but GPT-4's failures were notably less frequent than the claim implies (e.g. it mostly behaved well, with failures mainly under adversarial prompting). Mixed: one part holds and the other does not, e.g. alignment remained unsolved in general but GPT-4 was shown to follow shared human values reliably in practice. Mostly false: GPT-4 was broadly reliable in following human values with only rare, narrow failures, and experts considered practical alignment largely achieved for such models. False: alignment was widely regarded as solved and GPT-4 reliably followed human values.

Set before the research began.

1. E1 GPT-4 System Card - cdn.openai.com OpenAI

   “Training on this data improves model robustness but does not fully solve the problem of “jailbreaks” leading to harmful content.”

2. E2 GPT-4 System Card - cdn.openai.com OpenAI

   “In most cases, GPT-4-launch exhibits much safer behavior due to the safety mitigations we applied.”

3. E3 Model alignment protects against accidental harms, not intentional ones normaltech.ai (Arvind Narayanan, Sayash Kapoor, Seth Lazar) · 1 December 2023

   “RLHF has very substantially solved this problem, and by solving it OpenAI created a multi-billion dollar industry where one didn't exist before.”

4. E4 Model alignment protects against accidental harms, not intentional ones normaltech.ai (Arvind Narayanan, Sayash Kapoor, Seth Lazar) · 1 December 2023

   “A wave of recent research has shown that RLHF is brittle.”

OpenAI's GPT-4 System Card says its safety training "does not fully solve the problem of 'jailbreaks' leading to harmful content" (E1), and later independent analysis calls RLHF "brittle" (E4). Both support the claim that alignment stayed an unsolved problem. However, the same system card says the launch version of GPT-4 showed "much safer behavior" in most cases (E2), and Narayanan, Kapoor and Lazar write that RLHF "very substantially solved" the problem of accidental harms (E3). The failures therefore appear mainly under adversarial prompting rather than in ordinary use, which matches the criteria's description of "mostly true".