Gary Marcus

Factual · AI safety and alignment

“We have no way of guaranteeing that large language models won’t be jailbroken into saying unethical things.”

Gary Marcus, author of Marcus on AI · 26 October 2023 · A CERN for AI and the Global Governance of AI (Marcus on AI)

There is currently no way to guarantee that large language models won't be jailbroken into saying unethical things.

Resolved true The record shows it happened or was so.

What would count as true or false

Judged as of 2023-10-26. True: the evidence from that time shows that no technique existed that provably or reliably stopped LLMs from being jailbroken into producing harmful or unethical output, and that jailbreaks of leading models were being shown routinely (e.g. the universal adversarial suffixes reported in 2023). Mostly true: the core claim holds, with only minor caveats such as narrow formal guarantees for small or restricted settings. Mixed: some methods offered strong but not complete protection, so whether a 'guarantee' existed was genuinely disputed. Mostly false: robust defences were broadly accepted as effective, and the claim overstates the remaining gaps. False: a method that guarantees protection against jailbreaks was established and accepted by then.

Evidence

1. Researchers Discover New Vulnerability in Large Language Models Carnegie Mellon University (News) · 28 July 2023

   ““Right now, we simply don’t have a convincing way to stop this from happening, so the next step is to figure out how to fix these models,” Fredrikson said.”

2. Researchers Discover New Vulnerability in Large Language Models Carnegie Mellon University (News) · 28 July 2023

   “By training the attack suffix on multiple prompts and models, the researchers have also induced objectionable content in public interfaces like Google Bard and Claud and in open-source LLMs such as Llama 2 Chat, Pythia, Falcon and others.”

3. Jailbroken: How Does LLM Safety Training Fail? arXiv (authors from academic institutions) · 13 April 2023

   “We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models’ red-teaming evaluation sets”

4. AI Safety Newsletter #17 Center for AI Safety · 1 August 2023

   “and the companies have taken preliminary measures to mitigate the attacks. But no robust defenses are currently known, necessitating two key responses to the threat of adversarial attacks.”

Reasoning

In July 2023, CMU researchers reported universal adversarial suffixes that drew objectionable content from Bard, Claude, Llama 2 Chat and other models. One of them said there was no convincing way to stop such attacks (E1, E2). An academic paper found that jailbreaks still worked despite extensive red-teaming and safety training (E3). The Center for AI Safety wrote in August 2023 that no robust defences were known (E4). Together, these independent sources show that as of October 2023 no method could guarantee protection against jailbreaks, and that leading models were being jailbroken routinely.

Cite this claim

Gary Marcus, “There is currently no way to guarantee that large language models won't be jailbroken into saying unethical things.” (said 26 October 2023). Verdict: resolved true. Reliability API, claim c_eeee4d06cd37, method 0.1.0+c876030c. https://epistacklabs.com/claims/c_eeee4d06cd37