AI language models learn to speak fluent Base64, which humans do not typically learn.

Judged as of 2023-08-08. True: published research or demonstrations from before or around that date show that frontier language models (for example GPT-4 or Claude) could read and write Base64 fluently, decoding and encoding it and following instructions given in it, a skill that humans do not normally have without tools. Mostly true: models could handle Base64 well but not fully fluently (frequent errors on longer text), or only the strongest models could do it. Mixed: models showed only partial Base64 ability, for example decoding short strings but failing at encoding or at understanding. Mostly false: Base64 ability was rare, limited to isolated or unreliable demonstrations described as fluent. False: evidence shows that the language models of the time could not meaningfully read or write Base64.

Set before the research began.

1. E1 GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher arXiv (mirrored on huggingface.co, huggingchat/papers-content)

   “recent works show that LLMs exhibit unexpected capabilities in understanding non-natural languages like the Morse Code(Barak, 2023), ROT13, and Base64(Wei et al., 2024).”

2. E2 GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher arXiv (mirrored on huggingface.co, huggingchat/papers-content)

   “These findings imply that while the model primarily relies on character-level cipher priors from pretraining to comprehend ciphers, it can also understand the rules of self-defined word-level ciphers.”

E1 says only that LLMs showed unexpected capabilities in understanding Base64 (citing a 2024 work), without describing how well models decoded or encoded it or followed instructions in it, and E2 concerns ciphers generally, not Base64. The evidence does not establish the level of Base64 skill as of 2023-08-08 that the criteria require to choose among true, mostly true or mixed.