Anthropic does societal impacts research on the short- and medium-term effects of language models.

Judged as of 2022. True: by 2022 Anthropic was doing research on the societal impacts of language models, shown for example by a Societal Impacts team or papers on societal effects or harms (such as 'Predictability and Surprise in Large Generative Models' or related work). The research must be on short- and medium-term effects, as the proposition says. Mostly true: the research was under way but organised informally, or covered mostly one kind of impact. Mixed: there was only incidental work touching on societal impacts. Mostly false: such research had been announced but had not started by 2022. False: the evidence shows Anthropic did no such research in 2022.

Set before the research began.

1. E1 Red teaming language models to reduce harms Anthropic · 22 August 2022 Own record

   “We describe our early efforts to red team language models in order to simultaneously discover, measure, and attempt to reduce their potentially harmful outputs.”

2. E2 Predictability and surprise in large generative models Anthropic · 15 February 2022 Own record

   “We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability.”

E2 is Anthropic's February 2022 paper 'Predictability and Surprise in Large Generative Models', which the criteria name as an example. It examines how large models can lead to socially harmful behavior and reports two new experiments on harms from unpredictability. E1, from August 2022, describes Anthropic's red-teaming work to find, measure and reduce harmful outputs of language models. Both papers deal with harms from language models: E1 with their potentially harmful outputs [E1], and E2 with socially harmful behaviour arising from their unpredictability [E2]. Taken together, these published research outputs show Anthropic was doing societal impacts research on language models in 2022.