Anthropic's experiment found its large model nearly matched a hired human team in changing people's views with persuasive essays.

True: an Anthropic persuasion study published by about April 2024 found that its largest model's persuasive essays changed people's stated views by an amount close to, and not statistically different from, the essays written by the human writers Anthropic hired. Mostly true: the finding is as described, with a minor difference in detail (e.g. the comparison was with crowdworker-written arguments, or the arguments were not strictly 'essays'). Mixed: the model approached human persuasiveness only under some conditions. Mostly false: the study found the model clearly below human persuasiveness, with only a narrow subset close. False: no such Anthropic experiment exists, or it found the model far less persuasive than humans.

Set before the research began.

1. E1 Measuring the persuasiveness of language models Anthropic · 9 April 2024 Own record

   “We also find that our latest and most capable model, Claude 3 Opus, produces arguments that don't statistically differ in their persuasiveness compared to arguments written by humans (Figure 1).”

2. E2 Measuring the persuasiveness of language models Anthropic · 9 April 2024 Own record

   “While the human-written arguments were judged to be the most persuasive, the Claude 3 Opus model achieves a comparable persuasiveness score, with no statistically significant difference.”

3. E3 Measuring the persuasiveness of language models Anthropic · 9 April 2024 Own record

   “For the human-written arguments, we randomly assigned three participants to each claim and asked them to craft a persuasive message of approximately 250 words defending the assigned claim.”

4. E4 Anthropic says its Claude model can change people's minds Axios · 10 April 2024

   “and that the most capable Anthropic model, Claude 3 Opus, "produces arguments that don't statistically differ" from arguments written by humans.”

Anthropic's study, published 2024-04-09, reports that Claude 3 Opus, its most capable model, wrote arguments whose persuasiveness did not differ statistically from human-written arguments (E1, E2). Axios, an independent outlet, reported the same finding from Anthropic's study (E4). However, the human arguments came from study participants, three randomly assigned to each claim, who were asked to write persuasive messages of about 250 words, and the evidence calls them messages rather than essays (E3). This is the kind of minor difference in detail that the criteria place under mostly true.