Anthropic's Claude 3 Opus reaches roughly 99.9% accuracy on 20-digit addition, well beyond what common internet data would give.

Judge as of 2024-04-12; the claim is hedged ('basically', 'roughly'). True: an Anthropic publication or credible evaluation available by that date shows Claude 3 Opus at about 99.9% accuracy (≥ about 99.5%) on 20-digit addition, and that this is beyond what memorizing common internet data would explain. Mostly true: documented accuracy is very high but somewhat below the figure (about 98–99.5%), or the result is documented but under slightly different conditions (e.g. a different number of digits close to 20). Mixed: high accuracy is documented, but well below 99.9% (about 90–98%), or the evaluation conditions differ materially. Mostly false: there is only anecdotal or unpublished support, or evidence of much lower accuracy, but some real arithmetic ability. False: credible evaluations from that time show Claude 3 Opus clearly failing at 20-digit addition (e.g. below 80% accuracy). If no published measurement exists at all, the claim is unresolvable rather than false.

Set before the research began.

1. E1 Right to Left (R2L) Integer Tokenization beren.io (guest post by Max Buckley, software engineer at Google) · 7 July 2024

   “| Supplier | Model | Default Performance | Default Strategy | Forced R2L 3 digit (With Commas) | Improvement | |---|---|---|---|---|---| | Anthropic | claude-2.1 | 0.3 | BPE | 72.7 | 72.4 | | OpenAI | gpt-3.5-turbo | 35.3 | L2R 3 digit | 85 | 49.7 | | OpenAI | gpt-4o | 89.7 | L2R 3 digit | 96.3 | 6.6 | | Anthropic | claude-3-haiku-20240307 | 94.3 | R2L 3 digit | - | -”

2. E2 Right to Left (R2L) Integer Tokenization beren.io (guest post by Max Buckley, software engineer at Google) · 7 July 2024

   “As it turns out, one of the improvements made in the Claude 3 series is the change to R2L number tokenization. With this change in place Claude 3 and beyond have far superior arithmetic capabilities.”

3. E3 States Hidden in Hidden States: Implicit Discrete State Representations Emerge in LLMs’ Hidden States arXiv (Tsinghua University authors: Junhao Chen, Shengding Hu, Zhiyuan Liu, Maosong Sun) · 31 July 2024

   “We exclude addition involving four-digit or more due to its increased single-step complexity, which complicates tracing implicit computation because of single-step errors.”

4. E4 Right to Left (R2L) Integer Tokenization beren.io (guest post by Max Buckley, software engineer at Google) · 7 July 2024

   “allowing it to learn simple serial algorithms for performing arithmetic operations instead of a vast and arbitrary lookup table.”

5. E5 States Hidden in Hidden States: Implicit Discrete State Representations Emerge in LLMs' Hidden States arXiv (authors from Department of Computer Science and Technology, Tsinghua University)

   “Specifically, we evaluate the exact accuracy of the predicted answers against the ground truth for different models directly performing iterative addition of varying lengths from 2 to 14.”

6. E6 States Hidden in Hidden States: Implicit Discrete State Representations Emerge in LLMs' Hidden States arXiv (authors from Department of Computer Science and Technology, Tsinghua University)

   “It is evident that larger and more advanced closed-source models exhibit higher accuracies emergently.”

None of the evidence gives a published accuracy figure for Claude 3 Opus on 20-digit addition, and none of it dates from on or before 2024-04-12. E1's table lists Claude 3 Haiku and other models, not Opus, and its quote does not say what task or digit length the figures measure. E2 and E4 only say in general terms that Claude 3's arithmetic improved after a tokenization change. E3, E5 and E6 cover iterative addition of other lengths and exclude 20-digit operands. Without a published measurement, the criteria call for a verdict of unresolvable.