Meta did not prioritize coding when working on Llama 2.

Judge as of 2024-04-18. True: Meta's documentation, papers or public statements about Llama 2 show that coding was not a training priority, for example little code in the training data, weak coding benchmarks compared with peer models, or statements that coding was deprioritised, with coding handled through the separate Code Llama. Mostly true: coding was clearly a lower priority, but the Llama 2 paper does show some code-focused effort. Mixed: the evidence is ambiguous. Mostly false: Meta's records show coding was a notable priority for Llama 2 but was smaller than other priorities. False: the records show coding was a major priority for Llama 2.

Set before the research began.

1. E1 Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv (ar5iv rendering of arXiv:2307.09288; authors are Meta) · 1 June 2023 Own record

   “Llama 2 70B is close to GPT-3.5 (OpenAI 2023) on MMLU and GSM8K, but there is a significant gap on coding benchmarks.”

2. E2 Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv (ar5iv rendering of arXiv:2307.09288; authors are Meta) · 1 June 2023 Own record

   “Llama 2 7B and 30B models outperform MPT models of the corresponding size on all categories besides code benchmarks.”

3. E3 LLaMA 1 vs LLaMA 2: A Deep Dive into Meta’s LLMs Ankur's Newsletter · 15 August 2023

   “In particular, LLaMA 2’s performance on coding is lacking in comparison to other LLMs.”

4. E4 LLaMA 1 vs LLaMA 2: A Deep Dive into Meta’s LLMs Ankur's Newsletter · 15 August 2023

   “It also performs better at tasks like reasoning, coding, and proficiency.”

Meta's Llama 2 paper reports that Llama 2 70B comes close to GPT-3.5 on MMLU and GSM8K but shows "a significant gap on coding benchmarks" (E1). It also reports that the 7B and 30B models beat MPT on every category except code (E2). An independent newsletter likewise calls Llama 2's coding performance lacking next to other LLMs (E3). The criteria list weak coding benchmarks against peer models as one way to meet "true", and none of the evidence shows a code-focused effort in Llama 2 that would lower the verdict. E4's remark that Llama 2 does better at coding appears to compare it with Llama 1, so it does not conflict with the other evidence.