In 2024, Llama 3 is competitive with the most advanced models and leading in some areas.

Judge as of 2024-07-23 (the Llama 3.1 release, including 405B). True: published benchmark results or independent evaluations from that time (e.g. MMLU, GSM8K, HumanEval, LMSYS Chatbot Arena) show Llama 3/3.1's best model scoring close to the leading frontier models of the time (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro), and ahead of all of them on at least some benchmarks. Mostly true: broadly competitive, but any lead is marginal, comes only from Meta's own reported numbers, or is limited to one or two benchmarks. Mixed: competitive on some major measures but clearly behind on others, with no real lead anywhere. Mostly false: clearly a tier below the leading models on most independent measures, and leading only in narrow or cherry-picked comparisons. False: clearly below the frontier models on essentially all measures, with no area where it leads.

Set before the research began.

1. E1 Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024

   “Llama 3.1 405B beats GPT-4 Turbo on 9 of 10. It beats GPT-4o on 3 of 10 (GSM8K, ARC-Challenge, MGSM).”

2. E2 Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024

   “Against GPT-4o and Claude 3.5 Sonnet: it's competitive but not quite at parity. The gaps on MMLU (87.3 vs 88.7) and HumanEval (89.0 vs 92.0) are small but real.”

3. E3 Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024

   “On my evaluation, Llama 3.1 405B (4.05) trails GPT-4o (4.17) and Claude 3.5 Sonnet (4.33).”

4. E4 AI Visibility & Premium Brand Design | Will Spurlock Will Spurlock (personal/consultancy blog) · 23 July 2024

   “The honest bottom line: Llama 3.1 405B is not uniformly better than GPT-4o. It trails on MATH and GPQA, matches on MMLU and HumanEval, and leads on IFEval.”

One independent blog from the release date calls Llama 3.1 405B competitive with GPT-4o and Claude 3.5 Sonnet but not quite at parity [E2], and a second reports it trailing, matching or leading GPT-4o depending on the benchmark [E4]. E1 reports it ahead of GPT-4o on 3 of 10 benchmarks (GSM8K, ARC-Challenge, MGSM), and E4 reports a lead on IFEval. Both also show it slightly behind on other measures: MMLU and HumanEval in E2, MATH and GPQA in E4, and the author's own overall evaluation in E3. The leads reported in the evidence are over GPT-4o, on a few benchmarks [E1][E4], and the evidence does not show the model leading Claude 3.5 Sonnet or Gemini 1.5 Pro on any benchmark, which matches "mostly true" under the criteria rather than "true".