Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 74 86 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 4 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
85 74 94 -
Guidance
A company's own figures for its results over a period, checked against what it then reported.
Insufficient record: 0 guidance claims resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 86 claims: 70 factual claims, 16 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Mark Zuckerberg's claims
40 claims
“This year, Llama 3 is competitive with the most advanced models and leading in some areas.”
Mark Zuckerberg, Meta CEO, writing on Meta Newsroom 23 July 2024 Open Source AI is the Path Forward (Meta Newsroom)
In 2024, Llama 3 is competitive with the most advanced models and leading in some areas.
Mostly true True in substance, wrong in a detail.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-07-23 (the Llama 3.1 release, including 405B). True: published benchmark results or independent evaluations from that time (e.g. MMLU, GSM8K, HumanEval, LMSYS Chatbot Arena) show Llama 3/3.1's best model scoring close to the leading frontier models of the time (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro), and ahead of all of them on at least some benchmarks. Mostly true: broadly competitive, but any lead is marginal, comes only from Meta's own reported numbers, or is limited to one or two benchmarks. Mixed: competitive on some major measures but clearly behind on others, with no real lead anywhere. Mostly false: clearly a tier below the leading models on most independent measures, and leading only in narrow or cherry-picked comparisons. False: clearly below the frontier models on essentially all measures, with no area where it leads.
Evidence
- Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024
“Llama 3.1 405B beats GPT-4 Turbo on 9 of 10. It beats GPT-4o on 3 of 10 (GSM8K, ARC-Challenge, MGSM).”
- Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024
“Against GPT-4o and Claude 3.5 Sonnet: it's competitive but not quite at parity. The gaps on MMLU (87.3 vs 88.7) and HumanEval (89.0 vs 92.0) are small but real.”
- Llama 3.1 405B: the first truly GPT-4 class open model. My benchmark data. | dataku dataku (blog, by an individual author) · 23 July 2024
“On my evaluation, Llama 3.1 405B (4.05) trails GPT-4o (4.17) and Claude 3.5 Sonnet (4.33).”
- AI Visibility & Premium Brand Design | Will Spurlock Will Spurlock (personal/consultancy blog) · 23 July 2024
“The honest bottom line: Llama 3.1 405B is not uniformly better than GPT-4o. It trails on MATH and GPQA, matches on MMLU and HumanEval, and leads on IFEval.”
Reasoning
One independent blog from the release date calls Llama 3.1 405B competitive with GPT-4o and Claude 3.5 Sonnet but not quite at parity [E2], and a second reports it trailing, matching or leading GPT-4o depending on the benchmark [E4]. E1 reports it ahead of GPT-4o on 3 of 10 benchmarks (GSM8K, ARC-Challenge, MGSM), and E4 reports a lead on IFEval. Both also show it slightly behind on other measures: MMLU and HumanEval in E2, MATH and GPQA in E4, and the author's own overall evaluation in E3. The leads reported in the evidence are over GPT-4o, on a few benchmarks [E1][E4], and the evidence does not show the model leading Claude 3.5 Sonnet or Gemini 1.5 Pro on any benchmark, which matches "mostly true" under the criteria rather than "true".
Cite this claim
Mark Zuckerberg, “In 2024, Llama 3 is competitive with the most advanced models and leading in some areas.” (said 23 July 2024). Verdict: mostly true. Reliability API, claim c_851110c128a8, method 0.1.0+3f84d14d. https://epistacklabs.com/claims/c_851110c128a8
Claims without a verdict
- Unresolvable 144
- Not yet due 17
- Own records only 41
- Not researched 223
- Not checkable 1,060
Is a verdict here wrong, or do you speak for Mark Zuckerberg? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.