Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 74 86 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 4 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
85 74 94 -
Guidance
A company's own figures for its results over a period, checked against what it then reported.
Insufficient record: 0 guidance claims resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 86 claims: 70 factual claims, 16 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Mark Zuckerberg's claims
42 claims
“we didn't prioritize coding because people aren't going to ask Meta AI a lot of coding questions in WhatsApp.”
Mark Zuckerberg, Meta CEO 18 April 2024 Mark Zuckerberg - Llama 3, Open Sourcing $10b Models, & Caesar Augustus (Dwarkesh Podcast)
Meta did not prioritize coding when working on Llama 2.
Resolved true The record shows it happened or was so.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-04-18. True: Meta's documentation, papers or public statements about Llama 2 show that coding was not a training priority, for example little code in the training data, weak coding benchmarks compared with peer models, or statements that coding was deprioritised, with coding handled through the separate Code Llama. Mostly true: coding was clearly a lower priority, but the Llama 2 paper does show some code-focused effort. Mixed: the evidence is ambiguous. Mostly false: Meta's records show coding was a notable priority for Llama 2 but was smaller than other priorities. False: the records show coding was a major priority for Llama 2.
Evidence
- Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv (ar5iv rendering of arXiv:2307.09288; authors are Meta) · 1 June 2023 Own record
“Llama 2 70B is close to GPT-3.5 (OpenAI 2023) on MMLU and GSM8K, but there is a significant gap on coding benchmarks.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models arXiv (ar5iv rendering of arXiv:2307.09288; authors are Meta) · 1 June 2023 Own record
“Llama 2 7B and 30B models outperform MPT models of the corresponding size on all categories besides code benchmarks.”
- LLaMA 1 vs LLaMA 2: A Deep Dive into Meta’s LLMs Ankur's Newsletter · 15 August 2023
“In particular, LLaMA 2’s performance on coding is lacking in comparison to other LLMs.”
- LLaMA 1 vs LLaMA 2: A Deep Dive into Meta’s LLMs Ankur's Newsletter · 15 August 2023
“It also performs better at tasks like reasoning, coding, and proficiency.”
Reasoning
Meta's Llama 2 paper reports that Llama 2 70B comes close to GPT-3.5 on MMLU and GSM8K but shows "a significant gap on coding benchmarks" (E1). It also reports that the 7B and 30B models beat MPT on every category except code (E2). An independent newsletter likewise calls Llama 2's coding performance lacking next to other LLMs (E3). The criteria list weak coding benchmarks against peer models as one way to meet "true", and none of the evidence shows a code-focused effort in Llama 2 that would lower the verdict. E4's remark that Llama 2 does better at coding appears to compare it with Llama 1, so it does not conflict with the other evidence.
Cite this claim
Mark Zuckerberg, “Meta did not prioritize coding when working on Llama 2.” (said 18 April 2024). Verdict: resolved true. Reliability API, claim c_ab29315eccaa, method 0.1.0+3f84d14d. https://epistacklabs.com/claims/c_ab29315eccaa
Claims without a verdict
- Unresolvable 144
- Not yet due 17
- Own records only 41
- Not researched 223
- Not checkable 1,060
Is a verdict here wrong, or do you speak for Mark Zuckerberg? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.