Scores by kind of claim
Each kind of claim is scored on its own, out of 100: the share that held up, with the range it likely falls in. Fewer resolved claims give a wider range.
-
Factual claims
Statements about the present or the past, checked against the record: did they hold up?
80 74 86 -
Predictions
Statements about events outside the person's control, checked once they fell due: did they come true?
Insufficient record: 4 predictions resolved, 5 needed for a score.
-
Promises
Commitments to the person's or organisation's own future action, with something to deliver: was it delivered?
85 74 94 -
Guidance
A company's own figures for its results over a period, checked against what it then reported.
Insufficient record: 0 guidance claims resolved, 5 needed for a score.
All types combined: 81 (likely 76 to 86, high confidence), from 86 claims: 70 factual claims, 16 promises. It mixes kinds of claim in different proportions for different people, so compare people kind by kind.
Mark Zuckerberg's claims
5 claims
“Developers can run inference on Llama 3.1 405B on their own infra at roughly 50% the cost of using closed models like GPT-4o, for both user-facing and offline inference tasks.”
Mark Zuckerberg, Meta CEO, writing on Meta Newsroom 23 July 2024 Open Source AI is the Path Forward (Meta Newsroom)
Developers can run inference on Llama 3.1 405B on their own infrastructure at roughly 50% the cost of using closed models like GPT-4o, for both user-facing and offline inference tasks.
Mixed Partly true and partly false in roughly equal measure.
The second judge would not resolve this claim; the verdict shown is the first judge's.
Said for an organisation they lead or work for; it counts on both records.
What would count as true or false
Judge as of 2024-07-23. The claim is that self-hosted inference on Llama 3.1 405B costs roughly half as much as using closed models like GPT-4o through their APIs. 'Roughly 50%' is met by a cost ratio of about 40–60%. True: cost analyses from that time (Meta's cited source, for example Artificial Analysis, or independent estimates) put self-hosted 405B inference at about 40–60% of GPT-4o's cost per token, for both user-facing and offline or batch use. Mostly true: the ratio is about 30–40% or 60–70%, or it holds clearly for one use (online or offline) and only loosely for the other. Mixed: estimates differ widely, with some supporting about half the cost and others showing parity, or the result depends heavily on assumptions such as utilization or hardware. Mostly false: the saving holds only under narrow assumptions (for example very high utilization of owned hardware), and typical self-hosting costs are close to or above GPT-4o's. False: analyses show self-hosted 405B inference generally costs as much as GPT-4o or more.
Evidence
- The cost of running Llama 3.1 405B: cloud vs self-hosted, the full math | dataku dataku (independent blog, author "dataku") · 5 August 2024
“Self-hosted on Lambda Labs: $0.55 per million output tokens vs $15 for GPT-4o. That's 27x cheaper for roughly 95% of the quality (based on my evaluation from last week).”
- The cost of running Llama 3.1 405B: cloud vs self-hosted, the full math | dataku dataku (independent blog, author "dataku") · 5 August 2024
“When you add DevOps time and redundancy, the true cost of self-hosting roughly doubles.”
- Llama 3.1 405B Jeff Wang (Substack) · 25 July 2024
“If a company is really serious about hosting this model, you’d need to run about 33B of tokens on GPT-4o or 700B tokens on GPT-4o mini to break even, not to mention the extra maintenance cost.”
- Llama 3.1 405B Jeff Wang (Substack) · 25 July 2024
“To buy one of these boxes, you’ll need around $250-300k.”
Reasoning
An independent analysis [E1] estimates self-hosted 405B output tokens at $0.55 per million, against $15 for GPT-4o. That is far below GPT-4o's cost, not about 50% of it, though the same author says DevOps time and redundancy roughly double the true self-hosting cost [E2]. Another independent analysis [E3, E4] notes that hardware costs about $250–300k. It says self-hosting only breaks even after about 33B tokens' worth of GPT-4o usage, plus maintenance, so any saving depends heavily on volume and utilization. The estimates differ widely, none clearly supports a ratio of about 40–60%, and the result depends on assumptions, which matches the criteria for mixed.
Cite this claim
Mark Zuckerberg, “Developers can run inference on Llama 3.1 405B on their own infrastructure at roughly 50% the cost of using closed models like GPT-4o, for both user-facing and offline inference tasks.” (said 23 July 2024). Verdict: mixed. Reliability API, claim c_2670b6a066d6, method 0.1.0+3f84d14d. https://epistacklabs.com/claims/c_2670b6a066d6
Claims without a verdict
- Unresolvable 144
- Not yet due 17
- Own records only 41
- Not researched 223
- Not checkable 1,060
Is a verdict here wrong, or do you speak for Mark Zuckerberg? Reply on the claim's own page, with the form at its foot (how replies work). Verdicts are made automatically; how verdicts and scores are made.