Developers can run inference on Llama 3.1 405B on their own infrastructure at roughly 50% the cost of using closed models like GPT-4o, for both user-facing and offline inference tasks.

Judge as of 2024-07-23. The claim is that self-hosted inference on Llama 3.1 405B costs roughly half as much as using closed models like GPT-4o through their APIs. 'Roughly 50%' is met by a cost ratio of about 40–60%. True: cost analyses from that time (Meta's cited source, for example Artificial Analysis, or independent estimates) put self-hosted 405B inference at about 40–60% of GPT-4o's cost per token, for both user-facing and offline or batch use. Mostly true: the ratio is about 30–40% or 60–70%, or it holds clearly for one use (online or offline) and only loosely for the other. Mixed: estimates differ widely, with some supporting about half the cost and others showing parity, or the result depends heavily on assumptions such as utilization or hardware. Mostly false: the saving holds only under narrow assumptions (for example very high utilization of owned hardware), and typical self-hosting costs are close to or above GPT-4o's. False: analyses show self-hosted 405B inference generally costs as much as GPT-4o or more.

Set before the research began.

1. E1 The cost of running Llama 3.1 405B: cloud vs self-hosted, the full math | dataku dataku (independent blog, author "dataku") · 5 August 2024

   “Self-hosted on Lambda Labs: $0.55 per million output tokens vs $15 for GPT-4o. That's 27x cheaper for roughly 95% of the quality (based on my evaluation from last week).”

2. E2 The cost of running Llama 3.1 405B: cloud vs self-hosted, the full math | dataku dataku (independent blog, author "dataku") · 5 August 2024

   “When you add DevOps time and redundancy, the true cost of self-hosting roughly doubles.”

3. E3 Llama 3.1 405B Jeff Wang (Substack) · 25 July 2024

   “If a company is really serious about hosting this model, you’d need to run about 33B of tokens on GPT-4o or 700B tokens on GPT-4o mini to break even, not to mention the extra maintenance cost.”

4. E4 Llama 3.1 405B Jeff Wang (Substack) · 25 July 2024

   “To buy one of these boxes, you’ll need around $250-300k.”

An independent analysis [E1] estimates self-hosted 405B output tokens at $0.55 per million, against $15 for GPT-4o. That is far below GPT-4o's cost, not about 50% of it, though the same author says DevOps time and redundancy roughly double the true self-hosting cost [E2]. Another independent analysis [E3, E4] notes that hardware costs about $250–300k. It says self-hosting only breaks even after about 33B tokens' worth of GPT-4o usage, plus maintenance, so any saving depends heavily on volume and utilization. The estimates differ widely, none clearly supports a ratio of about 40–60%, and the result depends on assumptions, which matches the criteria for mixed.