AI inferencing, like AI training, requires a lot of compute and a lot of storage located close to that compute.

Judge as of 2026-01-28. The claim is that AI inference workloads, like training jobs, need compute and storage provisioned in balanced proportion. True: industry or technical sources (cloud provider architecture documents, analyst or academic work) show that large-scale inference needs storage and memory capacity (for example KV cache, model weights, context and data) scaled with compute, much as training does. Mostly true: this is generally supported, but the ratio or degree differs notably from training. Mixed: sources are split, or it holds only for some kinds of inference (for example agentic or long-context work). Mostly false: inference is shown to be mostly compute- or memory-bandwidth-bound, with little storage dependence, so the parallel is overstated. False: sources show inference has no meaningful storage-to-compute ratio requirement.

Set before the research began.

1. E1 [PDF] Scaling AI Inference with KV Cache Offloading - Samsung Samsung Electronics · 2025

   “Techniques such as model distillation and reasoning optimization help improve efficiency, but they also drive rapid growth in the size of key-value (KV) cache. As context lengths increase and multi-turn interactions become more frequent, GPU memory limitations become a critical bottleneck, making KV cache offloading an essential method for scaling inference workloads.”

2. E2 LLM inference: How it works and how to optimize it Together AI · 14 October 2026

   “That cached attention state is the KV cache: the keys and values for past tokens, stored so decode doesn't recompute them at every step. It grows with every token of every active request, and that memory often caps concurrency before the weights do.”

3. E3 LLM inference: How it works and how to optimize it Together AI · 14 October 2026

   “Training runs forward and backward passes and updates the weights. Inference runs forward passes only, with weights frozen, so there are no gradients or optimizer state to hold in memory.”

4. E4 Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference arXiv (academic authors) · 31 January 2026

   “we reveal that large-batch inference remains memory-bound, with most GPU compute capabilities underutilized due to DRAM bandwidth saturation as the primary bottleneck.”

E1 (Samsung white paper) says KV cache grows with context length and multi-turn use, making GPU memory a bottleneck and KV cache offloading essential to scale inference, and E2 says KV cache memory grows with every active request and often caps concurrency. However, E3 notes inference has no gradients or optimizer state, so its memory profile differs from training, and E4 finds large-batch inference is mainly limited by memory bandwidth with compute underused. E1 links the growth in KV cache to longer contexts and multi-turn use, and E2 says KV cache memory grows with every active request, but E3 notes inference holds no gradients or optimizer state and E4 finds large-batch inference limited mainly by memory bandwidth, so the parallel to training holds only in part.