The economics of large-model inference on GPUs

Most teams size their inference hardware by looking at peak FLOPs on a spec sheet. Then they deploy, watch utilization sit at 20%, and wonder where the money went. The gap is almost always the same: large-model inference is bound by memory, not compute. Understanding why is the difference between a serving stack that prints tokens cheaply and one that burns a GPU to idle.

Why decoding is memory-bound

Autoregressive generation produces one token at a time. To produce each token, the GPU has to read the entire set of model weights out of high-bandwidth memory (HBM) and stream them through the compute units. At batch size one, almost no arithmetic is reused between elements — the arithmetic intensity is low, so the hardware spends most of its time waiting on memory rather than doing math.

The practical consequence: the ceiling on single-stream decode throughput is roughly HBM bandwidth ÷ model size in bytes. A GPU with twice the FLOPs but the same bandwidth will not decode meaningfully faster for one request. This is why memory bandwidth, not TFLOPs, is the first number to look at when you evaluate a chip for serving.

Prefill — processing the prompt — is the exception. It runs over many tokens at once, so it is compute-bound and looks great on benchmarks. Real workloads are a mix, and the decode phase usually dominates cost for chat-style traffic.

The KV cache is a second tenant in VRAM

Every token the model has already seen leaves behind key and value tensors that must be kept around so future tokens can attend to them. That KV cache grows linearly with batch size, sequence length, layer count and hidden size. It lives in the same VRAM as the weights, and it is frequently the thing that runs you out of memory — not the model itself.

Two things fall out of this:

• Long contexts are expensive in memory, not just compute. Doubling your context window can double KV footprint per request.
• Concurrency is capped by KV, not weights. Once the weights are resident, how many simultaneous requests you can hold is a function of how much VRAM is left for KV cache.

Batching turns idle bandwidth into throughput

If a single stream leaves the memory bus underused, the fix is to serve many streams at once. With enough concurrent requests, each weight read from HBM is amortized across many sequences, and the GPU shifts from memory-bound toward compute-bound — exactly where it is efficient.

Modern servers do this with continuous (in-flight) batching: requests join and leave the batch token by token instead of waiting for a fixed group to finish. The trade-off is latency versus throughput — bigger batches raise tokens-per-second for the fleet but add queueing delay for any one user. Your service-level target for time-to-first-token and inter-token latency sets how far you can push it.

What actually moves cost per token

Putting it together, the levers that change your unit economics are:

• Model size in bytes — the dominant term. Halving precision roughly halves the per-token memory traffic.
• Quantization — FP8 or INT8 weights cut both VRAM footprint and bandwidth pressure, usually with small, measurable quality loss on well-tested models.
• Effective batch size — driven by how much VRAM is free for KV cache after weights load.
• Context length — longer prompts inflate KV cache and prefill cost.
• Utilization — a half-idle GPU has double the cost per token of a busy one, whatever the sticker rate.

Notice that four of those five are memory stories.

Sizing the box

A useful mental model: VRAM must hold the weights, plus enough KV cache to reach a batch size that keeps the memory bus busy. If weights alone nearly fill the card, you will never batch enough to be efficient, and your cost per token stays high no matter how cheap the hardware looked per hour.

That is the case for keeping model shards inside a single node connected by high-bandwidth NVLink rather than splitting them across a slower network link. When a model is too large for one GPU, tensor parallelism across GPUs in the same node keeps the inter-GPU traffic on the fast fabric; crossing a node boundary for every layer is a good way to reintroduce the bandwidth wall you were trying to escape.

On TheAI Cloud you rent whole nodes — all eight GPUs and their full NVLink domain — so weights and KV cache have room to breathe and tensor-parallel shards stay on the fast interconnect. Live rates are on the pricing section; the docs cover images, SSH and data.

FAQ

Is a bigger GPU always cheaper per token?

No. If decode is memory-bound, per-token throughput tracks memory bandwidth and model size, not FLOPs. A pricier, higher-FLOP card only wins if you can keep it busy with large batches or if it also brings more bandwidth and VRAM. Measure tokens per second per dollar on your actual model before deciding.

How much VRAM do I need for inference?

Enough for the model weights in your chosen precision, plus KV cache for your target concurrency and context length, plus working overhead. The weights set the floor; the KV cache sets how many requests you can serve at once. Running out of memory under load is almost always the KV cache, not the weights.

Does quantization hurt quality?

FP8 and INT8 weight quantization typically cause small, measurable quality changes that are acceptable for most production workloads, and they cut both VRAM use and memory traffic. The right answer is empirical: evaluate the quantized model on your own tasks rather than trusting a generic benchmark.

Why is my GPU utilization low during inference?

Usually because you are decoding at small batch size, so the GPU is stalled waiting on HBM instead of computing. Continuous batching and higher concurrency raise utilization. Persistent low utilization on a whole node also points to a model that fits comfortably in memory with room to serve far more traffic than you are sending it.