Every Token has a Hidden Price

The Five-Minute Demo Hides the Real Question

Getting an open model to talk back takes about five minutes. Pull meta-llama/Llama-3.1-8B-Instruct, run vllm serve, send a request, watch it answer. That part is genuinely easy, and it tells you almost nothing about whether the thing survives contact with real traffic.

The two questions that actually matter tend to show up later, usually at 2 a.m., usually as a page. How many people can this one GPU actually serve at once? And why did the server just fall over with an out-of-memory (OOM) error when nothing about the request looked unusual?

Neither question is a guess. Both come down to one memory calculation you work out once per model, plus a handful of flags that all draw from the same pool of GPU memory in different directions. Once you know that pool and which way each flag pushes it, sizing a vLLM deployment stops being trial and error and starts being arithmetic.

This is the walkthrough I wish someone had handed me before my first production deploy.

Every Request Has Two Phases, and they Compete for Different Resources

Before the memory math, it helps to pull apart what actually happens when vLLM handles a request. Every request runs through two distinct phases, and they load the GPU in opposite ways.

Prefill comes first. The engine reads the whole prompt in a single pass and builds the initial KV cache from it. It’s compute-heavy, dense matrix multiplication across every prompt token at once, and it’s what sets your time-to-first-token (TTFT). A long prompt means a long prefill, and a long prefill means the user waits longer before the first word shows up.

Decode comes after. The engine writes one token at a time, and for each new token it reads back through the entire KV cache built so far. This phase is bound by memory bandwidth rather than compute, the GPU spends most of its cycles moving cached keys and values rather than doing fresh math. Decode sets inter-token latency, the gap between each streamed word, and it’s the phase that keeps the cache growing for as long as generation continues.

Note

Specifically, prefill is usually compute-bound, meaning it’s limited by how fast the GPU can do math. Decode is usually memory-bandwidth-bound, meaning it’s limited by how fast the GPU can move data around.

Long prompts mostly hurt time-to-first-token. Long conversations mostly consume GPU memory. They’re different costs, charged at different times, for different reasons, and mixing them up is a common source of confusion the first time someone tunes a vLLM deployment.

Everything from here on is about the resource decode competes for, i.e., the KV cache.

The KV Cache: The One Number that Decides Capacity

A model writes text one token at a time, and to produce the next token it has to attend back across every token that came before it. Recomputing that attention from scratch at every step would waste an enormous amount of work, since almost none of the earlier tokens have changed. So the engine saves the keys and values it already worked out and reuses them on every later step. That saved state is the KV cache, and it is the single biggest lever on how many concurrent users a GPU can serve.

Model weights are a one-time cost, loaded once and left untouched for the life of the server. The KV cache runs the opposite way. Every active sequence adds to it, and it keeps growing for as long as that sequence keeps generating. Serving is a memory problem before it’s a speed problem, full stop. A fast GPU can sit mostly idle simply because it ran out of room to hold state for more users, long before it ran out of raw compute.

The Formula

Here’s the calculation for how much memory one token costs, per layer, per sequence:

Where:

• $L$ is the number of transformer layers in the model.
• $H_{kv}$ is the number of key-value heads defined by GQA, i.e., grouped-query attention, not the total number of attention heads.^[2] More on why that distinction matters below.
• $D$ is the head dimension, the size of each individual attention head, equal to hidden size divided by the number of attention heads.
• $B$ is bytes per element for whatever dtype the cache is stored in. FP16 and BF16 use 2 bytes. FP8 uses 1 byte.
• The leading 2 accounts for storing two separate tensors per layer per token: the key vector and the value vector.

That formula gives you the cost of one token. Stretch it across an entire sequence and the total KV cache for a sequence of length $T$ is just:

Every growth example later in this piece is nothing more than this equation evaluated at a different $T$.

Warning

This applies to standard multi-head and grouped-query attention, the kind used by Llama, Qwen, Gemma, and Mistral. DeepSeek’s Multi-head Latent Attention, introduced in DeepSeek-V2, projects queries, keys, and values into a much smaller latent vector instead of caching full per-head keys and values, and its authors report cutting KV cache size by more than 90 percent compared to a same-quality standard-attention baseline.^[3] None of that runs through the formula above. If you’re sizing an MLA-based model, go find that architecture’s own numbers instead of reusing this one.

Why the Formula Uses KV Heads, Not Attention Heads

Without grouped-query attention, every attention head computes and stores its own separate keys and values, and $H_{kv}$ just equals the total head count. On a model with 32 or 64 heads, that adds up fast.

GQA breaks that link by letting several query heads share one small set of key-value heads.

Llama 3.1 8B runs 32 query heads against just 8 KV heads, a 4-to-1 grouping.^[4] Queries stay expressive, but the tensors that actually get cached shrink by that same factor of four. It’s the single biggest reason KV cache footprints have dropped across recent model generations, and it’s why $H_{kv}$, not total head count, is the number that belongs in the formula.

A Worked Example: Llama 3.1 8B in FP16

Llama 3.1 8B has 32 layers, 8 KV heads, and a head dimension of 128. In FP16, each element costs 2 bytes.

128 KiB is your basic unit of GPU memory consumption. Every token, in every active sequence, costs roughly that much. Run the per-sequence formula above at $T = 2,000$ and a single request holds around 250 MB of KV cache for its entire lifetime. Multiply by however many sequences are active and you have your total demand at any given moment.

Note

For those not familiar with KiB, it stands for Kibibyte, and it’s equal to 2^10 i.e., 1024 bytes, as opposed to KB which is equal to 1000 bytes. 1 kilobyte is just 1000 bytes and not 1024 bytes!

The Cache Grows While the Model Is Still Talking

It’s tempting to assume only the prompt occupies KV cache and that generation itself is free. It isn’t. Every generated token gets appended to the same cache the prompt tokens landed in.

Average completion length matters just as much as average prompt length here. A chat product where people paste short prompts but get long, detailed answers back will burn through more KV cache per conversation than the prompt length alone suggests.

PagedAttention: How vLLM Actually Stores This Memory

None of the math above holds up if the KV cache has to live in one contiguous allocation per sequence, which is how early inference engines actually did it. Sequences finish at different times and different lengths, and constantly allocating and freeing large contiguous chunks fragments GPU memory, wasting capacity on gaps nothing else can fill.

PagedAttention is the mechanism vLLM was originally built around to fix exactly this.^[1] Instead of one contiguous allocation per sequence, the cache is split into small fixed-size blocks, 16 tokens each by default, and handed out to sequences wherever free blocks happen to sit in GPU memory, the same idea operating systems use for virtual memory paging. When a sequence finishes, its blocks return to the free pool immediately.

That block structure means the real capacity number is slightly less clean than the pure division suggests. The budget gets carved into whole blocks first:

and each sequence rounds its own usage up to the nearest full block. A sequence sitting at 1,001 tokens still reserves a 63-block allocation, 1,008 tokens’ worth, wasting seven tokens of headroom in that last block. With a 16-token block size that’s at most 15 wasted tokens per sequence, small enough to ignore for capacity planning. It only starts to matter if you’re reconciling the exact token count vLLM prints in its startup logs against the back-of-envelope math above and the two don’t line up to the last decimal.

The original paper reports that this fine-grained block allocation nearly eliminates fragmentation waste in the KV cache, which is the whole reason the arithmetic above holds up about as well in production as it does on paper.^[1] Because memory allocates in small blocks instead of large contiguous chunks, vLLM packs far more concurrent sequences onto one GPU than a naive allocator would allow.

What Actually Fills the GPU

GPU memory during serving splits into four pieces:

Weights themselves follow their own small formula:

where $N_{params}$ is the parameter count and $B_w$ is bytes per parameter: 2 for FP16 or BF16, 1 for FP8, roughly 0.5 for INT4. This is the number that shrinks when you quantize weights, and it draws from a completely different pool than the KV cache number above.

vLLM’s --gpu-memory-utilization flag, 0.9 by default in most recent releases though this has moved slightly across versions and is worth confirming against whatever you’re running, sets the ceiling on how much of the card vLLM may touch at all. Inside that ceiling vLLM loads the weights, profiles how much scratch space it needs for activations and CUDA graphs, and hands whatever’s left to the KV cache. You never set the KV cache size directly. It falls out of everything else.

Weight Quantization is a Different Lever Than KV Cache Quantization

The weight math above assumed FP16, 2 bytes per parameter. Plenty of production deployments don’t run FP16 weights at all. AWQ, GPTQ, FP8, and INT4 all shrink $B_w$ substantially, sometimes down to a quarter of its original size or less.

Important

Quantizing your weights does nothing to your KV cache. They’re separate pools controlled by separate flags. Take an 8B model’s weights from roughly 16 GB in FP16 down to 4 GB with INT4 and that freed 12 GB flows straight into the KV budget, since it’s no longer spoken for by weights. But the cache itself sits at whatever precision --kv-cache-dtype specifies, FP16 by default, still 128 KiB per token, until you separately quantize it too. Two different flags, two different problems, and conflating them is one of the more common sizing mistakes I’ve watched people make.

Sizing a Real GPU: H100 80GB Running Llama 3.1 8B

Put the pieces on real hardware. An H100 with 80 GB running Llama 3.1 8B in FP16:

That 52 GB is the memory budget every active sequence draws its KV cache from. It isn’t something you configure, it’s what’s left once weights and overhead come out of the utilization ceiling.

For tighter control, recent vLLM versions expose --kv-cache-memory-bytes, which sets the cache size explicitly instead of deriving it automatically, and overrides --gpu-memory-utilization entirely once set. Most deployments never touch this and just trust the automatic budget. It earns its keep when several engines share one card and each needs a guaranteed slice rather than whatever’s left over.

How Many Requests Can I Actually Serve?

Two different questions hide inside “how many users can this GPU handle,” and conflating them is where most sizing mistakes come from.

Concurrency is a Memory Question

How many sequences can run at once is pure division once you know the KV budget, since the budget is a fixed pool and every token costs the same fixed number of bytes no matter which sequence it belongs to:

Divide that by how many tokens an average conversation runs, prompt plus generated output combined. At 2,000 tokens a conversation, that’s roughly 200 concurrent sequences. Push the average to 8,000 tokens and the same 52 GB only holds about 50. Same GPU, same model, wildly different capacity, because what actually costs memory is tokens in flight, not requests in flight.

Throughput is a Different Question, and Memory Alone Can’t Answer It

Requests per second isn’t something the memory math answers directly. A sequence holds its slice of the cache for as long as it takes to finish, and that duration depends on output length, which varies by request. The memory calculation gives you a ceiling on concurrency, and you find sustainable throughput by testing against that ceiling. vLLM ships a benchmarking tool for exactly this:

1vllm bench serve \
2  --model meta-llama/Llama-3.1-8B-Instruct \
3  --num-prompts 1000 \
4  --request-rate 20

--num-prompts sets total requests sent. --request-rate sets requests per second attempted. Point this at your real deployment with a realistic prompt and output length distribution, and you get the requests-per-second figure the memory math can’t hand you on its own.

Continuous Batching: Why “200 Users” Doesn’t Mean 200 Fixed Slots

Picturing 200 concurrent users as 200 dedicated slots running in lockstep, like 200 parallel threads, is a natural mental model, and it’s wrong. vLLM doesn’t hold a fixed batch of requests until they all finish before starting the next one.

At every decode step the scheduler mixes together whatever tokens are due next across every active sequence, some finishing, some just starting, some in the middle, and runs all of them through the model in one pass. A sequence that finishes frees its slot immediately, and a new request can join on the very next step instead of waiting out a batch cycle. That’s what lets one GPU keep hundreds of independent conversations moving at once without burning compute on padding or idle slots.

The 200-user figure describes how many sequences can hold KV cache space at the same time. It isn’t a batch shape the scheduler is locked into.

A Sequence Isn’t Always the Same Thing as a User

The KV budget math is really counting active sequences, not users, and that distinction matters more than it sounds. Parallel sampling, multiple candidate completions off one prompt, beam search, concurrent tool calls inside a single conversation, all of these spin up more than one sequence for the same user. “200 concurrent users” is shorthand for “200 concurrent sequences,” and if your product generates several sequences per user turn, real user capacity sits below the raw sequence count.

When Requests Share a Prefix, They Can Share Cache Too

Everything so far assumes each sequence needs its own independent KV cache, which is usually true but not always. When a pile of requests share an identical prefix, a common system prompt, a shared set of few-shot examples, the same long document, engines like vLLM can detect the overlap and reuse the already-computed cache for that shared span instead of recomputing it per request. That’s prefix caching, and on workloads with a lot of shared context it changes the effective capacity math above by a wide margin. It’s a big enough topic to earn its own piece, which is where the next part of this series picks up.

Putting the Numbers Together

Before moving into the tuning flags, here’s the whole chain from raw hardware to concurrent users in one place:

Note

Every number in that chain traces back to something already covered - be it hardware, a config flag, an architecture detail, or a traffic assumption. Move any one of them and the concurrency figure moves with it.

The Memory Dials: What Each Setting Actually Trades

Every memory-related flag in vLLM is a dial on that same 52 GB pool.^[5] Which direction each dial pushes is most of what you need to actually operate this thing.

Why FP8 KV Cache Roughly Doubles Capacity

The mechanism is plain arithmetic. Every element costs 2 bytes in FP16 and 1 in FP8, so storing the same token count costs half the memory, and the same 52 GB budget now holds twice as many tokens. Attention tolerates this precision drop reasonably well since the softmax step right after it smooths over small rounding errors, so quality loss tends to be modest for most production workloads. Check it against your own eval set anyway, particularly for anything leaning on precise numerical reasoning, rather than taking that on faith.

When It Breaks: The Field Guide

You’ll run into these eventually. Here’s how to read them.

Preemption specifically deserves more than a reflex fix. It happens because the scheduler let in more sequences than the cache can hold simultaneously, and vLLM’s answer is to evict some and recompute them later rather than crash outright. Recomputation isn’t free, so frequent preemption quietly taxes latency and throughput even while the server looks healthy from the outside. vLLM exposes preemption counts through its Prometheus metrics, and that number is what to watch if you suspect this is happening, rather than waiting for someone to notice slower responses first.^[6]

The Sizing Checklist

Before any deploy, work through these in order:

1. Work out bytes-per-token for your exact model. Use the number of key-value heads, not total attention heads, and confirm your architecture uses standard attention rather than something like MLA.
2. Work out your KV cache budget: GPU memory times utilization, minus weights, minus overhead.
3. Divide by your busiest realistic average context length, prompt plus completion combined, to estimate concurrent sequences.
4. Load test with vllm bench serve against that estimate to find the real requests-per-second you can sustain. Don’t treat the memory ceiling as a throughput number.
5. Decide your fixes ahead of time. FP8 KV cache for more room, a lower utilization ceiling for more stability, tensor parallelism if you need both.

Common Mistakes

Two hundred concurrent users means two hundred fixed execution slots. Not with continuous batching in the picture. Every decode step mixes tokens from many sequences together, and the 200 figure is a ceiling on how many sequences can hold cache space at once, not a batch shape the scheduler is locked into.

Quantizing weights to AWQ or INT4 is sometimes assumed to shrink the KV cache along with them. It doesn’t, not unless KV cache quantization gets turned on separately. Weight precision and cache precision are independent settings, and shrinking weights only frees memory that then flows into the cache budget. The cache itself keeps whatever precision --kv-cache-dtype specifies.

The bytes-per-token formula works for any model. Only for standard multi-head and grouped-query attention. DeepSeek’s Multi-head Latent Attention compresses the KV representation into a latent vector instead of full per-head keys and values, and the arithmetic above simply doesn’t apply there.

Speculative decoding is sometimes pitched as a way to squeeze more concurrent users out of a GPU. It isn’t. It speeds up how fast tokens come out of a sequence that’s already running, using a small draft model to propose tokens the larger model verifies in parallel, but it doesn’t touch how much KV cache that sequence needs while it’s active. Throughput optimization and concurrency optimization are not the same lever.

One user, one KV cache slot, is the assumption until parallel sampling, beam search, or concurrent tool calls enter the picture. Any of those can spin up several sequences under a single user’s request, and each sequence carries its own KV footprint regardless of how many humans are actually attached to it.

Production Notes

A few things the math above doesn’t cover, but that matter once real traffic shows up.

Size against a realistic worst case, not just an average. The concurrency estimate leans on an average context length, and production traffic is never that tidy. A burst of unusually long requests can eat the KV budget faster than the average implies, triggering preemption even when the average-case math looked comfortable. Sizing against something closer to your p95 context length, not the mean, leaves headroom that actually holds up.

Watch KV cache utilization, not raw GPU memory usage. A card that isn’t idle doesn’t tell you how close you are to the preemption threshold. vLLM’s Prometheus metrics expose cache usage percentage, running and waiting request counts, and cumulative preemption counts, and those are the numbers to alert on rather than whatever the driver reports.

Tip

Re-run this math whenever something upstream changes. A GPU swap, a model upgrade, a new --max-model-len, or even a product change that quietly encourages longer conversations, all of these move the inputs into this calculation. Capacity planning here isn’t a one-time exercise done at launch and forgotten. Recheck it against real production request-length distributions periodically.

Scaling past one GPU is a different decision than tuning one GPU. Tensor parallelism (TP) shards weights across GPUs and, in vLLM’s implementation, also shards attention heads, including KV heads, so the per-GPU KV cache cost drops roughly in proportion to the TP degree as well, not just the weight footprint. A model with 8 KV heads split across a TP degree of 4 gives each GPU roughly 2 KV heads’ worth of cache to hold, at the cost of cross-GPU synchronization on every layer. The exact split can look different for mixture-of-experts architectures, so treat that ratio as a first approximation and check your engine’s own startup logs for the number it actually computed. Pipeline parallelism spreads layers instead, trading some added latency for headroom without that synchronization cost. Past that, horizontal scaling with several vLLM instances behind a load balancer is usually easier to reason about than pushing one instance further than its hardware comfortably allows.

Caution

Leave real slack on shared infrastructure. If the GPU isn’t dedicated to this one vLLM instance, don’t push --gpu-memory-utilization to the edge of what the math technically allows. Reserve room for spikes the automatic scratch space estimate never saw coming.

Summary

Serving an open model was never the hard part. Knowing exactly how many concurrent conversations one GPU can hold, and which setting to reach for the moment it can’t hold any more, is what separates a demo from something you’d put a pager rotation behind.

Model weights are a fixed, one-time cost. The KV cache is a variable cost that grows with every active sequence. Prefill fills it for the prompt, decode fills it for everything generated after. Once you know your bytes-per-token number and your KV cache budget, you know your concurrency ceiling, and every flag after that, gpu-memory-utilization, max-model-len, kv-cache-dtype, max-num-seqs, max-num-batched-tokens, is a dial on that same budget, trading memory for throughput, stability, or latency in a specific, predictable direction. Continuous batching is what turns that ceiling into hundreds of conversations moving forward together, instead of 200 requests waiting their turn in a line.

That’s the whole first half of running vLLM well, i.e., — knowing what fits. The second half is what to do once it fits, and that calls for a different set of levers entirely, the kind that make a correctly sized deployment a genuinely fast one. Stay tuned for more.

Thanks for reading. Comments and corrections are welcome.

P.S.: No tokens were harmed in writing this.