How Much GPU Memory an LLM Needs: The Footprint Multiple

A 4B model whose KV cache could not hold one request

How much GPU memory does an LLM need? More than the model card says, and the gap is the KV cache. On 27 November 2024, someone committed a vLLM log to a public Hugging Face Space while trying to serve Sailor-4B-Chat, a 4-billion-parameter model, on a GPU reporting 14.58 GiB, and the weights loaded without trouble:

text

Loading model weights took 7.4150 GBtotal_gpu_memory=14.58GiB ... peak_torch_memory=9.31GiB ... non_torch_memory=0.20GiBkv_cache_size=2.88GiB gpu_memory_utilization=0.85Maximum concurrency for 32768 tokens per request: 0.23xERROR ... The model's max seq len (32768) is larger than the maximum number of tokensthat can be stored in KV cache (7536).

The model fit. The deployment did not. After weights and activations, vLLM had 2.88 GiB left for the key-value (KV) cache, the per-request store of attention keys and values. That holds 7,536 tokens, against the 32,768 the model is configured for. One full-length request needed 12.5 GiB of cache, more than the 12.39 GiB vLLM was allowed to use for everything, so the server reported room for 0.23 of one request and refused to start.

You can check that 12.5 GiB against the model's config.json with no GPU at all: 40 layers, 20 KV heads (Sailor predates grouped-query attention), head dimension 128, 16-bit values. The arithmetic gives exactly the log's 12.5 GiB per request, and its 7,536-token capacity to within one 16-token block, because vLLM allocates the cache in blocks of 16. Nothing here was a bug. And this is the friendly case: vLLM's startup check covers one full-length sequence, so it would have passed a configuration that holds one request and fails at sixteen.

direction: down
budget: "vLLM budget: 14.58 GiB x 0.85 = 12.39 GiB" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 8
  w: "Weights + activation peak\n9.31 GiB" {width: 300; style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  n: "Other\n0.20 GiB" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"}}
  kv: "KV cache\n2.88 GiB\n7,536 tokens" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
need: "One 32,768-token request needs 12.5 GiB of cache\nOnly 0.23 of one request fits" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
budget -> need: "what is left for the cache"

In the diagram, the two halves of the log sit side by side: weights took most of the budget and the cache got the remainder, which is how every managed serving engine allocates memory. That remainder is what the workload consumes, and it never appeared in the decision.

The thesis: parameter count understates memory and overstates speed

Teams size a model with one of two numbers: its parameter count, or its weight size at the precision they plan to serve. Both understate the deployment by the same factor, and parameter count does not determine that factor. I call it the Footprint Multiple:

Footprint Multiple = (weight memory + KV cache at your worst realistic workload) / weight memory

"Worst realistic workload" means the longest context you will accept and the highest concurrency you will accept, together. Runtime overhead is deliberately left out, so the ratio measures only what the workload adds.

For a dense model with full attention you can compute it from three config.json fields. The cache per token is 2 x layers x KV heads x head dimension x 2 bytes at 16-bit. Multiply by context and concurrency, divide by the weight bytes, and add one. Above about 3x, the cache is your deployment. Below about 2x, the weights are.

That product of context and concurrency is an upper bound, and on some traffic a loose one. With prefix caching, a system prompt or tool block shared by all 16 requests is stored once, and real requests are spread across positions rather than all sitting at full length. If your traffic shares a long prefix, count the prefix once and the per-request suffix times concurrency. If you have traffic data, size on the distribution of tokens in flight. Without either, the product is the safe assumption.

The multiple matters in two places. It multiplies the memory the model card implies, and for a dense model it also divides the decode speed the spec-sheet arithmetic implies, because every decode step reads the whole cache as well as the weights. And the usual right-sizing move, quantizing weights to 4 bits, raises the multiple, because the cache stays at 16 bits while the weights shrink.

I am arguing against a consensus my own earlier writing leans on: that a small language model is cheap infrastructure because it is small. A small model is cheap to hold. Whether it is cheap to serve is a property of its KV heads, its layer layout, and your traffic, and the parameter count describes none of them.

Why parameter count is the wrong sizing number

Parameter count correlates with capability, so choosing on it feels safe. It is also the headline number on every model card, and choosing the larger model is easy to defend if it fails. None of that makes it a sizing input. Three different numbers get quoted when someone asks "how big is this model", and each predicts a different thing.

grid-rows: 4
grid-columns: 3
grid-gap: 10
h1: "Number you quote" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
h2: "What it predicts" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
h3: "What it says nothing about" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
r1: "Parameter count\n7B" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
p1: "A rough capability trend" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
m1: "Memory, speed, cost" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
r2: "Weight size at 4-bit\n3.4 GiB" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
p2: "Whether it loads at batch 1" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
m2: "Concurrency, context length" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
r3: "Weights + cache at worst workload\n19.4 GiB" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
p3: "Whether it fits in production\nand its worst-case speed" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
m3: "Quality on your task" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}

Read the grid by row. Only the bottom row predicts whether the deployment runs, and even it says nothing about whether the answers are good. Model cards print the first two rows; the third has to be computed from the architecture and your traffic, which is why it gets skipped.

Grouped-query attention (GQA), where several query heads share one key-value head, is the main reason the third row varies so much between models of similar size. Sailor, the model above, has 20 KV heads. Llama 3.2 3B, a newer model with the same head size, has 8. That one config field is a 2.5x difference in cache per layer, and nothing about it shows up in a parameter count.

The wrong way: size from the weights

This is the check almost every team writes first. It multiplies parameters by bytes per parameter and compares the result with the device:

python

# Wrong: answers "does it load?" and nothing else# (weight_gib is defined in footprint.py, in the next section)def fits_naive(params_b: float, bits: float, device_gib: float) -> bool:    return weight_gib(params_b, bits) < device_gibfits_naive(7.0, 4, 8.0)   # True: 3.4 GiB of weights, "plenty of room"

Follow what happens next. A 7B at 4-bit needs 3.4 GiB of weights, so the team provisions an 8 GiB GPU, and a test at batch 1 and 2k context passes easily. Production then sends 16 concurrent requests at 8k context, and the cache alone needs 16.0 GiB.

direction: down
a: "7B at 4-bit = 3.4 GiB of weights" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
b: "Provision an 8 GiB GPU" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
c: "Test at batch 1, 2k context: passes" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
d: "Production: 16 requests x 8k context" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
e: "Needs 19.4 GiB before overhead" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
a -> b: "1. size from weights"
b -> c: "2. demo regime"
c -> d: "3. real traffic"
d -> e: "4. cache grows 64x"

Step 2 is where the decision goes wrong. Step 3 is where it stays hidden. Every demo runs in the regime where the cache is negligible, so the test confirms the wrong number.

What happens at step 4 depends on the runtime. An unmanaged allocator, such as plain transformers or a hand-rolled server, runs out of memory under load. vLLM behaves differently: it gives the cache whatever gpu_memory_utilization (default 0.92 in v0.31.0) leaves after the weights, then either refuses to start, as in the Sailor log, or preempts and recomputes requests when the cache fills. Your service stays up, and its p95 latency, the 95th-percentile response time, climbs for reasons no weight-size check will explain.

Speed estimates in this pattern fail the same way. Many calculators divide memory bandwidth by weight bytes and report the result as the decode ceiling. That figure is the batch-1, empty-cache ceiling, and production runs neither at batch 1 nor with an empty cache.

The right way: estimate weights, KV cache and the multiple

Fixing this takes a small estimator that reads the shape from config.json, takes your worst workload as input, and reports three things: the weights, the cache, and the Footprint Multiple. It also reports decode speed at the workload rather than at batch 1, and it runs before you download a single weight.

python

# footprint.pyfrom dataclasses import dataclassGIB = 1024**3@dataclass(frozen=True)class Shape:    """Read these from config.json. Parameter counts come from the model card."""    name: str    total_b: float  # billions of parameters held in memory    active_b: float  # billions read per token (equals total_b for dense)    layers: int  # layers whose cache grows with context    kv_heads: int    head_dim: int    window_layers: int = 0  # sliding-window layers, cache capped at window    window: int = 0    state_mib: float = 0.0  # fixed recurrent state per sequence, all layersdef weight_gib(params_b: float, bits: float, group: int = 128) -> float:    # Sub-8-bit formats add a 16-bit scale and a 4-bit zero-point per group.    # params_b should count only the tensors stored at `bits`; most 4-bit    # formats keep embeddings at 16-bit, so add those separately.    bits_per_weight = bits if bits >= 8 else bits + 20 / group    return params_b * 1e9 * bits_per_weight / 8 / GIBdef cache_gib(s: Shape, context: int, batch: int, kv_bits: int = 16) -> float:    per_token_layer = 2 * s.kv_heads * s.head_dim * kv_bits / 8    tokens = s.layers * context + s.window_layers * min(context, s.window)    per_seq = per_token_layer * tokens + s.state_mib * 2**20    return per_seq * batch / GIBdef footprint_multiple(s: Shape, context: int, batch: int, bits: float) -> float:    w = weight_gib(s.total_b, bits)    return (w + cache_gib(s, context, batch)) / wdef decode_ceiling(s: Shape, bandwidth_gib_s: float, bits: float,                   context: int = 0, batch: int = 1) -> float:    """Per-stream tokens/s upper bound. Each step reads active weights    plus the whole batch's KV cache. context=0 gives the batch-1 figure    that spec sheets and many calculators quote. For MoE, active_b is a    batch-1 read: at larger batches tokens pick different experts, so    the result is optimistic."""    read = weight_gib(s.active_b, bits) + cache_gib(s, context, batch)    return bandwidth_gib_s / read

Three details carry weight. weight_gib charges the per-group scale and zero-point that group-wise 4-bit formats such as GPTQ and AWQ (Activation-aware Weight Quantization) store, so "4-bit" costs about 4.16 bits per weight at group size 128. That is the convention this code assumes, and the format matters more than the correction: llama.cpp's Q4_K_M averages 4.89 bits per weight on Llama 3.1 8B, because it keeps some tensors at 6-bit. Set bits to what your format actually stores.

cache_gib uses kv_heads, not the query head count, which is where GQA enters. And decode_ceiling adds the cache to the bytes read per step. Databricks defines achieved memory bandwidth the same way, as "(total model parameter size + KV cache size) / TPOT", where TPOT is time per output token.

direction: down
inputs: "Inputs, all readable before download" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 12
  cfg: "config.json\nlayers, KV heads, head dim" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  card: "Model card\ntotal and active params" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
  wl: "Your worst workload\ncontext, concurrency" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
est: "Weights + KV cache\nFootprint Multiple" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
gates: "Two gates" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 12
  mem: "Memory gate\nweights + cache + overhead <= device" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
  spd: "Speed gate\nworst-case tok/s >= target" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
}
out: "Survivors earn a quality measurement" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
inputs -> est: "1. arithmetic"
est -> gates: "2. check both"
gates -> out: "3. eliminate the rest"

Placing the workload in the input row, next to the architecture, is the whole correction the diagram makes. Without any workload input, the wrong-way check could only answer a question nobody needed answered.

Save that as footprint.py. Now run it against a shortlist of four shapes at one worst workload: 8k context, 16 concurrent requests, 4-bit weights, a 24 GiB device, and 900 GiB/s of bandwidth, a round number chosen for the arithmetic rather than any particular card.

python

from footprint import Shape, weight_gib, cache_gib, footprint_multiple, decode_ceilingCONTEXT, BATCH = 8192, 16   # longest context you accept, highest concurrencyBANDWIDTH = 900.0           # GiB/s, a round number for the arithmeticDEVICE = 24.0               # GiBOVERHEAD = 2.0              # GiB, high end of the runtime workspace bandshortlist = [    Shape("1B dense", 1.0, 1.0, layers=16, kv_heads=8, head_dim=64),    Shape("3B dense", 3.0, 3.0, layers=28, kv_heads=4, head_dim=128),    Shape("7B dense", 7.0, 7.0, layers=32, kv_heads=8, head_dim=128),    Shape("30B-A3B MoE", 30.0, 3.0, layers=48, kv_heads=4, head_dim=128),]print(f"{'shape':<12}{'weights':>8}{'cache':>7}{'multiple':>9}"      f"{'fits':>6}{'tok/s b1':>9}{'tok/s worst':>12}")for s in shortlist:    w = weight_gib(s.total_b, 4)    c = cache_gib(s, CONTEXT, BATCH)    fits = w + c + OVERHEAD <= DEVICE    print(f"{s.name:<12}{w:>8.1f}{c:>7.1f}"          f"{footprint_multiple(s, CONTEXT, BATCH, 4):>8.1f}x"          f"{'yes' if fits else 'no':>6}"          f"{decode_ceiling(s, BANDWIDTH, 4):>9.0f}"          f"{decode_ceiling(s, BANDWIDTH, 4, CONTEXT, BATCH):>12.0f}")

That prints:

text

shape        weights  cache multiple  fits tok/s b1 tok/s worst1B dense         0.5    4.0     9.3x   yes     1860         2013B dense         1.5    7.0     5.8x   yes      620         1067B dense         3.4   16.0     5.7x   yes      266          4630B-A3B MoE     14.5   12.0     1.8x    no      620          67

None of these rows is a specific model. They are representative shapes with every weight stored at 4 bits, and the MoE's worst-case speed is optimistic for a reason the mixture-of-experts section explains. Read your own model's values from its config before you trust any row; the next section shows how much that changes.

The Footprint Multiple, read off a real shortlist

Sort the shortlist by parameter count and you get 1B, 3B, 7B, 30B. Sort it by Footprint Multiple and you get the exact reverse.

grid-rows: 5
grid-columns: 4
grid-gap: 8
h1: "Shape" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
h2: "Weights" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
h3: "Cache, 8k x 16" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
h4: "Footprint Multiple" {style: {fill: "#95A5A6"; stroke: "#6F7F80"; font-color: "#2C2C2A"; bold: true}}
a1: "1B dense" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
a2: "0.5 GiB" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
a3: "4.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
a4: "9.3x" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"; bold: true}}
b1: "3B dense" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
b2: "1.5 GiB" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
b3: "7.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
b4: "5.8x" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"; bold: true}}
c1: "7B dense" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
c2: "3.4 GiB" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
c3: "16.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
c4: "5.7x" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"; bold: true}}
d1: "30B-A3B MoE" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
d2: "14.5 GiB" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
d3: "12.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
d4: "1.8x" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"; bold: true}}

In the grid, the smallest shape carries the largest multiple. Its cache is eight times its weights, so a 1B shape that stores as half a gigabyte needs 4.5 GiB before the runtime takes anything.

Real configs reorder that column. Most GPTQ and AWQ tools quantize only the decoder's linear layers and keep the embedding table at 16 bits, and with a 128,256-token vocabulary the embedding is a large share of a small model. Llama 3.2 1B has 1.24 billion parameters, 263 million of them in its tied embedding. Quantized that way, its weights take 0.96 GiB and its multiple at this workload is 5.2x. Llama 3.2 3B has 8 KV heads rather than the 4 in the representative shape, so its cache is 14.0 GiB against 2.10 GiB of weights, a multiple of 7.7x. Among the real Llama models, the 3B carries the larger multiple. Size did not decide that ordering; the KV head count and the quantization format did.

The relationship is a tendency, not a law. Both the 3B and the 7B land at nearly the same multiple, 5.8x and 5.7x, because the 3B has half the KV heads. What sets the multiple is the cache bytes per token per parameter, an architecture choice that layers, KV heads and head dimension decide together, and parameter count hides all three. That is the reason the multiple needs a name instead of a rule of thumb about size.

Which lever to pull also falls out of the multiple. At 9.3x, the 1B shape's weights are 11% of its footprint, so quantizing them further changes nothing that matters. Cutting the cache does: shorter maximum context, capped concurrency, or an 8-bit cache, which halves the dominant term. At 1.8x, the MoE is the opposite case, because its weights are the problem and no cache work will make it fit.

Why 4-bit quantization raises the Footprint Multiple

Quantization is the standard right-sizing move, and it works on the term it touches. It also changes the ratio between that term and the one it leaves alone. Here is the same 7B at both precisions, under the same workload.

grid-rows: 1
grid-gap: 30
fp16: "7B at 16-bit" {
  label.near: top-center
  grid-rows: 4
  grid-gap: 8
  w: "Weights 13.0 GiB" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  c: "Cache 16.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  m: "Multiple 2.2x" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  s: "Batch 1: 69 tok/s\nWorst case: 31 tok/s" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
q4: "7B at 4-bit" {
  label.near: top-center
  grid-rows: 4
  grid-gap: 8
  w: "Weights 3.4 GiB" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  c: "Cache 16.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  m: "Multiple 5.7x" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  s: "Batch 1: 266 tok/s\nWorst case: 46 tok/s" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}

Compare the two columns cell by cell. The weights drop by 3.8x. The cache does not move. So the multiple rises from 2.2x to 5.7x, and the speed gain splits in two: 4-bit makes the 7B 3.8x faster at batch 1 (69 to 266 tokens per second) and 1.5x faster at 16 requests of 8k context (31 to 46).

Both columns are honest, and only one is printed on the quantization announcement. If your traffic runs near the worst case, you were sold 3.8x and you will see 1.5x. Kim et al. reach a related conclusion from the accuracy side in "Not All Bits Are Equal" (ICLR 2026): for reasoning workloads, the 4-bit prescription fails because "the KV cache rather than model size can dominate memory", and they measure a Qwen3-4B with 2.49 GB of 4-bit weights needing 4.42 GB of cache for "a 32k-token generation".

When the cache dominates, the fix that follows from the multiple is to quantize the cache as well. An 8-bit cache halves the 16.0 GiB term, which lifts the worst-case ceiling from 46 to about 79 tokens per second. That is a bigger speed gain than the 31-to-46 the whole 16-bit-to-4-bit weight change bought, though it saves less memory (8.0 GiB against 9.6 GiB). Whether your quality survives it is a measurement, not an assumption.

The Footprint Multiple is also a decode-speed divisor

Decoding produces one token per step, and each step reads every active weight from memory. That is why decode is limited by memory bandwidth. kipply's Transformer Inference Arithmetic works through the bound in detail. What many calculators then leave out is that the same step also reads the KV cache of every sequence in the batch.

direction: down
step: "One decode step, 7B at 4-bit, 16 requests x 8k" {
  label.near: top-center
  grid-rows: 1
  grid-gap: 12
  w: "Active weights\n3.4 GiB\nread once, shared" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  c: "KV cache\n16 x 1.0 GiB\nread per sequence" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
read: "19.4 GiB read per step" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
out: "900 GiB/s / 19.4 GiB = 46 steps/s\n= 46 tok/s per stream" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
step -> read: "1. bytes per step"
read -> out: "2. divide bandwidth"

Drawn side by side in the diagram, the yellow read is nearly five times the blue one. Divide bandwidth by weights alone and you get 266 tokens per second. Divide by what the step actually reads and you get 46. Those two numbers differ by a factor of 5.7, which is exactly the Footprint Multiple, because for a dense model the bytes read per step and the bytes held in memory are the same two terms.

So one number answers two questions. Memory is the weights times the multiple. Worst-case speed per stream is the batch-1 speed divided by the multiple. Aggregate throughput still rises with batch size, because 16 streams at 46 tokens per second is 736 tokens per second in total. But a product with a per-user latency budget is judged per stream, and per stream the multiple sets the ceiling.

All of this assumes decode is still limited by memory traffic at your batch size. At batch 16 it is. Each weight does 16 multiply-adds per step, about 32 per byte at 4 bits, which is well under the roughly 150 to 300 operations per byte at which an A100 or H100 becomes compute-bound. The cache reads, most of the bytes here, do only a few operations per byte. At much larger batches, the compute cost of unpacking 4-bit weights starts to matter. Steady-state decode is also the only phase the bound covers. Peak memory comes during prefill of the longest prompt, while other requests hold their caches, so size the memory gate for that moment rather than for the average.

This bound also changes what to buy. Memory capacity and bandwidth are sold separately, and at 16 GB the spread is about 3x: the Tesla V100 16GB is rated at 900 GB/s (about 838 GiB/s, so the shortlist's 900 GiB/s is deliberately not this card) and the RTX 4060 Ti 16GB at 288 GB/s. NVIDIA also markets a 554 GB/s "effective" figure for the 4060 Ti, credited to its 32 MB L2 cache. Model weights and KV caches do not fit in 32 MB, so use 288 GB/s for decode. A VRAM figure tells you the model will start. For generation, the bandwidth figure tells you how fast it will answer, and it is the one to shop on.

MoE memory vs speed: mixture-of-experts models send two bills

A mixture-of-experts (MoE) model routes each token through a few of its experts. Every expert must stay resident, so memory is charged on total parameters. Only the selected experts are read per step, so speed is charged on active parameters, and the 30B-total, 3B-active shape from the table makes the split concrete.

grid-rows: 1
grid-gap: 30
mem: "Memory bill" {
  label.near: top-center
  grid-rows: 3
  grid-gap: 8
  a: "All 30B params resident\n14.5 GiB" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  b: "Cache 12.0 GiB" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  c: "27.0-28.5 GiB with overhead\ndoes not fit 24 GiB" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
}
spd: "Speed bill" {
  label.near: top-center
  grid-rows: 3
  grid-gap: 8
  a: "3B active params read\n1.5 GiB at batch 1" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
  b: "Cache 12.0 GiB read" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
  c: "Batch 1: 620 tok/s\nWorst case: at most 67 tok/s" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
}

In the left column, 14.5 GiB of weights makes any cache look small, which is why the MoE has the smallest Footprint Multiple in the table. This MoE is also the only shape that fails the memory gate. Its weights alone take 14.5 of the 24 GiB, so the model card was enough to reject it.

A popular claim falls apart in the right column. "It decodes at the speed of a 3B" is true at batch 1, where both shapes reach 620 tokens per second. Under load it fails for two reasons. The MoE has 48 layers of cache to read against the 3B's 28, so even the code's figure drops to 67 against the 3B's 106. And the code charges the MoE's weight read at batch 1, which is wrong at batch 16: each of the 16 tokens routes to its own experts, so one step reads the union of everything they selected.

Qwen3-30B-A3B has exactly this shape, with 128 experts per layer and 8 active per token. Under uniform routing, 16 tokens touch 1 - (120/128)^16 of the experts, about 64%, in every layer. That is about 9.6 GiB of weights read per step instead of 1.5, and a worst-case ceiling near 42 tokens per second. Real routing favours popular experts, so the true figure lies between 42 and 67. Either way the MoE is slower per stream than the dense 3B it is said to match: its memory multiple is the smallest in the table, and its speed penalty under load is among the largest.

Where memory is plentiful, as on an 80 GB server card, the MoE gives you a large model's knowledge at a small model's per-token read, at least at low concurrency. Where memory is the binding limit, the low active count is irrelevant, because the model never loads.

Hybrid and sliding-window models change the multiple's shape

Everything so far assumed every layer runs full attention, so the cache grows with every token in every layer. Several small-model families released since 2025 break that assumption, and the multiple changes shape with them:

Gemma 3 alternates "5 local layers for every global layer". The local layers attend over a 1,024-token window in the 4B and larger models, and 512 in the 1B.
Qwen3.5-4B runs exactly 3 Gated DeltaNet linear-attention layers for every full-attention layer, 24 of its 32 layers, per its config.json.
Granite 4.0 H lists "4 attention / 36 Mamba2" layers in IBM's model card.
Nemotron Nano 2 9B has 4 attention layers in 56; the rest are Mamba-2 and multi-layer perceptron (MLP) layers.

grid-rows: 1
grid-gap: 16
full: "Full attention" {
  label.near: top-center
  grid-rows: 2
  grid-gap: 8
  a: "Caches every token" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
  b: "Grows with context\nand concurrency" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
}
win: "Sliding window" {
  label.near: top-center
  grid-rows: 2
  grid-gap: 8
  a: "Caches the last\n1,024 tokens" {style: {fill: "#FFA07A"; stroke: "#D9785A"; font-color: "#2C2C2A"}}
  b: "Stops growing\nat the window" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}
rec: "Linear attention / Mamba-2" {
  label.near: top-center
  grid-rows: 2
  grid-gap: 8
  a: "Fixed state\nabout 2 MiB per layer" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
  b: "Same size at any\ncontext length" {style: {fill: "#98D8C8"; stroke: "#5FA898"; font-color: "#2C2C2A"}}
}

Only the red column grows without limit; the other two are bounded, so a model built mostly from them pays something closer to a fixed charge per request. For Qwen3.5-4B the 2 MiB figure is exact for the recurrent state: 32 value heads of 128 by 128 in 32-bit floats.

Here is one 4B-class shape (36 layers, 8 KV heads, head dimension 128, 1.9 GiB of 4-bit weights) at 16 concurrent requests, under four layer layouts. Each layout maps onto the Shape fields above: full-attention layers in layers, window layers in window_layers, and recurrent state in state_mib.

Layout	Cache at 8k	Multiple at 8k	Cache at 32k	Multiple at 32k
All full attention	18.0 GiB	10.3x	72.0 GiB	38.2x
5:1 sliding:full, 1,024-token window	4.9 GiB	3.5x	13.9 GiB	8.2x
3:1 linear:full, 2 MiB state per linear layer	5.3 GiB	3.8x	18.8 GiB	10.7x
4 attention + 32 recurrent, 2 MiB state each	3.0 GiB	2.6x	9.0 GiB	5.7x

The layouts are representative of the families above and match no single model's config. Charging every layer as full attention overstates the cache of the other three by 3x to 8x, and the gap widens with context, because only the first row grows at the full rate.

Two practical consequences follow. On a hybrid, concurrency becomes the axis you size against, because context length matters much less. And your estimator has to read the layer layout, which the configs do not store the same way: Qwen3.5 lists layer_types under a nested text_config, Nemotron encodes it in a hybrid_override_pattern string, and the shipped Gemma 3 configs give a sliding_window_pattern instead. A parser that misses those falls back to all-full-attention, which for Nemotron overstates the cache about 14x and rejects a model that fits.

Arithmetic eliminates candidates. It never selects one.

Of the four bounds a deployment must clear at once, the estimator decides two: memory and speed. Quality and cost per request are absent. Quality can only be measured by running candidates on examples from your own product, which is why choosing a model is a systems decision and not a benchmark lookup. Cost depends on throughput and utilization on rented hardware, and was fixed at purchase on owned hardware. Here is what the arithmetic decides for a single 24 GiB GPU, 8k context, 16 concurrent requests, and a product that needs 30 tokens per second per stream.

grid-rows: 4
grid-columns: 2
grid-gap: 24
q1: "1. Weights + cache + overhead\nfit in 24 GiB?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a1: "MoE out at 28.5 GiB\n1B, 3B, 7B survive" {style: {fill: "#E74C3C"; stroke: "#B03A2E"; font-color: "#FFFFFF"}}
q2: "2. Worst-case tok/s >= 30\nper stream?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a2: "All three pass\n7B only by 1.5x (46 tok/s)" {style: {fill: "#FFD93D"; stroke: "#C9A800"; font-color: "#2C2C2A"}}
q3: "3. Start with the smallest\ncredible survivor" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a3: "Measure the 3B on\nyour own task slices" {style: {fill: "#4A90E2"; stroke: "#2C6FB0"; font-color: "#FFFFFF"}}
q4: "4. 3B misses the quality bar?" {style: {fill: "#7B68EE"; stroke: "#5A4BC4"; font-color: "#FFFFFF"}}
a4: "Try the 7B, with a hard cap\non context and concurrency" {style: {fill: "#6BCF7F"; stroke: "#3F9E52"; font-color: "#2C2C2A"}}
q1 -> q2
q2 -> q3
q3 -> q4

The first gate removes the MoE. The second removes nobody, but it changes the 7B's status. With a batch-1 calculator, the 7B looks like it clears 30 tokens per second by 9x. With the cache counted, it clears by 1.5x, and any growth in context or concurrency eats that margin. Its memory headroom is also thin: 2.6 GiB against the high end of its range on the full card, and only 0.7 GiB on vLLM, which by default budgets 92% of it (22.1 GiB). Each extra 8k request costs 1.0 GiB.

So the 3B goes first. Its pessimistic total is 10.5 GiB, which leaves 13.5 GiB free on the card, and it holds 106 tokens per second at the worst case. On most real tasks the 1B is not a credible first candidate, even though it is cheaper still, and that judgment is the part arithmetic cannot make. Swap in the 7B only on two conditions: the 3B falls short of your quality bar and the 7B clears it on the same evaluation slices, and you accept the cap on context and concurrency. Passing the memory and speed gates gets a model onto the test bench, not into production. What you build around the survivor is a separate problem, covered in design patterns for SLM-first systems.

What is already known, and what the Footprint Multiple adds

Others have already published the observation that small models carry proportionally larger caches. Sadhukhan et al. state it directly in Kinetics (NeurIPS 2025): "Smaller models tend to require significantly more KV cache relative to their parameter size." Their example is a Qwen3-0.6B needing 3.5 GB of cache for 32K tokens against 1.2 GB of weights, while Qwen3-32B needs 8 GB. Their argument is about the cost of test-time scaling, and they conclude that small models' efficiency is overestimated. Kim et al., quoted above, show the same cache dominance for 4-bit reasoning models, from the accuracy side.

IBM's Granite 4.0 announcement makes the industry version of the point: memory requirements "are often reported in terms of how much RAM is needed just to load up model weights", while enterprise workloads need long context, concurrency, or both. That is a vendor making the case for its hybrid architecture, so treat its 70% savings figure as a claim. On the serving side, the original measurement came from the PagedAttention paper: about 65% of memory for weights and close to 30% for request state, on a 13B model on a 40 GB A100.

The Footprint Multiple packages those findings into one deployment number and adds three things. It is a single ratio taken at the worst workload, used to eliminate candidates before download, and it turns the quantization effect Kim et al. measured into a number you can compute for any config. For dense models it equals the worst-case speed divisor, so one number answers both the memory question and the latency question. And it exposes the MoE and hybrid cases as different in kind: a small multiple from huge weights in one case, and a cache that stops scaling with context in the other.

For the large-model version of this sizing problem, with real 70B configs, vLLM's memory reserve and a per-GPU request count, see the KV-first sizing script in the inference engine guide.

Footprint Multiple checklist for your next model choice

Write down the worst workload first. Longest context you will accept, highest concurrency you will accept. If nobody can name them, that is the first finding.
Read the shape from config.json, not from the model name. Layers, num_key_value_heads (fall back to the attention head count if it is missing, or you will understate the cache by the head ratio), head dimension, and the layer layout for hybrids. Take parameter counts from the model card, and check which tensors your format leaves at 16 bits (usually the embeddings). A "7B" is often 7.6B, and a "1B" can be 1.24B.
Compute the Footprint Multiple for every candidate. Above about 3x, the cache is your deployment, and cache levers (context cap, concurrency cap, 8-bit cache) beat weight levers. Below about 2x, the weights are, and quantization or a smaller model is the lever.
Quote decode speed at the worst workload. For dense models, divide the batch-1, empty-cache figure by the multiple. For MoE, dividing by (active weights + cache) / active weights gives an optimistic bound, because at real batch sizes a step reads many more experts than one token selects. Compare the result with your per-stream latency target.
Judge memory against the high end of a range. Add 0.5 to 2 GiB of runtime workspace and compare the pessimistic total with usable memory, which on vLLM is the device times gpu_memory_utilization. Underestimating memory is the expensive direction.
Re-check after any quantization change. Moving weights from 16-bit to 4-bit raised the 7B's multiple from 2.2x to 5.7x. The model got cheaper to hold and relatively more expensive to serve.
Use the arithmetic to eliminate, then measure. The smallest credible survivor gets measured first on your own evaluation slices. A larger survivor displaces it only by closing a quality gap the smaller one leaves open.
Decide what happens past the worst case. Rejection, truncation, or queueing. On vLLM the default answer is preemption, which shows up as latency. A system with no stated answer has accepted whatever the runtime does.

Parameter count is the number on the model card, and the 4-bit weight size is the number in the quantization announcement. Neither one contains your traffic, so both understate the deployment by the same factor: the Footprint Multiple. It is largest exactly where right-sizing pushes you, toward small dense models with quantized weights. Compute it before you download anything, and let it decide which lever you pull.

This article is adapted from Chapter 1 of my book Right-Sized AI: Small Language Models in Production, which builds the full estimator and its tests.