How much VRAM an LLM needs: weights, KV cache and concurrent users

The question "which GPU do we need for a local model" is almost always asked backwards: budget first, then a guess at what will fit inside it. Reverse the order. Take the model, the context length and the number of people working at the same time, run them through the NVIDIA formulas, and you get a number in gigabytes.

What the memory budget is made of

NVIDIA describes the memory split in NIM like this: out of the card's full capacity the engine claims a gpu_memory_utilization share, 0.9 by default, and inside that budget it lays out weights, overhead, peak activations and the KV cache. The last one is allocated greedily: "it expands to fill all remaining space within the budget after weights, activations, and overhead are accounted for". So the number of concurrent users is not a setting. It is a remainder.

Weights: a single multiplication

The official formula from the NIM docs: weight memory per card = parameter count × bytes per parameter / TP, where TP is the number of cards the weights are split across by tensor parallelism. Bytes per parameter: two for BF16 and FP16, one for FP8, half a byte for INT4 and NVFP4. INT8 is missing from the official NVIDIA table, so we are not quoting a figure for it.

KV cache, and why a 7B eats more than a 70B

The second half of the bill is the KV cache: the model stores keys and values for every token it has read so it does not have to recompute them. NVIDIA's formula: 2 × number of layers × (number of heads × head dimension) × bytes of precision per token. The leading 2 is K and V counted separately. The volume grows linearly with sequence length and with the number of parallel requests.

Here is the interesting part. The 7B with old multi-head attention spends 512 KB per token, while the 70B with grouped-query attention spends 320 KB: the model is ten times larger and needs 38% less cache. In GQA the key and value heads are grouped, so there are fewer of them than query heads. Which means "how much context will fit" does not follow from the parameter count.

Add the two tables together: a 70B in FP16 with a 128K context is 140 GB of weights plus 40 GB of cache. That is 180 GB for one person.

The overhead you only see in the logs

CUDA graph capture takes, by NVIDIA's own estimate, "1 to 5 GB depending on GPU architecture and model size". On a 24 GB card the default 10% reserve is 2.4 GB, and the documentation admits that on some architectures the graphs will not fit in it. Now the trap that never makes it into a slide deck. It does not apply to every model, but it exists: the NIM support matrix says of the Llama 4 Scout profile that "for the FP8 profile, the same memory as BF16 is required because FP8 quantization happens on the fly, implying that BF16 weights must be loaded into memory first". Where quantization runs on the fly, eight bits save memory during inference, but startup demands as much as BF16. So check the specific profile before you buy, not the FP8 column of that table.

Card, model, context, users

The method reproduces cell by cell: multiply card capacity by 0.9, subtract the weights and another 2 GB for graphs and activations, divide the remainder by the cache of one session. The numbers below are the memory ceiling. Latency will cut them further.

The same cards under a 70B look very different.

Look at the second row. Forty-eight gigabytes technically "hold" a 70B at four bits, but only 6.2 GB survive the weights, which buys two sessions of eight thousand tokens. A single 32K context will not fit at all.

What the real measurements show

Arithmetic gives you the memory ceiling. Latency comes next. NVIDIA's own benchmarks for Llama 3.1 8B on a single H100 80 GB:

Memory alone would have allowed more sessions on an 8B, but at a 20,000-token input the throughput drops eightfold and the first token takes roughly five minutes to appear. The service is technically alive. Nobody will use it.

The most direct lever here is prompt length, and the NIM logs hand you the number: "Estimated VRAM (45.2 GB) exceeds available GPU memory (39.6 GB). Consider reducing context length with --max-model-len=4096 (estimated 30.1 GB)". Cutting the context to 4,096 freed 15.1 GB.

Cache quantization: the claim against the measurement

The logical next step is compressing the cache itself. NVIDIA's measurement compares NVFP4 against FP8, not against FP16: an NVFP4 cache takes roughly half the memory of an FP8 cache, and latency drops by up to three times. The accuracy cost is small: MMLU-PRO 78.2% on BF16, 78.1% on FP8 and 77.4% on NVFP4.

Now a measurement pointing the other way. Nemotron-3-Nano-30B was run under llama.cpp on a DGX Spark with the cache in f16 against q4_0. At 64K tokens the process memory with q4_0 came out not smaller but larger: 2.06 GB versus 1.94 GB, 6% more, because the scaling metadata ate the entire saving on Spark's unified memory. Prompt processing collapsed from 282.7 to 21.3 tokens per second at the same time. Both sources are correct. NVFP4 in TensorRT-LLM is hardware accelerated on Blackwell, while q4_0 in llama.cpp is unpacked in software: the format guarantees nothing, the engine and the hardware deliver the win.

What we have in stock

• RTX PRO 4500 Blackwell, 32 GB: models up to 13B, embeddings and rerankers
• RTX PRO 5000 Blackwell, 48 GB: 8B and 13B with a long context for a team
• RTX PRO 5000 Blackwell, 72 GB: the same card with more memory, a 70B at four bits can breathe
• RTX PRO 6000 Blackwell Workstation Edition, 96 GB: a 70B in FP8 on a single card
• H200 NVL, 141 GB HBM3E: a 70B in FP8 with headroom for long context and dozens of sessions

Common questions

These models are in our catalogue

Graphics Card PNY NVIDIA RTX PRO 4500 BlackwellView in catalogue → Graphics Card PNY NVIDIA RTX PRO 5000 Blackwell 48GBView in catalogue → Graphics Card PNY NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96GBView in catalogue → Graphics Card PNY NVIDIA H200 NVL 141GB HBM3E PCIeView in catalogue →