The KV cache: why long context eats your VRAM
Every token in your conversation costs memory: 128 KiB each for Llama 3.1 8B, so a full 128K context needs 16 GiB of cache. Here is the arithmetic, and the three levers that shrink it.

On this page
In brief
- Every token stores a key and a value vector in every layer: for Llama 3.1 8B that is 128 KiB per token at 16-bit precision.
- At a full 128K context the cache needs 16 GiB, more than the model's own weights and three and a half times its 4.58 GiB Q4_K_M file.
- The size is arithmetic: 2 x layers x KV heads x head dimension x bytes per value x tokens, four numbers from the model's config file.
- llama.cpp 1.0.0 adds FP8 cache types (-ctk/-ctv f8_e4m3); Ollama's OLLAMA_KV_CACHE_TYPE offers q8_0 at about half the memory and q4_0 at about a quarter.
- Grouped-query attention is why the numbers are manageable: Llama 3.1 8B has 32 query heads but only 8 KV heads.
A 4.6 GB model should fit on a 24 GB graphics card with room to spare. So it is a surprise the first time you paste a long document into the chat, watch the memory meter climb past the model, and land on an out-of-memory error. The model file did not grow. The conversation did.
Every token you feed a local model leaves something behind: two small lists of numbers, a key and a value, stored in every layer of the model. Together they are the KV cache, and at long context it can be bigger than the model itself. For Llama 3.1 8B, one token costs 128 KiB of cache at full precision. A full 128K-token context needs 16 GiB for the cache alone, more than the model's own weights and three and a half times the 4.58 GiB Q4_K_M file most people actually download.
What the cache actually stores
A transformer reads text one token at a time, and a token is a chunk of a word, a few characters each. To choose its next word, each token has to look back at the ones before it. Recomputing that look-back from scratch for every new token would mean re-reading the whole conversation each time, so the model takes notes instead: for every token, in every layer, it saves the key it searched with and the value it found.
A layer is one of the model's processing stages, stacked one after another. Llama 3.1 8B has 32 of them, so each token leaves 32 keys and 32 values behind. The key is the label a later token matches against; the value is the information it pulls out. Once the notes are written, generating the next token is mostly a lookup, which is why the first token of a long prompt takes a while and the rest come fast.
You cannot switch the cache off; it is part of how the model works. The only questions are how big it gets and where it lives.
The arithmetic
The size is a straight multiplication, and every number in it comes from the model's config file:
bytes per token = 2 (one key, one value) x layers x KV heads x head dimension x bytes per value
Take Qwen2.5-7B, whose config is public: 28 layers, 4 KV heads, a head dimension of 128, and 2 bytes per value at 16-bit precision. That works out to 57,344 bytes per token, or 56 KiB. At the model's 32,768-token context window, one conversation holds about 1.75 GiB of cache.
Llama 3.1 8B is the more familiar case, and the numbers are starker:
| Context | KV cache at 16-bit |
|---|---|
| 4,096 tokens | 0.5 GiB |
| 32,768 tokens | 4 GiB |
| 131,072 tokens | 16 GiB |
The model's weights, at full 16-bit precision, are about 15 GiB. So at full context the cache is bigger than the model. And almost nobody runs the full-precision weights: the Q4_K_M file is 4.58 GiB, which means a long conversation's cache is three and a half times the size of the model file that supposedly defines the memory footprint. The weights are the price of admission. The conversation is the rent.
Each conversation needs its own cache, so running several chats at once multiplies the bill again.
The head-count trap
The easiest way to get the arithmetic wrong is to use the wrong head count. Llama 3.1 8B has 32 attention heads, but only 8 KV heads. Groups of query heads share a single key/value head, a design called grouped-query attention, and it is the reason the cache is a quarter of what it would be if every head kept its own notes. Plug in 32 heads and the answer comes out four times too big.
The number you want is num_key_value_heads in the model's config.json. The head dimension is usually not listed separately: it is hidden_size divided by num_attention_heads, 4096 divided by 32, or 128, for Llama 3.1 8B, and 3584 divided by 28, also 128, for Qwen2.5-7B.
Three ways to shrink it
The cache does not have to be stored at 16-bit precision. llama.cpp lets you set the key and value types separately, with -ctk and -ctv: q8_0 uses about half the memory of f16, q4_0 about a quarter. Ollama exposes the same choice as a single setting, OLLAMA_KV_CACHE_TYPE, and its FAQ recommends q8_0 if you are not using f16. One condition applies in both: quantized cache needs Flash Attention turned on, OLLAMA_FLASH_ATTENTION=1 in Ollama. And llama.cpp 1.0.0, released in November, added an FP8 option, f8_e4m3, which stores each value in a single byte. Ollama's FAQ adds one warning worth heeding: models with a high grouped-query ratio, Qwen2 named as the example, may lose more precision from cache quantization than others. Try it on a long conversation, not a short one.
The second lever is the context window itself. The cache grows with tokens, so the context setting is a memory setting. In llama.cpp it is -c; in Ollama, OLLAMA_CONTEXT_LENGTH. Cutting a 128K window to 32K quarters the cache, and most chats never come close to filling the window they were given.
The third is letting the cache spill out of VRAM. llama.cpp's server keeps KV offload on by default, so cache that does not fit on the graphics card can sit in system RAM instead. It works, and it is slower: the GPU has to fetch across the bus for every token. Fine for a quiet chat, painful for a busy server.
Some newer architectures attack the problem at the source. DeepSeek's multi-head latent attention compresses the key and value into a smaller shared representation, so the cache grows more slowly with context. That is a model design choice, not a setting, but it is the direction the field is moving: the cache is now a first-class cost, and architectures are starting to be judged on it.
The rule of thumb
Before you raise the context window or download a bigger model, do the multiplication. Budget the cache first and the weights second: a 24 GB card holding a 4.6 GB model at 128K context is really holding a 20.6 GB model, before a single activation is counted. llama.cpp prints the cache size in its startup log, so you can check the arithmetic against the real number the next time you load a model. Do the multiplication first, and the memory meter will stop surprising you.


