Model quantization, explained: what Q4_K_M actually costs you
Q4_K_M shrinks a model to under a third of its size for a quality loss you can barely measure. The name is a spec sheet — here's how to read it, and when to pick a different level.

On this page
In brief
- Q4_K_M stores weights at about 4.89 bits each — Llama 3.1 8B shrinks from 14.96 GiB at 16-bit to 4.58 GiB, and generates 72 tokens per second instead of 29.
- The name is a recipe: Q is the bit count, K marks the 2023 k-quants, S/M/L say how many sensitive tensors get extra bits.
- Quality loss is tiny down to 4 bits — a January 2026 study scored Llama 3.1 8B Instruct 69.47 at 16-bit and 69.15 at Q4_K_M — and falls off a cliff below 3 bits.
- A bigger model at lower bits usually beats a smaller model at higher bits; a 30B model at Q2_K out-scored a 13B model at full precision.
- The KV cache is separate memory that grows with context — quantize it with -ctk/-ctv in llama.cpp or OLLAMA_KV_CACHE_TYPE in Ollama.
Every local model you have ever downloaded came with a strange suffix: Q4_K_M, Q8_0, IQ3_XXS. It is actually a spec sheet — how much of the model was thrown away, how big the file is, how much quality you gave up. Q4_K_M is the default download in Ollama, llama.cpp and LM Studio: it shrinks a model to under a third of its 16-bit size, runs about two and a half times faster, and costs so little quality that most benchmarks can barely find it.
What quantization actually does
A model is mostly numbers — billions of weights, each a 16-bit float. Quantization re-stores those weights with fewer bits: 8, 6, 5, 4, sometimes 3 or 2. A 4-bit weight takes a quarter of the space, so the file shrinks by roughly the same factor.
Smaller also means faster, and the reason is not compute. Generating a token reads every active weight from memory once, and that read is the bottleneck — halve the bytes, nearly double the tokens per second. llama.cpp's own numbers for Llama 3.1 8B show it: 29 tokens per second at 16-bit, 51 at Q8_0, 72 at Q4_K_M. Prompt processing barely moves.
How to read the name
GGUF is the file format llama.cpp, Ollama and LM Studio use, and the quant name is the recipe:
- Q plus a number is the rough bit count. Q4 is about four bits per weight, Q8 about eight.
- _0 and _1 are the original formats. Weights are split into small blocks of 32 that share a scale. Q8_0 is still in wide use — at eight bits the simple approach is near-lossless. The rest are legacy.
- K marks the k-quants, added in 2023. Weights are grouped into super-blocks of 256, split into smaller blocks whose scales are themselves quantized.
- S, M and L say how many sensitive tensors get extra bits. In Q4_K_M, half the attention value and feed-forward down-projection tensors sit at 6-bit, the rest at 4-bit — so the name says 4 but the real cost is 4.89 bits per weight.
- IQ marks the i-quants, newer formats built for low bit rates. They work best with an importance matrix, built by running the full-precision model over calibration text.
Q4_K_M is a 4-bit k-quant, medium mix; Q8_0 is 8-bit, original format, no mix.
What you lose at each level
The original k-quant work measured perplexity on the original LLaMA 7B. Against 16-bit: Q6_K cost 0.07%, Q5_K_M 0.24%, Q4_K_M 0.91%, Q3_K_M 4.1%, Q2_K 14.7%. A January 2026 study of Llama 3.1 8B Instruct found the same shape: 69.47 at 16-bit, 69.15 at Q4_K_M, 68.07 at Q3_K_M, 65.49 at Q3_K_S. Down to Q4_K_M the average moved by a third of a point — inside the noise. At Q3_K_S, maths fell first: GSM8K dropped from 77.63 to 68.31.
Size and speed, from llama.cpp's own table for Llama 3.1 8B:
| Quant | Bits per weight | File size | Generation |
|---|---|---|---|
| F16 | 16.00 | 14.96 GiB | 29 t/s |
| Q8_0 | 8.50 | 7.95 GiB | 51 t/s |
| Q6_K | 6.56 | 6.14 GiB | 59 t/s |
| Q5_K_M | 5.70 | 5.33 GiB | 67 t/s |
| Q4_K_M | 4.89 | 4.58 GiB | 72 t/s |
| Q3_K_M | 4.00 | 3.74 GiB | 72 t/s |
| Q2_K | 3.16 | 2.95 GiB | 80 t/s |
Below four bits the speed barely improves while the quality cost climbs — the argument for stopping at four.
The bigger model question
Small model at high precision, or big model at low precision? Usually the bigger model, down to about four bits and often to three. The k-quant work found a 30-billion model at Q2_K had lower perplexity than a 13-billion model at 16-bit — the extra parameters were worth more than the lost precision. On a 24 GB card, a 27-billion model at Q5_K_M beats a 9-billion model at Q8_0 for almost any task. Below three bits the losses grow fast; 1- and 2-bit files are for very large mixture-of-experts models. The same applies one level up: Ollama and llama.cpp are two doors into the same engine.
Don't forget the context
The weights are only half the memory. The other half is the KV cache, holding your conversation and growing with every token of context. llama.cpp stores it at 16-bit by default; -ctk and -ctv switch keys and values to q8_0 or q4_0. In Ollama, the same switch is OLLAMA_KV_CACHE_TYPE. Ollama's docs say q8_0 uses about half the memory with usually no noticeable impact; q4_0 uses a quarter, with a loss that may show at long context. For a 27-billion model at 128K tokens: roughly 8.6 GB at 16-bit, 4.3 GB at 8-bit, 2.1 GB at 4-bit — estimates, but the shape is the point. Same mechanism as the FP8 KV cache in llama.cpp 1.0: the weights fit, the conversation did not.
Beyond GGUF
GGUF is not the only format. Apple's MLX quantizes by default to 4 bits in groups of 64 — about 4.5 bits per weight, close to Q4_K_M. MXFP4 is a 4-bit floating-point format with a shared 8-bit scale per block of 32, about 4.25 bits per weight; OpenAI's gpt-oss models ship their mixture-of-experts weights in it, which is why the 120-billion model fits on one 80 GB GPU and the 20-billion one in 16 GB. The rule is the same everywhere: use the native format as released — a gpt-oss MXFP4 file should stay MXFP4.
How to choose
- Find out what fits — a model page or VRAM calculator gives file sizes; leave 10 to 15% of memory free.
- Start at Q4_K_M for the largest model that fits.
- Spend spare memory on context first, then Q5_K_M or Q6_K. Q8_0 makes sense when memory is plentiful.
- Below four bits, use an importance matrix or a dynamic quant, and only when it gets you a much larger model.
- Use native formats as released.
Quantizing a model yourself
llama.cpp ships a quantizer; the process is two steps: convert to GGUF, then quantize.
python convert_hf_to_gguf.py --outfile model-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it
./build/bin/llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M
From llama.cpp's own quantize guide, which uses Gemma 4 as its example. Add --imatrix for an importance matrix, or --leave-output-tensor to keep the output layer unquantized. One warning from the same guide: never requantize an already-quantized file — --allow-requantize exists, but the guide says it "can severely reduce quality compared to quantizing from 16-bit or 32-bit". Hugging Face's GGUF-my-repo space builds quants in the browser, syncing from llama.cpp's main branch every six hours.
The name on the file is the whole deal, written small. Q4_K_M is the default because it sits at the knee of the curve — the last level where the model gets much smaller and faster for almost nothing. Above it is insurance; below it is a trade to make on purpose, for a bigger model or longer context. Read the name, check the table, pick the level that fits the machine you have.


