Why compress a model in the first place
A model’s weights are stored at high precision by default — 16 bits per number — and at that precision even a mid-sized model eats an enormous amount of VRAM. Quantization drops that precision down to 8, 6, or 4 bits per weight. The model shrinks by roughly 2 to 4x and fits on hardware that couldn’t hold it before. For most business workloads, the quality hit is small enough that you’d struggle to spot it in a blind test.
What Q4, Q6, and Q8 actually mean
| Quantization | Memory | Quality |
|---|---|---|
| FP16 (uncompressed) | maximum | the reference baseline, but overkill for most use cases |
| Q8 | high | nearly indistinguishable from the baseline |
| Q6 | medium | very close to Q8 |
| Q4 | low (about half of Q8) | the gap is barely noticeable on business tasks |
How this ties back to VRAM
Quantization is the reason a 32-billion-parameter model can run in 18–24 GB of VRAM instead of demanding 60+ GB. For a full breakdown of memory requirements by model size, see how much VRAM a local LLM actually needs, and for a rundown of the models themselves, see the best open-source LLMs to self-host.
Where teams get this wrong
- •Over-compressing a small model hurts a lot more than over-compressing a big one — a 7–9B model at Q4 degrades noticeably faster than a 70B model at the same setting,
- •Sizing memory right up to the model’s footprint leaves no room for context length or concurrent users, and that’s usually where things fall over first,
- •Picking a quantization level “because a benchmark said so” doesn’t hold up — the only test that matters is your own workload.
We pick the model and the quantization level for your hardware and your workload — more on how that works on the AI server page.
Frequently asked questions
Is Q4 that much worse than Q8?
For most business workloads, the gap between Q4 and Q8 on large models is small enough that you won’t notice it in a blind test, and Q4 uses about half the memory. On harder reasoning and coding tasks, the extra precision of Q8 can give a modest edge. The right call comes from testing on your own workload, not from a spec sheet.
What is GGUF?
It’s the most common file format for storing quantized models for local inference — the format llama.cpp and Ollama both build on. A single model typically ships in several compression levels (Q4, Q5, Q6, Q8), and you pick the one that fits your hardware.
Can you run a model without quantization at all?
Yes, at full precision (FP16), but that takes 2–4x more VRAM and rarely buys a meaningful quality improvement on business tasks. That’s why almost every production deployment runs a quantized version instead.