A Practical Beginner’s Guide to LLM Quantization
If you have ever tried to run a serious Large Language Model on your own machine, you already know the pain.
You download the model. You get excited. And then reality hits.
Out of memory.
VRAM not sufficient.
System crawls.
VRAM not sufficient.
System crawls.
Modern LLMs are powerful precisely because they are huge. Billions of parameters. Billions of numbers. And those numbers are stored in very high precision formats by default.
Here’s the brutal math.
A 70 billion parameter model stored in FP32 needs about 280 GB of memory just to load. Not to run fast. Just to exist in memory.
Most personal computers do not even come close.
So how are people running these models locally on Mac Minis, laptops, or modest desktops?
That answer is quantization.
And once you understand it, a lot of things suddenly click.
That practicality is what allows people to seriously consider running AI systems locally instead of defaulting to cloud APIs.
The Big Problem With Big Models
When we say a model is “large,” we are not talking about complexity in an abstract sense. We are talking about raw storage.
LLMs are essentially giant collections of weights. Each weight is a number that helps the model decide what comes next. Those numbers are usually stored as 32 bit floating point values.
That precision is great for training. It is terrible for running models on consumer hardware.
Even dropping from FP32 to FP16 cuts memory usage in half. Helpful, but still not enough for most people.
So the real breakthrough comes when we go much lower.
Quantization, Explained Like a Human
Quantization is about reducing precision on purpose.
Instead of storing every weight with extreme accuracy, we accept a little bit of noise in exchange for massive savings in memory and compute.
The easiest way to think about it is images.
Imagine a photo with millions of colors. Now imagine reducing it to a limited color palette. You lose some subtle shading. But the image is still very recognizable. And the file size drops dramatically.
That is quantization.
We are trading a bit of numerical nuance for practicality.
The key insight is this.
LLMs are surprisingly tolerant of small inaccuracies.
LLMs are surprisingly tolerant of small inaccuracies.
And that is why quantization works as well as it does.
A Quick Tour of Data Types
Before going further, it helps to understand the formats involved.
FP32
Full precision. Big. Accurate. Completely impractical for local inference at scale.
Full precision. Big. Accurate. Completely impractical for local inference at scale.
FP16
Half precision. Cuts memory in half. Common for training and some inference setups.
Half precision. Cuts memory in half. Common for training and some inference setups.
BF16
A deep learning favorite. Wide numerical range like FP32, lower precision like FP16.
A deep learning favorite. Wide numerical range like FP32, lower precision like FP16.
INT8
Eight bit integers. Much smaller. Often faster. Some loss in fidelity.
Eight bit integers. Much smaller. Often faster. Some loss in fidelity.
Once you go below this into 6 bit, 5 bit, or 4 bit territory, you are firmly in quantization land.
The Squeezing Process and Its Tradeoffs
At the heart of quantization is one unavoidable fact.
You are mapping many possible values into fewer buckets.
There are two common ways to do this.
Symmetric quantization maps values evenly around zero. Simple. Efficient. Common.
Asymmetric quantization maps values to fit the original range more tightly, but requires tracking an offset. More accurate in some cases. Slightly more complex.
Either way, you introduce quantization error.
Two numbers that were slightly different before may become identical after quantization. When converted back, that difference is gone.
The art of quantization is minimizing how much that matters.
Every clever trick you see in modern formats is really about one thing.
Reducing error without blowing up size.
Reducing error without blowing up size.
GGUF and Why It Matters
This is where things get practical.
If you are running models locally today, you are almost certainly dealing with GGUF files.
GGUF is the modern unified format from the llama.cpp ecosystem. And it is a big deal.
Why?
Because it bundles everything into one efficient file. Model weights. Architecture. Tokenizer. All of it.
Even better, GGUF models can run entirely on CPU. Or partially offload to GPU if you have one. That flexibility is what makes local AI realistic for normal people.
Choosing the Right Quantization
Not all quantizations are equal.
Here is the reality most users land on after experimenting.
Q4_K_M
Four bit. Excellent quality. Roughly 75 percent smaller.
This is the default recommendation for a reason.
Four bit. Excellent quality. Roughly 75 percent smaller.
This is the default recommendation for a reason.
Q5_K_M
Better quality. Slightly larger. Good if you have extra RAM.
Better quality. Slightly larger. Good if you have extra RAM.
Q6_K
Near original quality. Noticeably heavier.
Near original quality. Noticeably heavier.
Q8_0
Maximum fidelity. Minimum compression. Mostly for benchmarking or high end setups.
Maximum fidelity. Minimum compression. Mostly for benchmarking or high end setups.
If you do not want to think too hard, start with Q4_K_M. It punches far above its weight.
How Much RAM Do You Actually Need?
This is the question everyone asks.
Rough estimates, assuming GGUF:
- 7B model at Q4_K_M: about 5 to 6 GB
- 13B model at Q4_K_M: about 9 to 10 GB
- 70B model at Q4_K_M: around 40 GB
Once you see these numbers, choosing hardware stops being guesswork and starts aligning with real-world guidance on selecting machines for local LLM work.
Post Training vs Training Aware Quantization
There are two big approaches behind the scenes.
Post Training Quantization (PTQ)
Train the model normally. Quantize it afterward.
This is what almost all downloadable GGUF models use.
Train the model normally. Quantize it afterward.
This is what almost all downloadable GGUF models use.
Quantization Aware Training (QAT)
Simulate quantization during training so the model adapts to lower precision.
Harder to do. Often better results.
Simulate quantization during training so the model adapts to lower precision.
Harder to do. Often better results.
From a user perspective, this distinction mostly matters philosophically. You are almost always consuming PTQ models today.
Why Quantization Is a Big Deal
Quantization is the reason local AI is not just for researchers anymore.
It is the bridge between massive data center models and the machines sitting on your desk.
Without it, running LLMs locally would still be a novelty. With it, it becomes practical. Useful. And increasingly powerful.
This is not just about saving memory. It is about control. Privacy. Cost. And ownership.
And honestly, once you run a strong quantized model locally and realize how good it still is, it is hard to unsee.
Big models got smaller.
And that changed everything.
And that changed everything.
