Model Compression

Shrink a trained model so it ships on phones, browsers, and cheap GPUs — quantization, pruning, knowledge distillation, low-rank factorization, and the accuracy/size/latency trade-off that decides which one to reach for.

Learn Model Compression in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and a…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

A full-precision model is an over-packed suitcase you can barely close. Compression is packing it smartly so it fits the carry-on limit — the same trip, far less bulk. Each technique is a different packing trick:

And the trade-off is the airline's scale: pack too aggressively and you leave behind something you needed (accuracy). The art is the lightest bag that still has everything for the trip — small enough for an edge device, cheap to run, and fast to respond.

A trained model is just a big pile of numbers (its weights ). The more numbers and the more bits each one uses, the more memory it eats and the slower it runs. Compression trades a sliver of accuracy for three things you nearly always need in production:

Most models store each weight as a 32-bit float (FP32) — 4 bytes that can represent tiny fractions very precisely. Quantization asks: do you really need that precision? Often you can map those floats onto 8-bit integers (INT8) — just 256 possible values, 1 byte each. That is an instant 4x size cut, and integer maths runs faster on most CPUs and accelerators.

The trick is a single scale factor. You find the largest weight, divide the range into 256 steps, and store which step each weight is closest to. To use the model you multiply the integer back by the scale to get an approximate float. The gap between the original and the recovered float is the quantization error — small if your weights are well-behaved.

Going further — INT4 (16 levels) gives 8x compression but a bigger error, so it needs smart schemes like GPTQ or AWQ. The two ways to apply quantization: post-training (PTQ) quantizes an already-trained model (easy, slight loss), while quantization-aware training (QAT) simulates the rounding during training so the model learns to tolerate it (better, more work).

Worked example — one-line INT8 quantization in PyTorch:

The INT8 state dict is roughly a quarter of the FP32 one — exactly the 4x you'd expect from 4 bytes down to 1. You'll build this scale-and-round logic by hand in the runnable example below.

Trained networks are wasteful: a large fraction of weights are so close to zero they contribute almost nothing. Pruning deletes them. The simplest recipe is magnitude pruning — pick a threshold, then zero out every weight whose absolute value is below it. The fraction of weights now zero is the sparsity .

There is a crucial catch. Magnitude (unstructured) pruning scatters zeros anywhere in the matrix — you get high sparsity, but the matrix is still the same shape, so ordinary GPUs do the same amount of work and you get no speedup , only a smaller file (when stored sparsely). Structured pruning instead removes whole rows, channels, or attention heads, so the matrix genuinely shrinks and standard hardware runs it faster. Rule of thumb: unstructured for size, structured for latency.

Rather than shrink a model, knowledge distillation trains a brand-new small student model to imitate a large, accurate teacher . The clever part is what the student copies. A normal label is hard — "this image is a cat, everything else is 0%". The teacher's full probability distribution is a soft label : "94% cat, 4% dog, 1% bird". That extra structure — the teacher quietly saying "this looks a bit like a dog too" — is the dark knowledge the student learns from.

To expose that structure you raise the softmax temperature . Higher temperature flattens the distribution so the small probabilities become visible and informative. The student is trained to match these softened teacher outputs (often plus the real labels). The result is a small model that punches well above its size — DistilBERT is 40% smaller than BERT while keeping about 97% of its accuracy.

Worked example — the distillation loss in PyTorch:

The KL-divergence measures how far the student's softened output is from the teacher's. Multiplying by T*T keeps the gradient scale right when the temperature is high. Minimise this and the student's distribution slides toward the teacher's.

A linear layer is a weight matrix . A 1000×1000 matrix holds a million numbers. Low-rank factorization approximates that matrix as the product of two skinny matrices — say 1000× r times r ×1000 for a small rank r . With r = 50 that is 1000×50 + 50×1000 = 100,000 numbers instead of 1,000,000 — a 10x reduction for that layer.

It works because real weight matrices are often low rank : their information lives in far fewer dimensions than their shape suggests, so a smaller rank captures almost all of it. The same idea powers LoRA , the dominant way to fine-tune large language models cheaply — you train only a tiny low-rank update instead of the full matrix. The cost is that too small a rank throws away real signal, so you pick the smallest r your accuracy can tolerate.

There is no free lunch — every technique spends a little accuracy to buy size and speed. Choosing well means knowing your target . Deploying to a watch? You need maximum size and latency savings and can accept more accuracy loss. Serving a medical model? Guard accuracy and compress conservatively.

In practice you stack techniques: distill to a smaller architecture, prune it structurally, then quantize to INT8 — combinations reach 10–50x compression on edge devices. Crucially, some tasks live near an accuracy cliff : maths reasoning and code generation collapse under aggressive quantization, while text classification barely notices. Always benchmark on your task, not a generic one.

This is INT8 quantization with no libraries — just a scale factor, rounding, and the error it costs. Read the comments, then press run and watch the recovered floats land close to the originals.

Fill in the two blanks marked ___ so the round-trip works: build the scale from INT8's largest value, and dequantize with the same scale. Check your output against the # ✅ Expected output comment.

Fill in the two blanks so magnitude pruning zeroes every weight below the threshold and counts the survivors. Half the weights should fall away here.

These four mistakes sink most first compression attempts:

❌ Too-aggressive quantization (the accuracy cliff)

Jumping straight to INT4 on a sensitive task — maths reasoning, code generation — and watching quality fall off a cliff while text-classification benchmarks looked fine.

✅ Fix: step down gradually (FP16 → INT8 → INT4), use a smart scheme like GPTQ/AWQ for 4-bit, and benchmark on your actual task at each step.

Pruning 90% of weights to zero and being baffled that inference is exactly as slow — the matrix is the same shape, just full of zeros, so a dense GPU does the same work.

✅ Fix: use structured pruning (remove whole channels/heads) for latency, or sparse-aware kernels/hardware if you must keep unstructured sparsity.

Pruning or quantizing and shipping immediately, then losing several points of accuracy that were completely recoverable.

✅ Fix: fine-tune the compressed model for a few epochs (or use quantization-aware training) so the surviving weights adapt and claw the accuracy back.

❌ Trusting average accuracy, hitting an accuracy cliff in production

A compressed model that looks fine on average but collapses on a specific slice (a rare class, long inputs) because you only checked an aggregate number.

✅ Fix: evaluate before vs after on the same held-out set, broken down by slice, and set an accuracy budget you refuse to cross.

Now combine both skills with only a comment outline — no filled-in logic. Magnitude-prune a layer, then quantize the survivors to INT8 and print a tiny compression report.

Lesson 39 complete — you can shrink a model for deployment!

You can quantize FP32 weights to INT8 with a scale factor, prune small weights with magnitude and structured pruning, distill a teacher into a small student, factorize a big matrix into two small ones, and weigh the accuracy/size/latency trade-off for any target device — while dodging the accuracy cliff, the no-speedup pruning trap, and skipping the fine-tune.

🚀 Up next: Hardware Optimization — tune a compressed model for a specific chip with kernels, fusion, and accelerators.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary