Clicked Gallery

What is Quantization?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

The 4-bit quantization cut memory use enough to run the model on a single consumer GPU.

The reader highlighted one word in the docs. Clicked explained the technical term “quantization” in simple terms:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

Quantization shrinks an AI model by storing its numbers with less precision. Each number takes less memory, so the model runs on smaller, cheaper hardware. Done well, quality barely drops.
●○○

Overview

Quantization is putting a model on a diet: same brain, numbers stored sloppier. Half or a quarter of the memory, and it barely notices. 😎

A quick take — often all you need.

●●○

Detail

A model is billions of numbers, called weights, normally stored at 16 bits each. Quantization re-encodes them at 8 or 4 bits, so a 70-billion-weight model drops from about 140 gigabytes to roughly 35 at 4 bits. The full version doesn't fit even on a data-center GPU, while the shrunken one squeezes onto a single high-end consumer graphics card. The trick works because weights cluster in narrow ranges, so a well-chosen coarse scale loses little: benchmark scores typically dip a few percent at 8 bits and stay usable at 4. Below that it gets rough, and 2-bit versions are often noticeably worse. Speed usually improves as well: the slow part of running a model is fetching all those numbers from memory, and smaller numbers mean less to fetch. This is the main reason local AI exists at all, letting people run yesterday's frontier models on gaming hardware.
●●○

Detail

A model is just billions of numbers, and stock ones burn 16 bits per number. Requantize at 4 bits and a 70B model shrinks from about 140 GB to about 35, which is the line between not fitting on a data-center GPU and running on your gaming rig. The numbers huddle in tight ranges anyway, so coarse rounding costs shockingly little: a few percent on benchmarks at 8-bit, still solid at 4. Push to 2-bit and the wheels start wobbling. Bonus: it usually runs faster, since the slow part is fetching numbers, not crunching them. This one trick is basically why local AI is a hobby instead of a fantasy. 😎

Want more? One click digs deeper.

●●●

Analogy

Shrinking a photo to send it. The original file is huge; saving it at a lower quality makes the file a tenth of the size, and on a phone screen nobody can tell. Zoom in far enough and you will find the smudges, but for everyday viewing the picture does its job.
Deepest level reached.
●●●

Analogy

MP3s. Nobody stores albums as raw studio files; you keep a tenth of the data and your earbuds can't tell the difference. Audiophiles swear they hear the loss, blind tests say barely, and meanwhile the whole library fits in your pocket.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Quantization maps model parameters from higher-precision floating-point representations to lower-bit-width formats, reducing memory footprint and bandwidth requirements at inference. Post-training quantization and quantization-aware training trade implementation cost against accuracy retention, with degradation generally modest at 8-bit and increasingly significant below 4-bit.

Want Clicked to explain terms like “quantization” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required