Clicked Gallery

What is Model Distillation?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

Engineers used model distillation to shrink the flagship system into a phone-sized assistant.

The reader highlighted one word in the docs. Clicked made the technical term “model distillation” easy to understand:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

Model distillation is training one AI model on another model's answers instead of on raw data. A big teacher model answers huge numbers of examples, and a student model learns to reproduce those answers, picking up most of the skill. It is used to shrink big models into cheap fast ones, and to boost a model using a stronger rival's outputs.
●○○

Overview

Distillation is training a model on another model's answers: the big teacher answers a mountain of prompts, the student learns to match them. Used to shrink your own giant into something phone-sized, or to level up your model off a stronger rival's outputs. Most of the skill, a fraction of the cost. 😎

A quick take — often all you need.

●●○

Detail

The teacher's finished answers become the student's textbook. Run a large model across enormous numbers of prompts, collect its outputs, sometimes including how confident it was across the options, and train the student to reproduce them. Studying finished judgments turns out to be a far more concentrated lesson than studying the raw internet. The classic purpose is compression: the student often keeps a large share of the teacher's ability at a tenth of the size, which is how flagship-level AI ends up running on phones. That differs from quantization, which squeezes the same model into coarser numbers rather than training a new one. The second purpose makes the headlines: improving your own model by distilling a stronger rival's, since a competitor's model can be distilled through nothing but its public answers. That is why AI labs ban the practice in their terms of service and accuse each other of raising students on their teachers.
●●○

Detail

The teacher's answers are the curriculum. Blast a big model with a mountain of prompts, save everything it says, sometimes with its confidence spread across the options, and train your student to match. Skipping the raw internet and learning straight from finished judgments is a shortcut that often keeps most of the ability at a tenth of the size. Use one: shrink your own flagship until it fits on a phone, and no, that is not quantization, which squeezes the same model into coarser numbers instead of raising a new one. Use two, the spicy one: level up YOUR model by distilling someone else's, through nothing but its public answers. Which is why labs write bans into their terms of service and accuse each other of raising students on their teachers, and the receipts are just API responses. 😎

Want more? One click digs deeper.

●●●

Analogy

An apprentice who learns from the master chef's finished decisions instead of from years of cooking school. She does not re-taste ten thousand ingredient combinations; she studies what the master actually plated across ten thousand orders, what got salted, what got sent back, what got paired. The master spent a career learning from raw ingredients; the apprentice learns from the master's answers. The strange result is real: most of the skill transfers without any of the original journey.
Deepest level reached.
●●●

Analogy

Learning chess by studying a grandmaster's published games instead of grinding a million games yourself. Every move and every choice laid out across thousands of games: you study the decisions, not the decade behind them. Within a year you play a decent imitation of their style. The grandmaster is annoyed, because you built it entirely off games they made public, and no rule says studying someone's games is cheating.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

Model distillation is the training of a student model to reproduce the outputs, and often the output distributions, of a teacher model, serving both compression, retaining much of the teacher's capability at reduced size and inference cost, and capability transfer between models. Distillation performed against third-party models via their public interfaces raises contractual and intellectual-property disputes and is commonly prohibited in providers' terms of service.

Want Clicked to explain terms like “model distillation” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required