Clicked Gallery

What is a Mixture of Experts (MoE)?

Highlighted from a real engineering doc. Explained by Clicked.

Used in a sentence

Engineering Notes · AI Systems

The lab's flagship is a mixture of experts, waking only a fraction of itself for each query.

The reader highlighted one word in the docs. Clicked explained the technical term “mixture of experts” in plain language:

Explained in three depths

Same facts, different vibe — Slang mode 😎

The Clicked way

●○○

Overview

A mixture of experts is a model built as many smaller specialist sections, with only a few of them switched on for any one question. The model can be enormous in total while doing the work of a much smaller one every time it answers. That is how several of the biggest models stay affordable to run.
●○○

Overview

A mixture of experts is a model built as many specialist sections, with only a couple of them switched on for any one question. Total size can be gigantic while each answer is handled by a sliver of it. Huge model, small amount of work per answer. 😎

A quick take — often all you need.

●●○

Detail

The point of a mixture of experts is that total size and work per answer stop being the same thing. In an ordinary model every question wakes the entire network, so doubling the model doubles the work done on every single answer. A mixture of experts is instead built as separate sections, the experts, with a small router in front that picks which few to consult for each piece of the input. Ask a coding question and it might wake two sections out of sixty-four, so the answer is computed by a slice of the model rather than all of it. Every section still has to sit in memory waiting to be called, so the saving is in work done per answer, not in what it costs to keep the model available. Training takes extra care too, because a router that keeps picking the same favorites leaves much of the model barely trained. The payoff explains why the design keeps appearing in frontier models: capacity can grow without the work on each answer growing with it.
●●○

Detail

Normally every question lights up the whole model, so a bigger model means more work on every single answer. Mixture of experts breaks that link: build the thing as separate sections, put a router in front, and let it pick the two or three sections actually worth consulting for what you just asked. Sixty-four experts on the payroll, two clocked in for your question, the other sixty-two idle. Catch one, everybody still has to be in the building, so the memory bill stays brutal even when the work does not. Catch two, routers get lazy and keep calling their favorites, which leaves half your model barely trained, so trainers actively shove the work around. Worth it anyway, which is why the frontier keeps shipping it: capacity goes up, the work per answer does not. 😎

Want more? One click digs deeper.

●●●

Analogy

A mechanic's toolbox holding two hundred tools. Any one repair needs four or five of them, and he reaches straight for those while the rest stay in the drawer, so the job is no heavier than if the box were small. He still bought all two hundred and still hauls the whole box to every call, because the point of owning the set is that whatever rolls in is already covered. Light work on each repair, permanent cost to keep that range on hand.
Deepest level reached.
●●●

Analogy

One air conditioner cooling an entire building versus separate units per floor. With the single unit, the empty second floor gets cooled whether anyone is up there or not, and the bill says so. With separate units you only cool the floor people are actually on, so the running cost drops hard. You still had to buy a unit for every floor, because the day someone works up there it needs to be ready.
Deepest level reached.

Unfamiliar concept? A real-world example makes it click — fresh analogies on tap.

AI explanations may contain errors · Not professional advice

Formal definition — The same term, explained the usual way

A mixture-of-experts model partitions its parameters into multiple expert subnetworks, with a learned routing function activating a small subset per input token. This decouples total parameter count from per-inference computation, permitting high-capacity models at reduced inference cost, while retaining full memory residency requirements and introducing training challenges such as expert load imbalance.

Want Clicked to explain terms like “mixture of experts” directly in your browser — including on PDFs?

Add to Chrome — Free

50 free Explanations · No credit card required