Mixture of experts (MoE)

What is a mixture of experts?

A mixture of experts, usually shortened to MoE, is a model design that breaks one large neural network into many smaller specialist parts called experts. For each piece of input, a small routing network decides which one or two experts should handle it, and the rest sit idle.

The effect is that a model can hold a huge number of parameters while only using a small slice of them for any given input. You get the capacity of a very large model at close to the running cost of a small one.

A busy hospital is a useful comparison. A patient at reception is not seen by every doctor in the building. A triage nurse sends them to the one or two specialists their case needs. MoE routes each token the same way, to the experts most suited to it.

Why mixture of experts exists

In a normal "dense" model, every parameter is used for every input. Making the model smarter usually means making it bigger, and a bigger dense model costs proportionally more for every single word it processes. That cost adds up fast during both training and everyday use.

MoE breaks the link between how much a model knows and how much it computes per token. The total parameter count can grow into the hundreds of billions, while the amount of work done for each token stays roughly flat. That is why many of the largest recent language models are built this way.

How a mixture of experts works

Experts. Each expert is a small feed-forward network, the same kind of block found in a standard transformer. A single MoE layer holds several of them side by side. Experts are not assigned topics by hand; during training each one drifts towards the patterns it handles best.

The router. Also called the gating network, this is a small network that looks at each incoming token and scores every expert. It then sends the token to the highest-scoring few, commonly the top two.

Sparse activation. Because only the chosen experts run, most of the model stays switched off for any given token. This is what keeps the compute cost low at inference time even when the total model is enormous. The outputs of the active experts are combined, weighted by the router's scores, and passed on to the next layer.

A worked example

Mixtral 8x7B, released by Mistral AI, is a clear case. Each MoE layer holds 8 experts, and the router picks 2 of them for every token. The full model contains 46.7 billion parameters, but only about 12.9 billion are active for any single token. So it runs at the speed and cost of a roughly 13 billion parameter model while carrying the knowledge of a much larger one.

Mixture of experts versus a dense model

A dense model uses all of its parameters on every token. It is simpler to train and to serve, and its memory use is easy to predict, but it pays full price for its full size on every word.

An MoE model runs cheaper per token for the same amount of stored knowledge. The trade-off is that all those experts still have to be held in memory even though only a few run at a time, so MoE models need a lot of memory to serve. They are also harder to train well, because the router has to learn to spread work evenly across experts.

What to watch out for with mixture of experts

Memory, not compute, becomes the limit. Every expert must be loaded and ready even if it rarely fires. An MoE model that is cheap to run per token can still need far more memory than its active-parameter count suggests.

The router can play favourites. If routing is left unchecked, a handful of experts get most of the traffic while others barely train. Builders add balancing techniques during training to keep the load spread out.

Active parameters can mislead. A model advertised by its active-parameter count sounds small, but you still need the hardware to hold every expert. Read both the total and the active figures before you judge what it takes to run.