Mixture of Experts (MoE)

A design that splits a model into specialised sub-networks called experts, with a router choosing one or a few experts for each token, so only part of the model runs at each step. That gives large capacity at a lower running cost: DeepSeek-V3, for example, has 671 billion parameters but activates only 37 billion per token. The trade-off is that every expert stays loaded in memory.