Mixture of Experts (MoE)

A design where a model is split into several specialized "experts", activating only a subset for each input to save compute.