a big team where only two or three specialists look at each token, not the whole staff.

Mixture of experts (MoE)

Also calledMoE

An architecture where the model contains many parallel sub-networks ('experts') but routes each token through only a few of them. This gives large total parameter counts with much lower compute per token. Many frontier and open models are MoE.

Commonly confused with

Dense model

Dense model vs Mixture of experts (MoE) →