Comparison
Dense modelvsMixture of experts (MoE)
Dense model
every parameter runs for every token, so the memory you have to find is the whole model rather than a fraction of it.
The opposite of MoE: every parameter participates in every token. Simpler to run and reason about; most small local models are dense.
Full entry →Mixture of experts (MoE)
a big team where only two or three specialists look at each token, not the whole staff.
An architecture where the model contains many parallel sub-networks ('experts') but routes each token through only a few of them. This gives large total parameter counts with much lower compute per token. Many frontier and open models are MoE.
Full entry →