Applied AI·How models work
every parameter runs for every token, so the memory you have to find is the whole model rather than a fraction of it.
Dense model
Draft summary, pending review
The opposite of MoE: every parameter participates in every token. Simpler to run and reason about; most small local models are dense.