Comparison
DistillationvsQuantisation
Distillation
the big expensive model becomes the teacher, and what you actually ship is the small one that learned to imitate it.
Training a smaller, cheaper model to imitate a larger one's outputs. The large model is the teacher, the small one the student. It is the standard route to cutting inference cost once a frontier model has proven a task is solvable.
Full entry →Quantisation
rounding the model's numbers to save space; round moderately and quality barely notices.
Storing weights at lower numeric precision, such as 4-bit integers instead of 16-bit floats, shrinking memory and speeding inference for a small quality cost. The reason a 27B model runs on your 16GB laptop at all.
Full entry →