Applied AI·Local and self-hosted inference
you take the Q4_K_M file because that is the community default, and notice the quality slipping when you go below it.
Quantisation levels
Draft summary, pending review
GGUF names encode the recipe: Q4_K_M is roughly 4.5 bits per weight and the community default balance; Q8_0 is near-lossless and twice the size; below Q4 degradation gets noticeable, and very small quants can be false economy against a smaller model at higher precision.