Local and self-hosted inference

you go to the model card to find the licence, the prompt format and which quantised file to download.

Hugging Face

legal asks which licence the open-weight model ships under, and the answer is on the model card and does decide whether you can ship.

Model licences

you download the weight file the Python ecosystem uses and convert it to GGUF, essentially never the other way round.

Safetensors

you downloaded one file with the weights, tokeniser and metadata inside it and pointed the local runner at that.

GGUF

quick sizing check: a Q4 model needs roughly parameter-count-in-billions x 0.6 GB, plus room for context.

VRAM and unified memory

you halve the number of bits per weight and the memory you need roughly halves with it.

Precision (FP32, FP16, BF16, INT8, INT4)

rounding the model's numbers to save space; round moderately and quality barely notices.

Quantisation

you take the Q4_K_M file because that is the community default, and notice the quality slipping when you go below it.

Quantisation levels

the model did not fit in GPU memory, so some layers went to the CPU and generation slowed to a crawl.

Offloading

you compare two quantisations of the same model with a single number, and remember it is not a task-level eval.

Perplexity

you swap llama.cpp for vLLM, get different speed and different flags, and the model still knows exactly the same things.

Inference engine

you put the loaded model behind an OpenAI-shaped HTTP API with queueing and health checks, and your client code stops caring which engine is underneath.

Model server / serving

you run a serious model on your own laptop, and the thing making that possible underneath is llama.cpp.

llama.cpp

you pull a model with one command and get a local HTTP API shaped like the hosted one, which is right for your scripts and not for a fleet.

Ollama

you needed to serve an open model to real traffic on GPUs, and this is what you put in front of it.

vLLM

the long conversation got slower and then ran out of memory mid-generation, because the cache grows with every token in the window.

KV cache

your GPU sat mostly idle between single requests until you served many at once, and throughput went up several times over.

Batching / continuous batching

a small model guesses the next few tokens, the big one checks them in a single pass, and you get identical output faster.

Speculative decoding

the first request after a quiet hour takes thirty seconds because the weights had to be loaded off disk again.

Cold start / model loading

you measured tokens per second on your own hardware at your own context length, and it was nothing like the published number.

Throughput benchmarking