Comparison
Cold start / model loadingvsScale to zero
Cold start / model loading
the first request after a quiet hour takes thirty seconds because the weights had to be loaded off disk again.
The seconds-to-minutes cost of loading gigabytes of weights into memory before the first token. Ollama unloads idle models by default, so the first request after a lull pays this; keep-alive settings and warm pools are the fixes.
Full entry →Scale to zero
nothing is running and nothing is billed until a request arrives, and that first request waits for a container to start.
Allowing a workload's replica count to reach zero when idle, with something in front holding the first request while capacity comes back. It is the strongest cost lever available for spiky and internal workloads, and it moves the whole cost of a cold start onto whoever is unlucky enough to be first. It is a poor fit for anything with a latency objective on the first request, and a good fit for review environments, batch endpoints and internal tools.
Full entry →