Applied AI·Local and self-hosted inference
the model did not fit in GPU memory, so some layers went to the CPU and generation slowed to a crawl.
Offloading
Draft summary, pending review
Splitting model layers between GPU and CPU when the model does not fully fit in GPU memory. It works, at a steep speed cost; a model mostly on CPU generates painfully slowly.