Comparison
Batching / continuous batchingvsBulk load
Batching / continuous batching
your GPU sat mostly idle between single requests until you served many at once, and throughput went up several times over.
Serving many requests together to keep the GPU saturated. Continuous batching admits new requests into the batch as others finish, rather than waiting for the slowest, and is the single biggest throughput win in serving engines.
Full entry →Bulk load
you stopped inserting a row at a time and pointed the warehouse at a folder of files, and the load went from hours to a minute.
Loading data through a warehouse's file-oriented path rather than as individual statements. Analytical stores are built to ingest large sorted chunks and are catastrophically slow at single-row writes, so this is less an optimisation than the intended door. It is the first thing to check when someone reports that the warehouse 'cannot keep up with writes' — usually it is being used as if it were a transactional database.
Full entry →