Scaling and load management

you measure the delay before the work starts moving, separately from how long the work itself takes.

Latency

you measure what the user actually experiences: request sent to last byte received, queueing and all.

Response time

you measure how many requests the system finishes per second, which is a different question from how long any one of them takes.

Throughput

the median request is 40ms, the 99th percentile is four seconds, and every page that fans out to ten services hits it.

Tail latency

you push utilisation past eighty percent and latency stops rising gently and starts going vertical.

Saturation

you work out from load tests how much traffic one instance handles, and how many you need for Black Friday.

Capacity planning

you move the database to a machine with four times the RAM, which takes an afternoon instead of a re-architecture.

Vertical scaling

you add three more identical instances behind the load balancer instead of making the existing one bigger.

Horizontal scaling

one address takes all the traffic and spreads it across the instances, skipping the ones failing their health check.

Load balancer

you pin each user to the same instance so the state cached in that process is still there on their next request.

Session affinity

you add a cache node and only one Nth of the keys move, instead of every key rehashing to a different server.

Consistent hashing

the instance count goes up on its own when CPU crosses a threshold, and back down twenty minutes after the spike ends.

Autoscaling

one customer's enormous batch job makes every other customer's requests slow, on shared infrastructure they cannot see.

Noisy neighbour