Running it: capacity, failure and recovery

one datacentre lost power and a third of your instances went with it, which is the unit this is designed around.

Availability zone

everything you run is in one geography, and a customer in another continent pays a hundred and fifty milliseconds for it.

Cloud region

you can survive an entire region disappearing, and you now have a data replication problem in every single service.

Multi-region

both sites take live traffic all the time, so you already know the second one works when the first one dies.

Active-active

the second site sits there costing money and serving nothing, and the day you need it you find out which four things were never replicated.

Active-passive

a scaled-down copy runs continuously with the data replicating, and recovery is a scale-up rather than a build-out.

Warm standby

the original site came back and moving home turned out to be harder than the failover was, because the standby now holds the newer data.

Failback

the control plane was down for an hour and traffic kept flowing, because nothing in the request path needed to ask it anything.

Static stability

customers are divided across eight complete copies of the stack, so a bad deploy or a poison request takes out an eighth of them.

Cell-based architecture

each customer is assigned a random pair of workers rather than one, so two customers rarely share both and one bad tenant cannot take out everyone.

Shuffle sharding

the plan for the day a whole region, account or dataset is gone, written down and hopefully rehearsed once a year.

Disaster recovery

the business says four hours, and the restore has never been timed, so nobody knows whether that is true.

Recovery time objective

backups run nightly, so recovering means losing up to a day of writes, and nobody outside the platform team knew that.

Recovery point objective

you restore the backup into a scratch environment on a schedule, because a backup nobody has restored is a hypothesis.

Restore testing

you deliberately add two hundred milliseconds to one dependency in production and watch what the rest of the system does.

Fault injection

you run at sixty percent so a zone failure or a traffic spike does not immediately become an outage.

Headroom

the instance was fast for twenty minutes and then abruptly slow, because it had been spending credits it had run out of.

Burst capacity

one team's runaway job could not consume the whole cluster, because their slice had a declared ceiling.

Resource quota

several teams or customers share one cluster, and every performance question becomes 'is it us or is it them'.

Multi-tenancy

you decide whether a namespace is enough or whether each customer needs their own cluster, and the answer is about who you are defending against.

Tenant isolation

the managed database will be restarted by the provider at some point inside a four-hour window you chose months ago.

Maintenance window

you fell four versions behind and now the upgrade is a project, because you can only move one version at a time.

Patch cadence

before it can take real traffic, someone checks it has alerts, a runbook, a rollback, an owner and a restore that has been tested.

Operational readiness review

the twenty-minute manual thing you did every Tuesday for a year became a controller that does it without telling you.

Toil automation

you got a database, a queue and a domain name by opening a pull request instead of by waiting four days for a ticket.

Self-service platform

doing it the standard way gets you logging, tracing, secrets and a pipeline for free; doing it your own way is allowed and unsupported.

Paved road

a team's output is not a feature but the thing forty other engineers use to ship features safely.

Platform engineering

the platform team runs interviews, publishes a roadmap and measures adoption, because their users can always route around them.

Platform as a product