jargon

Backend & systems·topic 10 of 13

Observability and operations

You cannot fix what you cannot see, and you cannot see anything under an alert storm. This is the language of knowing what your system is doing and agreeing how good it has to be.

Read in order · tick what you already know

  1. 01

    you record a number over time so you can see the shape of the whole system at once without reading a single log line.

    Metric

  2. 02

    you log an object with named fields instead of a sentence, so you can query on user id rather than grep for it.

    Structured logging

  3. 03

    you generate one id at the edge and stamp it on every log line the request produces anywhere in the system.

    Correlation ID

  4. 04

    you follow one slow request across six services and see exactly which hop ate eight hundred milliseconds.

    Distributed tracing

  5. 05

    you wrap the database call in its own timed unit so it shows up as its own bar inside the request's waterfall.

    Span

  6. 06

    you keep one percent of traces because storing all of them costs more than the service does.

    Trace sampling

  7. 07

    you put latency, traffic, errors and saturation on one dashboard and stop trying to look at ninety graphs.

    Golden signals

  8. 08

    you pick the one number that actually represents whether users are having a good time, like the fraction of requests under 300ms.

    Service level indicator

  9. 09

    you commit internally to 99.9 percent of requests succeeding over 30 days, and you now have a number that says when to stop shipping features.

    Service level objective

  10. 10

    you promise the customer 99.5 percent in a contract with money attached if you miss it.

    Service level agreement

  11. 11

    your SLO allows 0.1 percent failure, you have used 80 percent of that this month, and the risky migration now waits.

    Error budget

  12. 12

    you add user id as a label and your metrics bill and query latency both go up by three orders of magnitude.

    Metric cardinality explosion

  13. 13

    the channel has forty alerts a day, everyone has muted it, and the real one goes unnoticed for two hours.

    Alert fatigue

  14. 14

    the alert links to a page that tells the person on call what to check and what to do, at four in the morning.

    Runbook

  15. 15

    you run a scripted fake user journey every minute so you find out the checkout is broken before a customer tells you.

    Synthetic monitoring

  16. 16

    you write up what happened focusing on why the system allowed it, not on who typed the command.

    Blameless postmortem