Comparison
BackfillvsPipeline backfill
Backfill
you added a column and now have to populate it for eighty million existing rows without locking the table or saturating the disk.
Populating a new field or table for existing data, usually in batches with a delay between them. Doing it in one statement takes a long lock and a large amount of replication lag with it. The batch job also has to handle rows written after it started, which is why backfills are normally paired with dual writes.
Full entry →Pipeline backfill
the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.
Re-running a pipeline over historical periods to correct or populate them. Whether it is routine or terrifying is decided entirely by idempotency and by blast radius: a backfill of an idempotent partitioned model is boring, and a backfill of an append-only table is a duplication incident. The other half is resource contention — ninety runs at once will happily starve the scheduled work — which is why concurrency limits and a deliberate order matter more than the code does.
Full entry →