Comparison
Partition overwritevsPipeline backfill
Partition overwrite
the run deletes its own day's partition and writes it again, so re-running never duplicates a row.
The standard implementation of pipeline idempotency: a run owns a partition and replaces it wholesale. It is simple, it needs no keys and no merge logic, and it bounds the damage of a bad run to one partition. Its limits are that late-arriving data belonging to an old partition needs that partition re-run, and that the partition has to actually be the unit a run produces, which is a modelling decision made much earlier.
Full entry →Pipeline backfill
the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.
Re-running a pipeline over historical periods to correct or populate them. Whether it is routine or terrifying is decided entirely by idempotency and by blast radius: a backfill of an idempotent partitioned model is boring, and a backfill of an append-only table is a duplication incident. The other half is resource contention — ninety runs at once will happily starve the scheduled work — which is why concurrency limits and a deliberate order matter more than the code does.
Full entry →