Data engineering·Orchestration and running it in anger
the run deletes its own day's partition and writes it again, so re-running never duplicates a row.
Partition overwrite
Also calledinsert overwrite, delete-insert, replace partition
The standard implementation of pipeline idempotency: a run owns a partition and replaces it wholesale. It is simple, it needs no keys and no merge logic, and it bounds the damage of a bad run to one partition. Its limits are that late-arriving data belonging to an old partition needs that partition re-run, and that the partition has to actually be the unit a run produces, which is a modelling decision made much earlier.