Comparison
Incremental modelvsPartition overwrite
Incremental model
the model rebuilds only the last three days each run instead of the whole four-year table, and once a month you rebuild it fully anyway.
A transformation that processes and appends or merges only new or changed data rather than rebuilding from scratch. It is how a large table stays affordable to maintain, and it introduces the possibility of the incremental result diverging from what a full rebuild would produce. That is why a periodic full refresh is not redundancy but a correctness check, and why teams that never run one eventually cannot explain a discrepancy.
Full entry →Partition overwrite
the run deletes its own day's partition and writes it again, so re-running never duplicates a row.
The standard implementation of pipeline idempotency: a run owns a partition and replaces it wholesale. It is simple, it needs no keys and no merge logic, and it bounds the damage of a bad run to one partition. Its limits are that late-arriving data belonging to an old partition needs that partition re-run, and that the partition has to actually be the unit a run produces, which is a modelling decision made much earlier.
Full entry →