Comparison
Schema driftvsSchema-on-write
Schema drift
the vendor added three columns and renamed one over the weekend, and nobody told anybody.
The source's structure changing without coordination — added, removed, renamed or retyped fields. It is distinct from planned schema evolution in exactly the way that matters: nobody agreed to it, so no compatibility rule was applied. Handling it means deciding in advance whether unknown fields are absorbed, ignored or fatal, and a pipeline that has never made that decision has made it implicitly by whichever the library defaults to.
Full entry →Schema-on-write
the load rejected the file because a column changed type, and the bad data never got into the table.
Validating and typing data as it is written, so everything in the table conforms. The rejection is the feature: a failed load is a loud, single, fixable event, while a silently accepted bad type is a quiet wrong number spread across every downstream consumer. The complaint against it is always speed of onboarding, which is exactly the cost it is charging on purpose.
Full entry →