Comparison
Schema-on-readvsSchema-on-write
Schema-on-read
the file was written as whatever the source sent and the shape is decided by the query that reads it, which is why two teams parse it differently.
Deferring interpretation of structure to read time, so anything can be stored immediately. It is what makes a lake able to accept a new source the same afternoon, and it moves the entire cost of a malformed record from the producer to every consumer, forever. The pattern is defensible for a landing zone and indefensible for a table that many teams query.
Full entry →Schema-on-write
the load rejected the file because a column changed type, and the bad data never got into the table.
Validating and typing data as it is written, so everything in the table conforms. The rejection is the feature: a failed load is a loud, single, fixable event, while a silently accepted bad type is a quiet wrong number spread across every downstream consumer. The complaint against it is always speed of onboarding, which is exactly the cost it is charging on purpose.
Full entry →