Comparison
Hard delete detectionvsTombstone record
Hard delete detection
the row is gone from the source and still in the warehouse, because your incremental load can only see rows that exist.
Working out what the source deleted, when your extraction method can only observe what is present. Query-based extraction cannot do it directly, so teams fall back to a periodic key-set comparison, a soft-delete flag negotiated with the source team, or moving to log-based capture. It is worth naming because it is nearly always discovered late, by an auditor or a privacy request rather than by a test.
Full entry →Tombstone record
the delete arrives as a message with a key and no body, and a consumer that skips empty payloads keeps the row forever.
A record whose presence means 'this key no longer exists' — typically a key with a null value on a compacted topic. It is how deletion survives a system built out of upserts, and it is how compaction and downstream stores learn to drop something. Consumers written against the happy path routinely ignore them, which is the mechanism behind deleted customers who are still in the dashboard a year later.
Full entry →