Comparison
Retention policyvsRight to erasure
Retention policy
raw events older than thirteen months are deleted automatically, and nobody has to remember to do it.
A rule for how long each class of data is kept and what happens then. It is simultaneously a cost control, a legal obligation and a risk reduction, since data you no longer hold cannot leak. It is nearly always easier to implement at the partition level than by row, which is another reason partitioning by time is worth doing before you need it.
Full entry →Right to erasure
a person asks to be deleted and you have to find them in the warehouse, the lake, every backup and last quarter's exports.
A person's legal right to have their personal data removed, which a data platform must be able to honour within a deadline. It is architecturally awkward because analytical storage is built to be append-only and immutable, so the answer is usually a combination of pseudonymisation, key deletion and targeted rewrites. Snapshot retention and time travel are the parts most often forgotten, since a deleted row remains readable in an old snapshot until it expires.
Full entry →