Comparison
Poison recordvsQuarantine table
Poison record
one malformed row fails the batch, the batch retries, and the same row fails it again every five minutes for two days.
A single input that cannot be processed and blocks everything behind it because the failure is retried indefinitely. Retry logic that cannot distinguish a transient failure from a permanent one turns a single bad record into a stopped pipeline. The standard treatment is a bounded retry count followed by diverting the record somewhere it can be inspected, so progress resumes and nothing is lost.
Full entry →Quarantine table
rows that fail validation go to a side table with the reason attached instead of failing the load or being silently dropped.
A destination for records that could not be processed, retained with enough context to diagnose them. It is the third option between stopping the pipeline and losing data quietly, and it is what makes 'we processed 99.4 percent of rows' a statement someone can act on. It only works if something watches its row count, otherwise it is a place data goes to be forgotten with extra steps.
Full entry →