Comparison
Manual interventionvsZombie task
Manual intervention
someone marked the failed task as successful at 03:40 so the rest of the pipeline could proceed, and did not write down why.
Overriding a pipeline's state by hand — clearing a task to re-run it, forcing a run past a failure, skipping a step. It is a necessary tool and an undocumented state change to a production system, and the two facts sit badly together. Treating it like a deploy — recorded, attributed, with a reason — is what stops next month's investigation ending at 'the data is odd for one day in March and nobody knows why'.
Full entry →Zombie task
the scheduler thinks the task is running, the worker that was running it died an hour ago, and nothing has noticed.
A task the orchestrator believes is alive but whose process is gone, usually after a worker crash or an evicted container. It is the failure that does not fail: no error, no retry, just a run that never finishes and a dependency chain that never advances. Every orchestrator has a heartbeat mechanism for detecting it, and the timeout on that mechanism is the real bound on how long a pipeline can be silently stuck.
Full entry →