Comparison
Data lineagevsPipeline dependency inversion
Data lineage
the number is wrong and you need to know every table between it and the source, without reading forty SQL files.
The recorded graph of which datasets are derived from which, ideally parsed from the transformation code rather than maintained by hand. It is what makes both directions of an incident tractable: upstream to find the cause, downstream to find who else is now wrong. Hand-maintained lineage is worse than none, because it is trusted and stale at the same time.
Full entry →Pipeline dependency inversion
two pipelines have no declared relationship and one of them only works because the other happens to run first.
A real data dependency that exists in the SQL but not in the orchestrator's graph, so ordering is a coincidence of scheduling. It survives for months because the timing usually holds, and it breaks the day one job gets slower. Deriving the graph from the code rather than maintaining it by hand is the structural fix, which is most of the argument for data-aware scheduling.
Full entry →