Comparison
Data pipelinevsOrchestrator
Data pipeline
the report was wrong and the reason turned out to be six hops upstream in something nobody on your team owns.
A chain of steps that moves data from where it is produced to where it is asked questions of, transforming it on the way. The word covers everything from a nightly script to a stream topology, and the shared property is that each step depends on the last, so failure and staleness travel forwards. What makes pipelines distinctive to run is that they are long, they are asynchronous, and nobody is sitting in front of one waiting — a break is discovered by its output being wrong rather than by anything going red.
Full entry →Orchestrator
something has to decide that the transform runs only after all four extracts finished, and retry the one that did not.
The system that decides what runs, in what order, when, and what happens when a step fails. It is the piece teams build themselves first and regret, because the hard parts are not scheduling but dependency resolution, retries, backfills, concurrency limits and a usable view of history. Choosing one is choosing a model of dependencies — time-based, task-based or data-based — and that model is much harder to change later than the tool is.
Full entry →