Comparison
DAGvsOrchestrator
DAG
the pipeline is drawn as boxes with arrows, nothing points backwards, and a step waits for everything pointing at it.
The dependency graph of a pipeline: steps as nodes, dependencies as edges, no cycles. Acyclicity is what makes a run finite and orderable, and it is why a pipeline that genuinely needs a loop has to express it as a sequence of runs instead. In practice the word is used loosely for the pipeline itself, so 'the DAG failed' means a run failed rather than anything about graph theory.
Full entry →Orchestrator
something has to decide that the transform runs only after all four extracts finished, and retry the one that did not.
The system that decides what runs, in what order, when, and what happens when a step fails. It is the piece teams build themselves first and regret, because the hard parts are not scheduling but dependency resolution, retries, backfills, concurrency limits and a usable view of history. Choosing one is choosing a model of dependencies — time-based, task-based or data-based — and that model is much harder to change later than the tool is.
Full entry →