Data engineering·Orchestration and running it in anger
something has to decide that the transform runs only after all four extracts finished, and retry the one that did not.
Orchestrator
Also calledscheduler, workflow engine
The system that decides what runs, in what order, when, and what happens when a step fails. It is the piece teams build themselves first and regret, because the hard parts are not scheduling but dependency resolution, retries, backfills, concurrency limits and a usable view of history. Choosing one is choosing a model of dependencies — time-based, task-based or data-based — and that model is much harder to change later than the tool is.