Data engineering·Orchestration and running it in anger
the job is set to hourly, takes seventy minutes, and now two copies are running at once.
Schedule interval
Also calledcron schedule, run cadence
How often a pipeline is triggered by the clock. It is the crudest dependency model and still the most common, and its failure is overlap: an interval shorter than the runtime silently turns into concurrent runs unless something prevents it. Time-based scheduling also encodes an assumption that upstream data has arrived by then, which is the assumption that breaks whenever the upstream is a few minutes late.