Comparison
Data-aware schedulingvsSchedule interval
Data-aware scheduling
the model runs when the table it depends on is actually updated, rather than at half past two because that is usually late enough.
Triggering work from the state of data rather than from the clock — a partition appearing, a table being written. It removes the guesswork of choosing an offset and makes the dependency graph match the real one, which is what makes lineage and scheduling the same object. The requirement it creates is that every producer must reliably announce completion, which is a coordination problem rather than a technical one.
Full entry →Schedule interval
the job is set to hourly, takes seventy minutes, and now two copies are running at once.
How often a pipeline is triggered by the clock. It is the crudest dependency model and still the most common, and its failure is overlap: an interval shorter than the runtime silently turns into concurrent runs unless something prevents it. Time-based scheduling also encodes an assumption that upstream data has arrived by then, which is the assumption that breaks whenever the upstream is a few minutes late.
Full entry →