Comparison
Logical datevsSchedule interval
Logical date
the run that started at 02:00 on Tuesday is processing Monday's data, and half the confusion in the team is about which date the run is named after.
The period a run is processing, as distinct from the moment it executed. Keeping the two apart is what makes runs re-runnable: parameterise every query on the logical date and a backfill is the same code with a different value. Every pipeline that hardcodes 'yesterday' as a function of the current clock has conflated them, and it will produce different results when re-run, which is the bug that makes backfills untrustworthy.
Full entry →Schedule interval
the job is set to hourly, takes seventy minutes, and now two copies are running at once.
How often a pipeline is triggered by the clock. It is the crudest dependency model and still the most common, and its failure is overlap: an interval shorter than the runtime silently turns into concurrent runs unless something prevents it. Time-based scheduling also encodes an assumption that upstream data has arrived by then, which is the assumption that breaks whenever the upstream is a few minutes late.
Full entry →