Data engineering·Orchestration and running it in anger
the run that started at 02:00 on Tuesday is processing Monday's data, and half the confusion in the team is about which date the run is named after.
Logical date
Also calledexecution date, data interval, as-of date
The period a run is processing, as distinct from the moment it executed. Keeping the two apart is what makes runs re-runnable: parameterise every query on the logical date and a backfill is the same code with a different value. Every pipeline that hardcodes 'yesterday' as a function of the current clock has conflated them, and it will produce different results when re-run, which is the bug that makes backfills untrustworthy.