Comparison
DeterministicvsLogical date
Deterministic
you run it twice with the same input and get byte-identical output, because nothing inside it reads the clock or a random number.
The property that the same inputs always produce the same output. It is about reproducibility, not about repeat-safety: a deterministic function can still be unsafe to run twice if it has side effects, and an idempotent operation can be non-deterministic in what it returns. Determinism is what makes replay, caching and testing possible.
Full entry →Logical date
the run that started at 02:00 on Tuesday is processing Monday's data, and half the confusion in the team is about which date the run is named after.
The period a run is processing, as distinct from the moment it executed. Keeping the two apart is what makes runs re-runnable: parameterise every query on the logical date and a backfill is the same code with a different value. Every pipeline that hardcodes 'yesterday' as a function of the current clock has conflated them, and it will produce different results when re-run, which is the bug that makes backfills untrustworthy.
Full entry →