Comparison
Data-aware schedulingvsSensor
Data-aware scheduling
the model runs when the table it depends on is actually updated, rather than at half past two because that is usually late enough.
Triggering work from the state of data rather than from the clock — a partition appearing, a table being written. It removes the guesswork of choosing an offset and makes the dependency graph match the real one, which is what makes lineage and scheduling the same object. The requirement it creates is that every producer must reliably announce completion, which is a coordination problem rather than a technical one.
Full entry →Sensor
the first step of the pipeline does nothing but sit there checking every minute whether the file has landed.
A task whose job is to wait for an external condition — a file, a partition, another team's table — before the rest proceeds. It replaces 'schedule it an hour later and hope', which is the arrangement that fails on the day the upstream is slow. Sensors are cheap to write and expensive to run badly: one that holds a worker slot while polling is how a scheduler deadlocks itself.
Full entry →