Data engineering·Orchestration and running it in anger
the first step of the pipeline does nothing but sit there checking every minute whether the file has landed.
Sensor
Also calledpoke, wait-for task, file sensor
A task whose job is to wait for an external condition — a file, a partition, another team's table — before the rest proceeds. It replaces 'schedule it an hour later and hope', which is the arrangement that fails on the day the upstream is slow. Sensors are cheap to write and expensive to run badly: one that holds a worker slot while polling is how a scheduler deadlocks itself.