Data engineering·Moving data out of the source
the job stores the largest updated_at it has seen and asks for everything greater than that next time.
High-water mark
Also calledwatermark column, incremental cursor, bookmark
The stored position an incremental load resumes from — usually a timestamp or a monotonically increasing id. Its correctness rests on an assumption about the source that nobody wrote down: that the column only ever moves forwards, and that a row is visible to your reader by the time its timestamp says it exists. Both are routinely false in a source with long transactions, which is where the missing-rows bug comes from.