Data pipeline

A chain of steps that moves data from where it is produced to where it is asked questions of, transforming it on the way. The word covers everything from a nightly script to a stream topology, and the shared property is that each step depends on the last, so failure and staleness travel forwards. What makes pipelines distinctive to run is that they are long, they are asynchronous, and nobody is sitting in front of one waiting — a break is discovered by its output being wrong rather than by anything going red.