Comparison
Data pipelinevsPipes and filters
Data pipeline
the report was wrong and the reason turned out to be six hops upstream in something nobody on your team owns.
A chain of steps that moves data from where it is produced to where it is asked questions of, transforming it on the way. The word covers everything from a nightly script to a stream topology, and the shared property is that each step depends on the last, so failure and staleness travel forwards. What makes pipelines distinctive to run is that they are long, they are asynchronous, and nobody is sitting in front of one waiting — a break is discovered by its output being wrong rather than by anything going red.
Full entry →Pipes and filters
each stage takes one thing and returns another, knows nothing about its neighbours, and you reordered two of them safely.
Decomposing processing into independent stages connected by a uniform channel, so each stage transforms and passes along. Unix pipelines, compilers, ETL jobs and stream processors are all this shape, and its virtue is that stages compose, reorder and parallelise because they share only the data format. The constraint is that shared format: anything a stage needs that the pipe does not carry has to be smuggled, and that is where the design breaks down.
Full entry →