Data engineering·Quality, tests and knowing it is wrong
you write down that this column should be between zero and one hundred and that ninety-nine percent of rows should have an email, and it is checked automatically.
Data expectation
Also calledexpectation, constraint check
A declarative statement of what data should look like, expressed as a rule and evaluated against a dataset. Writing expectations is where most of the value is, because the act of stating them surfaces assumptions nobody had said out loud. The recurring practical problem is thresholds: an expectation with no tolerance fails on the first legitimate outlier and is then disabled forever.