Comparison
Kappa architecturevsLambda architecture
Kappa architecture
there is one streaming job, and correcting history means replaying the log through it rather than running a different batch job.
Doing away with the separate batch path and treating reprocessing as replaying the same stream code over retained history. One implementation, one definition of every metric, and a reprocessing story that depends entirely on retention being long enough and the job being fast enough to catch up. It is the right default when your log keeps history; it is a trap when the log keeps seven days and someone asks for a two-year restatement.
Full entry →Lambda architecture
there are two implementations of the same metric, one fast and approximate and one slow and correct, and they disagree by two percent.
Running a streaming path for fresh approximate results alongside a batch path for correct ones, and serving a merge of the two. It solved a real problem when stream engines could not be trusted with correctness, and its permanent cost is that every business rule is now written twice in two languages by two people. When someone reports that the live figure and the daily figure disagree, this shape is usually why.
Full entry →