Comparison
Compression codecvsDictionary encoding
Compression codec
someone switched the codec and the files got a third smaller while the queries got slightly slower to decode.
The algorithm compressing each column chunk, chosen per table and traded off between file size and CPU on every read. On a per-byte-scanned billing model the smaller file is directly cheaper, so the choice is a money decision as much as a performance one. It is worth knowing as a lever because it is one of the few that can be changed by rewriting files, with no consumer needing to know.
Full entry →Dictionary encoding
the column holds four distinct country names in eleven million rows, and the file stores small integers plus a lookup.
Storing repeated values once and referring to them by a small integer. It is why a low-cardinality string column costs almost nothing in a columnar file, and it is a compounding reason to sort data before writing it. Engines can also filter directly against the dictionary, so it makes some predicates cheap without decoding the column at all.
Full entry →