Comparison
AvrovsParquet
Avro
each message carries a schema id rather than field names, and adding an optional field does not break the consumers already running.
A compact row-oriented binary format with a first-class schema and well-defined compatibility rules for evolving it. It is the standard on the wire — change feeds, message payloads, anything appended one record at a time — precisely where columnar formats are wrong. Its resolution rules, reading old data with a new schema and vice versa, are why it is bound up with schema registries in practice.
Full entry →Parquet
the files are columnar with statistics in the footer, so the engine can skip whole chunks without reading them.
The de facto columnar file format for analytics: typed columns, per-chunk statistics, dictionary and run-length encoding, and a footer describing the layout. Its statistics are what make predicate pushdown work, which is why the same query can be fast or slow on identical data depending on how the files were written and sorted. It is a vendor-neutral name people say as a noun, which is why it is here and a console feature is not.
Full entry →