Data engineering·Files, formats and table layers
each message carries a schema id rather than field names, and adding an optional field does not break the consumers already running.
Avro
Also calledApache Avro
A compact row-oriented binary format with a first-class schema and well-defined compatibility rules for evolving it. It is the standard on the wire — change feeds, message payloads, anything appended one record at a time — precisely where columnar formats are wrong. Its resolution rules, reading old data with a new schema and vice versa, are why it is bound up with schema registries in practice.