Applied AI·Tokens
you notice a common word costs one token and a rare surname costs five, because frequent pairs got merged and yours was left in pieces.
Byte pair encoding (BPE)
Also calledBPE
Draft summary, pending review
The dominant tokenisation algorithm. It starts from raw bytes and repeatedly merges the most frequent adjacent pairs, so common words become single tokens while rare words split into pieces. You never implement it, but the name comes up constantly.