Comparison
TokenisationvsTokeniser
Tokenisation
the card number is replaced by a meaningless token, and the only system that can turn it back is one you deliberately cannot query from the warehouse.
Replacing a sensitive value with a surrogate that has no mathematical relationship to it, keeping the mapping in a separate vault. Because the token carries no information, the analytical system can join and count on it without ever holding the real value. It is stronger than encryption for this purpose precisely because there is no key in the analytical environment that could reverse it.
Full entry →Tokeniser
the same string cost a different number of tokens on the other provider, because each model family ships its own.
The component that converts text into token IDs before the model sees it, and back into text after. Each model family has its own tokeniser, so the same string produces different token counts on different models.
Full entry →