Technical Terms
Tokenisation
Definition
The process of breaking text into tokens before feeding it to a language model. Different models use different tokenisation schemes — some split by word parts (subword tokenisation), while others use byte-pair encoding (BPE).
In Plain English
How AI chops up your text into pieces it can understand. Like breaking a sentence into individual building blocks. The word "unhappiness" might become "un", "happiness" — two tokens. Different AI models chop text up differently.