Technical Terms
Tokenizer
Definition
The model-specific system that turns text into tokens and, in some workflows, turns tokens back into text.
In Plain English
The thing that decides how your prompt gets broken into billable pieces.
The model-specific system that turns text into tokens and, in some workflows, turns tokens back into text.
The thing that decides how your prompt gets broken into billable pieces.