Core Concepts
Token
Definition
The basic chunk of text that language models read and generate, roughly a word fragment rather than a full word.
In Plain English
The small pieces of text AI counts, reads, and bills you for.
The basic chunk of text that language models read and generate, roughly a word fragment rather than a full word.
The small pieces of text AI counts, reads, and bills you for.