When people talk about an AI model’s limits or usage, they often talk about tokens. A token is the unit the model reads and generates. It is not a synonym for “word”: it can be a whole word, part of one, punctuation, or even a single character.
Text is encoded in pieces
Before a language model processes text, a tokenizer splits it into pieces and maps each piece to an ID. A familiar word may fit into one piece. A rare, long, or unusual word may be split into smaller pieces. For example, “unhappiness” could be split into parts like “un”, “happi”, and “ness”; the exact split depends on the tokenizer.
Spaces, punctuation, capitalization, and the text around a word can also affect the split. And different models may use different tokenizers, so the same sentence can have different token counts.
Why not make every word a token?
A vocabulary made from whole words would have to account for every spelling, inflection, name, typo, and new term. That vocabulary would grow quickly, and an unseen word could be difficult to represent. Subword tokenization gives a model a middle ground: common text can use familiar larger pieces, while less common text can be assembled from smaller ones.
What this approach helps with
Breaking text into reusable pieces gives models a practical representation for both language they see often and forms they see less often. It also gives the system a consistent sequence to process, whatever the original text looked like. In practice, that helps with:
- A manageable vocabulary: reusable pieces can represent many words without storing every complete word as its own entry.
- Handling unfamiliar text: a rare word or new name can often be represented as smaller known pieces instead of being entirely unknown.
- A measurable context budget: models process sequences of tokens, so token counts help determine how much input and output fits in a context window.
- A shared text pipeline: words, punctuation, and other text can all be converted into the same kind of model input.
The useful caveat
Token counts are an engineering measure, not a measure of meaning or writing quality. They also vary by model, language, and spelling. A rough English estimate can help with planning, but use the target model’s tokenizer when the exact count matters.