Tokenization
Definition
Tokenization breaks inputs into units that a model can process.
The division of text or other inputs into discrete units processed by a model or search system.
Simply explained
A text is broken into small pieces. These can be words or word parts with which the system continues to work.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A multilingual search service checks how its tokenizer handles article numbers and compound words.
Limits and pitfalls
- Token count does not equal word count; normalization, language, and unknown characters influence costs and quality.
Origin & history
Tokenization evolved from word and character methods to learnable subword methods like BPE and Unigram.
Use in mid-sized companies
- Possible application: A multilingual search service checks how its tokenizer handles article numbers and compound words.
Frequently asked questions
What does tokenization mean?
The division of text or other inputs into discrete units processed by a model or search system. What should you watch out for? Token count does not equal word count; normalization, language, and unknown characters influence costs and quality.
Sources
- SentencePiece · Kudo and Richardson, SentencePiece
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: