Model Vocabulary
Definition
The model vocabulary defines which text units a tokenizer and its model use jointly.
The finite set of discrete token IDs and mappings that a tokenizer and model use jointly.
Simply explained
The model has a fixed toolkit of text pieces. Each piece has a number that the model works with.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A domain-specific model extends its vocabulary only with subsequently adapted embeddings and compatibility tests.
Limits and pitfalls
- A dictionary is not a model vocabulary; changes break token IDs, checkpoints, and cost comparisons.
Origin & history
Subword tokenization made learnable vocabularies for open word sets common in neural language models.
Use in mid-sized companies
- Possible example: A domain-specific model extends its vocabulary only with subsequently adapted embeddings and compatibility tests.
Frequently asked questions
What does the vocabulary of a model mean?
The finite set of discrete token IDs and mappings that a tokenizer and model use jointly. What should one watch out for? A dictionary is not a model vocabulary; changes break token IDs, checkpoints, and cost comparisons.
Sources
- SentencePiece · Kudo and Richardson, SentencePiece
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: