WordPiece
Definition
WordPiece breaks words into reusable text pieces using a limited vocabulary.
A subword tokenization method that uses a limited vocabulary of frequent character sequences and segments words.
Simply explained
Long or unknown words are broken into smaller known pieces. With this toolkit, the model can further process the text.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A multilingual classifier checks segmentation of article numbers and compound technical terms.
Limits and pitfalls
- Implementations differ; do not equate WordPiece with BPE and never separate tokenizer from model vocabulary.
Origin & history
WordPiece originated in speech recognition and became known through BERT for neural language models.
Use in mid-sized companies
- Possible example: A multilingual classifier checks segmentation of article numbers and compound technical terms.
Frequently asked questions
What does WordPiece mean?
A subword tokenization method that uses a limited vocabulary of frequent character sequences and segments words. What should you watch out for? Implementations differ; do not equate WordPiece with BPE and never separate the tokenizer from the model vocabulary.
Sources
- Google Neural Machine Translation · Wu et al., Google Neural Machine Translation
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: