BERTScore
The short answer · 20 words
In short
Automatic evaluation metric for generated text that compares token similarities based on contextual embeddings instead of just exact word matches.
Flow-alerta.cab (2026): BERTScore. https://flow-alerta.cab/en/glossar/bertscore. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Helpful for translation, summary or text generation, if pure character or word hits are too strict. It shall not replace professional or human quality checks.
Example scenario: Two summaries are evaluated against reference texts, although they use different but semantically similar formulations.
Use as one of several metrics for generative systems and combine them with technical criteria, source testing and real-world examples.
At a glance
- Core
- BERTScore maps tokens from candidate and reference text with a pre-trained contextual language model and compares their embeddings. From this, precision, recall and F1-like values are calculated. The metric can take into account semantically similar formulations better than pure n-gram metrics.
- Use
- Helpful for translation, summary or text generation, if pure character or word hits are too strict. It shall not replace professional or human quality checks.
- Example
- Example scenario: Two summaries are evaluated against reference texts, although they use different but semantically similar formulations.
- Limits
- Interpret BERTScore as a truth check; compare different backbone models directly; consider only one score without task reference; completely replace human evaluation.
In context
BERT
2018 Google introduced Transformer Encoder model family for pre-trained, bidirectional voice representations.
Embedding
An embedding maps content as a number vector to compare similarities.
BLEU Score
Automatic metric for evaluating machine-generated translations based on the match of n-grams with reference translations.
Questions about this term
Does BERTScore Measure Factuality?
No.
Does it need reference texts?
In the classical application, yes.
Is a higher value always better for business?
No, only in the context of the same evaluation.
All details and background
BERTScore maps tokens from candidate and reference text with a pre-trained contextual language model and compares their embeddings. From this, precision, recall and F1-like values are calculated. The metric can take into account semantically similar formulations better than pure n-gram metrics.
ELI5 – Explained simply
Two texts do not have to use the same words: BERTScore checks whether their contextual representations are similar.
Why it matters in practice
Helpful for translation, summary or text generation, if pure character or word hits are too strict. It shall not replace professional or human quality checks.
Example
Example scenario: Two summaries are evaluated against reference texts, although they use different but semantically similar formulations.
Pitfalls and limits
- Interpret BERTScore as a truth check; compare different backbone models directly; consider only one score without task reference; completely replace human evaluation.
Origin & history
Zhang and co-authors proposed BERTScore 2019. The work describes the evaluation of generated text based on contextual token representations.
Use in mid-sized companies
- Use as one of several metrics for generative systems and combine them with technical criteria, source testing and real-world examples.
Related terms
- BERT
- Embedding
- BLEU score
- ROUGE score
- F1 score
Matching Flow-alerta service
AI opportunity analysis & roadmapAI consulting: AI potential analysis & roadmap on tasks, data and prerequisites of a possible first AI use case.Matching tool / use case
View use casesUse Case: supplementary automatic evaluation of summaries or translations.Sources
- arxiv.orgZhang et al., BERTScore
Quellenprüfung: 09.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.
Editorially reviewed on: