LLM-as-a-Judge
Definition
LLM-as-a-Judge uses a language model to evaluate outputs from other systems based on a scoring rubric.
The method can generate pairwise comparisons, scores, or justifications and scale evaluation. However, LLM-Judges exhibit biases, including position, verbosity, or self-preference effects.
Simply explained
How to deploy an AI evaluator that judges answers according to a rubric – whose judgments themselves must be verified.
Why it matters in practice
Relevant for quick evaluations of open text tasks when purely automatic metrics are insufficient.
Example from business
A team has two answer variants rated for factual accuracy, completeness, and tone, and regularly checks a sample manually.
Limits and pitfalls
- Treat judge score as ground truth; have the same model self-evaluate; do not randomize order; rubric unclearly formulated.
Origin & history
Zheng et al. systematically studied strong LLMs as evaluators in 2023 with MT-Bench and Chatbot Arena and documented multiple types of bias.
Use in mid-sized companies
- Possible applications: prompt/model comparison, regression testing, and preselection for human evaluation.
Frequently asked questions
Does LLM-as-a-Judge replace humans?
No. How to reduce bias? Clear rubrics, order randomization, reference examples, and human calibration.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Sources checked on 10.10.2026.
Editorially reviewed on: