MT-Bench
Definition
MT-Bench evaluates answers over multiple conversation rounds.
The benchmark uses multi-stage tasks and model-based evaluation to examine dialogue capabilities. Follow-up questions test how well context and instructions are considered.
Simply explained
Not only the first answer counts, but also the appropriate continuation.
Why it matters in practice
It complements tests that only consider individual questions.
Example from business
Hypothetical example: A team compares how models improve a draft after a follow-up question.
Limits and pitfalls
- An evaluating model can have preferences and errors. Prompt and evaluation model must be documented.
Origin & history
MT-Bench was introduced in 2023 in the context of FastChat and Chatbot Arena.
Use in mid-sized companies
- SMEs can adopt the idea for their own multi-stage service cases.
Frequently asked questions
Is the evaluation judge objective?
No, its judgments require contextualization and spot checks.
Sources
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — · Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena —
Editorially reviewed on: