Multi-Teacher Distillation
Definition
A smaller model learns from the outputs of multiple teacher models.
The training targets combine knowledge from various teachers, for example through averaged probabilities or selected answers. The student model is intended to adopt useful properties.
Simply explained
A student learns from multiple teachers with different strengths.
Why it matters in practice
This way, knowledge from an ensemble can be transferred into a single model.
Example from business
Hypothetical example: Two specialized classifiers generate training targets for a shared small model.
Limits and pitfalls
- Contradictions and errors of the teacher models can be transferred. The weighting of their contributions is crucial.
Origin & history
The approach extends knowledge distillation and the transfer of ensemble knowledge.
Use in mid-sized companies
- SMEs with many recurring tasks can check whether a small specialized model suffices.
Frequently asked questions
Does the student automatically become better than all teachers?
No, this is not a guaranteed property.
Sources
- Distilling the Knowledge in a Neural Network — · Hinton et al.: Distilling the Knowledge in a Neural Network —
Editorially reviewed on: