Online Distillation
Definition
Online distillation generates teacher and student signals within the same training process.
Distillation where teacher and student signals emerge jointly during the same training process or models learn from each other.
Simply explained
Multiple models learn from each other during training instead of following only a pre-trained teacher.
Why it matters in practice
Helps realistically assess model quality, limits, and technical decisions.
Example from business
Multiple models exchange soft target distributions during training.
Limits and pitfalls
- Confusing online distillation with streaming learning or ignoring unstable mutual error amplification.
Origin & history
Deep Mutual Learning from 2018 is a well-known approach without a fixed pre-trained teacher model.
Use in mid-sized companies
- Possible application: Multiple models exchange soft target distributions during training.
Frequently asked questions
What does online distillation mean?
Distillation where teacher and student signals emerge jointly during the same training process or models learn from each other.
Which mistakes should be avoided?
Confusing online distillation with streaming learning or ignoring unstable mutual error amplification.
Sources
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: