Self-Distillation
Definition
Self-distillation lets a model learn from its own or closely related predictions.
A variant of knowledge distillation where a model learns from predictions, intermediate representations, or an earlier training state of the same or a closely related model.
Simply explained
A model uses, for example, the answers from its earlier training state as additional learning aid.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
An image model additionally learns from the soft predictions of a previous training state.
Limits and pitfalls
- Treating teacher and student as independent evidence, equating self-distillation with self-training on pseudo-labels, or attributing improvements without identical data and training control.
Origin & history
Self-distillation was studied in the late 2010s as a variant of knowledge distillation.
Use in mid-sized companies
- Possible example: An image model additionally learns from the soft predictions of a previous training state.
Frequently asked questions
What does self-distillation mean?
A variant of knowledge distillation where a model learns from predictions, intermediate representations, or an earlier training state of the same or a closely related model. What mistakes should be avoided? Treating teacher and student as independent evidence, equating self-distillation with self-training on pseudo-labels, or attributing improvements without identical data and training control.
Sources
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: