Skip to content
    Flow-alerta.cab

    Knowledge Distillation

    AI fundamentals & modelsModel distillation
    Editorially reviewed on: 1 source
    Your reading view

    The short answer · 18 words

    In short

    A student model learns from issues or representations of a teacher model, such as compressing an AI model.

    Flow-alerta.cab (2026): Knowledge Distillation. https://flow-alerta.cab/en/glossar/knowledge-distillation. Editorially reviewed on: 09 October 2026.

    How much detail would you like?

    Practical

    Relevant if model size, duration or operating costs are limited; evaluate quality and overall effort together.

    Possible example: A company checks whether a smaller support model learns sufficiently good answers with examples of a larger model.

    Cost reduction in cloud inference, on-premise operation of lean AI models, and mobile inference.

    At a glance

    Core
    In the teacher-student process, information from a teacher model or ensemble is used for training a student model, such as probability distributions. A smaller model can be cheaper to operate, but does not automatically reach the quality of the teacher.
    Use
    Relevant if model size, duration or operating costs are limited; evaluate quality and overall effort together.
    Example
    Possible example: A company checks whether a smaller support model learns sufficiently good answers with examples of a larger model.
    Limits
    Loss of generalization capability and 'out-of-domain' knowledge; requires high processing power during the distillation process.

    Questions about this term

    What distinguishes distillation from quantization?

    Distillation transfers knowledge through training; quantization reduces the numerical precision of presentation.

    Does the student model have to be smaller?

    No. Compression is a common purpose, but not the only form of knowledge transfer.

    All details and background

    In the teacher-student process, information from a teacher model or ensemble is used for training a student model, such as probability distributions. A smaller model can be cheaper to operate, but does not automatically reach the quality of the teacher.

    ELI5 – Explained simply

    An experienced teacher shows a student how to solve tasks. The student can learn from this, but does not automatically assume all skills.

    Why it matters in practice

    Relevant if model size, duration or operating costs are limited; evaluate quality and overall effort together.

    Example

    Possible example: A company checks whether a smaller support model learns sufficiently good answers with examples of a larger model.

    Pitfalls and limits

    • Loss of generalization capability and 'out-of-domain' knowledge; requires high processing power during the distillation process.

    Origin & history

    Hinton, Vinyals and Dean describe an influential approach to transferring knowledge to a single model in 2015.

    Use in mid-sized companies

    • Cost reduction in cloud inference, on-premise operation of lean AI models, and mobile inference.

    Related terms

    • Model compression
    • Quantization
    • Teacher-student network
    • LLM Fine-tuning

    Matching Flow-alerta service

    AI opportunity analysis & roadmapAI Potential Analysis & Roadmap – Evaluating Model Requirements and Operational Expenditure

    Matching tool / use case

    View use casesUse Case: Evaluate a student model versus the teacher model on your own assignments.

    Sources

    • arxiv.orgHinton et al. (2015), Distilling the Knowledge in a Neural Network

    Quellen geprüft am 09.10.2026.

    Editorially reviewed on:

    Next step

    Put knowledge into practice

    Discuss your projectView use cases