What is Knowledge distillation?
Training a small student model to mimic a large teacher model's outputs so the small model achieves similar performance.
Field: AI Hardware & Software Stack
Knowledge distillation in the explanations
In knowledge distillation, a small model (the student) is trained to match outputs of an already trained large model (the teacher), such as its probability distribution over answers.
Which describes knowledge distillation?
Questions that mention Knowledge distillation (1)
Related terms
Pruning AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph Data parallelism