AI Hardware & Software Stack
Which describes knowledge distillation?
- AHeating model weights to evaporate and shrink them
- BWashing the training data with distilled water
- CTraining a small student model to mimic a large teacher model's outputs so the small model achieves similar performance
Show answer and explanation
Answer: C. Training a small student model to mimic a large teacher model's outputs so the small model achieves similar performance
In knowledge distillation, a small model (the student) is trained to match outputs of an already trained large model (the teacher), such as its probability distribution over answers. Learning not just the correct labels but also how likely the teacher considered each answer lets the small model approach the original's performance. It is used alongside quantization and pruning to reduce inference compute and memory.
- A. It is not a physical process.
- B. It has nothing to do with chemical distillation.
- C. Correct. It reduces inference compute and memory.