Towards Understanding Ensemble, Knowledge Distillation, and Self-Distillation in Deep Learning

Ref

• Zeyuan Allen-Zhu’s Home Page (mit.edu)
• Three mysteries in deep learning: Ensemble, knowledge distillation, and self-distillation - Microsoft Research