Knowledge Distillation Experiments: Comparative Analysis of Teacher–Student Architectures

This repository contains a series of reproducible Jupyter notebooks demonstrating and comparing different knowledge distillation strategies across multiple neural network architectures.The experiments explore how student networks can effectively learn compact representations from high-capacity teacher models, focusing on trade-offs in accuracy, compression ratio, and inference speed. Contents experiment-1.ipynb: Baseline training of teacher and student models on benchmark dataset. experiment-2.ipynb: Soft-target distillation with temperature scaling and loss weighting. experiment-3.ipynb: Layer-wise feature distillation and intermediate supervision. experiment-3a.ipynb: Cross-architecture distillation using lightweight convolutional students. Each notebook includes preprocessing steps, training configurations, hyperparameter tuning, and performance evaluation metrics.The experiments can be extended or adapted for transfer learning, model compression, or deployment optimization studies. Highlights Implements classical and modern KD methods (e.g., Hinton et al. 2015; feature-based KD). Provides consistent experimental environment in Jupyter notebooks. Enables reproducibility and further fine-tuning for lightweight deployment.