Knowledge Distillation Method Comparison

Artificial Intelligence (AI) has increasingly influenced modern society, recently in particular through significant advancements in Large Language Models (LLMs). However, high computational and storage demands of LLMs still limit their deployment in resource-constrained environments. Knowledge distillation addresses this challenge by training a smaller language model (student) from a larger one (teacher). Previous research has introduced several distillation methods for both generating training data and training the student model. Despite their relevance, the effects of state-of-the-art distillation methods on model performance and explainability have not been thoroughly investigated and compared. In our work, we enlarge the set of available methods by applying Critique-Revision Prompting to distillation and by synthesizing existing methods. For these methods, we provide a systematic comparison based on a popular commonsense question-answering dataset. While we measure performance via student model accuracy, we employ a humangrounded study to evaluate explainability. We contribute new distillation methods and their comparison in terms of both performance and explainability.