Knowledge Distillation for Deep Learning-Based Hydrological Prediction and Forecasting

Sample results and model weights are provided in this repo: Template code for KD can be found at: https://github.com/sinajahangir/KD-DL-LSTM The random numbers folder contains CSV files indicating 8-fold cross-validation catchments. Results (NSE and KGE) for model compression and generalization are provided Ensemble model weights are provided. These weights can be used for fine-tuning or KD This study demonstrates how a technique known as knowledge distillation can enhance the predictive capabilities of deep learning models, even when a smaller model is employed or when the available input data is noisy. Deep learning models, such as long short-term memory networks, work well for hydrological prediction. However, they typically require a substantial amount of high-quality input data to perform optimally, which is not always available. Additionally, averaging over many deep learning models often improves performance, but this increases computational costs during operational use. Knowledge distillation helps by training a smaller or simpler "student" model to learn from a more accurate "teacher" model (or group of models) rather than learning only from the raw data. Overall, this approach makes deep learning models more efficient and transferable. The proposed method is helpful in applications where data is poor or incomplete, or when computational costs are limiting.本仓库提供了示例结果与模型权重： 知识蒸馏（Knowledge Distillation, KD）的模板代码可从以下链接获取：https://github.com/sinajahangir/KD-DL-LSTM `random numbers` 文件夹内含CSV格式文件，用于标注8折交叉验证的集水区。 本仓库提供了模型压缩与泛化性能的评估结果（包含纳什效率系数（Nash-Sutcliffe Efficiency, NSE）与克林-古普塔效率系数（Kling-Gupta Efficiency, KGE））。 此外还提供了集成模型权重，此类权重可用于微调或知识蒸馏（KD）。 本研究阐明了知识蒸馏技术如何提升深度学习模型的预测性能，即便在使用小型模型或输入数据存在噪声的场景下依然有效。长短期记忆网络（Long Short-Term Memory, LSTM）等深度学习模型在水文预测任务中表现优异，但这类模型通常需要大量高质量输入数据才能达到最优性能，而此类数据并非总能获取。此外，对多个深度学习模型进行集成平均通常能提升性能，但会增加实际部署阶段的计算开销。知识蒸馏通过训练规模更小或结构更简单的「学生」模型，使其从精度更高的「教师」模型（或模型组）中学习，而非仅依赖原始数据进行训练。综上，该方法可使深度学习模型兼具更高的运行效率与更强的可迁移性。所提出的方法在数据质量较差、数据不完备，抑或计算成本受限的应用场景中具有重要实用价值。