Top-1 errors of ResNet20.

To break the three lockings during backpropagation (BP) process for neural network training, multiple decoupled learning methods have been investigated recently. These methods either lead to significant drop in accuracy performance or suffer from dramatic increase in memory usage. In this paper, a new form of decoupled learning, named decoupled neural network training scheme with re-computation and weight prediction (DTRP) is proposed. In DTRP, a re-computation scheme is adopted to solve the memory explosion problem, and a weight prediction scheme is proposed to deal with the weight delay caused by re-computation. Additionally, a batch compensation scheme is developed, allowing the proposed DTRP to run faster. Theoretical analysis shows that DTRP is guaranteed to converge to crical points under certain conditions. Experiments are conducted by training various convolutional neural networks on several classification datasets, showing comparable or better results than the state-of-the-art methods and BP. These experiments also reveal that adopting the proposed method, the memory explosion problem is effectively solved, and a significant acceleration is achieved.

为破解神经网络训练中反向传播（Backpropagation, BP）过程的三类锁定难题，近年来已有诸多解耦学习（decoupled learning）方法被提出并研究。但此类方法要么会造成模型精度的显著下降，要么会引发内存占用的大幅攀升。本文提出一种新型解耦学习范式——带重计算与权重预测的解耦神经网络训练方案（Decoupled Neural Network Training Scheme with Re-computation and Weight Prediction, DTRP）。在DTRP框架中，研究采用重计算机制以解决内存爆炸问题，同时提出权重预测方案以应对重计算带来的权重延迟问题。此外，本文还开发了批次补偿机制，可进一步提升所提DTRP的运行效率。理论分析表明，在特定条件下，DTRP可收敛至临界点。实验部分通过在多个分类数据集上训练各类卷积神经网络（Convolutional Neural Networks, CNN）展开，结果显示其性能可与当前最优方法及BP算法媲美，甚至更优。同时实验验证，采用本文所提方法可有效解决内存爆炸问题，并实现显著的训练加速。