Diagonal Hessian-Based Regularization in Multi-Head Continual Learning

This dataset contains the complete collection of empirical results and ancillary data for the article, \Diagonal Hessian-Based Regularization in Multi-Head Continual Learning.\ The paper investigates the ability of a diagonal Hessian approximation to\/from be used as a parameter importance estimation in order to reduce catastrophic forgetting in a neural network. The data presented here provides evidence to support the paper's main assertion that using a diagonal Hessian method, in conjunction with a multi-head architecture that decouples task-specific networks, is a more stable and effective method than traditional Fisher Information Matrix (FIM) based methods like Elastic Weight Consolidation (EWC). The experiments were performed on three standard continual learning benchmarks: Split Fashion-MNIST, Split CIFAR-10, and the more difficult Split CIFAR-100, using ResNet-18 networks. The results within this dataset highlight significant performance improvements including 7x less forgetting on Split CIFAR-100 with EWC. This dataset is provided in order to allow the reader to directly reproduce the findings of the paper, for the purposes of establishing a strong baseline for future regularization-based continual learning research, and to hope for wider dissemination and additional research into the properties of second-source importance metrics.