Kronecker-factored hessian approximation for continual learning

Neural networks have been successful on many individual tasks. However, they work poorly on a sequential stream of tasks when they do not have access to the previous data. This problem is called Catastrophic Forgetting and recently has been studied in the field of Continual Learning. In this thesis, we propose four Continual Learning methods which approximate the loss function on previous tasks by a second-order Taylor approximation and use them as regularizers to maintain the performance on previous tasks without any access to previous data. To do that, we use a Kronecker-factored Hessian approximation to make the training process memory and computationally efficient. We evaluate our methods on two Domain Incremental datasets called Permuted MNIST and Rotated MNIST and investigate the performance and efficiency trade-off on them.