Collaborative Research: Acceleration and Preconditioning Methods for Deep Learning

About This Grant

The remarkable progress of Artificial Intelligence (AI) in recent years is starting to greatly influence research across a wide range of disciplines. As Numerical Linear Algebra plays a crucial role in Deep Learning models, this trend presents unprecedented opportunities for experts in numerical analysis and linear algebra to contribute to ongoing AI research. This proposal represents a step toward capitalizing on this opportunity. The focus of the proposed work is not on applying AI to solve a specific problem, but rather on enhancing AI methods themselves by exploiting insights from numerical methods to optimize the Deep Learning process. This process is time-consuming, energy-intensive, resource-demanding, and overall very costly. Therefore, any improvements that can speed up the process are likely to have a significant impact. The investigators will leverage their experience in numerical methods to develop a number of techniques for accelerating the training of large AI models. The project aims to develop techniques that exploit both accelerators and preconditioners to speed up iterative procedures used in training deep learning models. The same combination of preconditioning and acceleration techniques is central to the effectiveness of iterative solution methods for linear systems. Acceleration methods such as Anderson/Pulay mixing or the Reduced Rank Extrapolation method, among others, have had immense success across various fields of science and engineering. However, in the context of deep learning, these methods face challenges, particularly since they were not developed for stochastic sequences common in deep learning. The team will investigate the relationship between mini-batching, a technique used for sampling subfunctions in stochastic methods, and its impact on both the convergence speed and the accuracy of the resulting models. Simple diagonal preconditioning methods have already been incorporated into optimization techniques in deep learning. The research team will explore more advanced preconditioning methods based on various approximations to the Fisher information matrix, a matrix that measures the amount of information that the observed data provides about the parameters. It has been shown that replacing the Hessian in second order methods by the Fisher matrix yields a more meaningful form of scaling of the variables, leading to better convergence and generalization. The investigating team will consider various methods for obtaining inexpensive approximations of the Fisher matrix and for combining them with accelerators. The proposed work is expected to benefit research in "scientific machine learning" by promoting participation of numerical linear algebra specialists in AI research. The PIs plan on several specific activities to promote the dissemination of knowledge in machine learning, such as writing a book on the topic of numerical methods in machine learning or offering tutorials and short courses. These activities will stimulate the interest of students in a field of increasing importance and that it will help with the immersion of those students from other areas, e.g., mathematics, into data-related disciplines. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.