Collaborative Research: Frameworks: MPI4AI: Enhancing Performance and Productivity of AI Science through Next-generation High Performance Communication Abstractions

About This Grant

This CSSI project is a multi-university collaboration between Tennessee Technological University, the University of Tennessee, Knoxville, Stony Brook University, and the Illinois Institute of Technology. This project improves how massively parallel computers run large-scale artificial intelligence (AI) applications by enhancing the Message Passing Interface (MPI), a widely used standard for coordinating work across many computers in parallel programs. Currently, the enabling data-transfer software used in AI, for communication between computers enhanced by Graphical Processing Units (GPUs), are often proprietary and/or limited in scope; they cannot be expanded or enhanced by an open community. That situation restricts innovation, making it harder for scientists to collaborate and enhance their science output on limited computer resources, while also creating dependency on a few vendors. By contrast, this project builds on and advances Open MPI, a major open-source implementation of MPI with a long history of broad impact, to make it more efficient, flexible, and better suited for modern AI tasks. In addition to improving the Open MPI implementation, MPI4AI aims at standardizing extensions to MPI so all implementations and users of MPI will benefit from this project's outcome. MPI4AI introduces key improvements to Open MPI, including native support for GPU communication, enhanced collective (group) communication operations including those that are AI-algorithm specific, compute stream integration, and optimized data movement. Specifically, these advances target performance bottlenecks in three AI patterns: neural architecture search with transfer learning, key-value prefix caching in large language model inference, and large-scale data-parallel training. The project improves resilience and malleability through fault-tolerant mechanisms, enabling AI applications to adapt dynamically to system changes and to use resources more efficiently. By forwarding these enhancements toward adoption in the upcoming MPI-5 and MPI-6 standards, the project ensures long-term impact across both academic research and industrial AI workflows. These contributions will lower the cost of running large AI workloads and broaden access to scalable AI infrastructure. MPI4AI's capabilities will enable researchers exploring new modalities of AI computation to express their algorithms and code efficiently and more effectively as compared to existing solutions that work within the confines of current MPI features and vendor-specific message-passing libraries. Underlying improvements devised for Open MPI will also be broadly beneficial to other use cases and users of this parallel programming system. Overall, key strengths of this effort are a strong commitment to standardization and emphasis on performance-portability across various hardware platforms with particular focus on AI-enablement. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.