Collaborative Research: Performance Guaranteed Statistical Learning with Multiple Classes of Models (guided by PCS)

About This Grant

This project will develop a next-generation statistical framework to improve the reliability and reproducibility of data science (DS) and artificial intelligence (AI). As DS and AI play an increasingly central role in science, healthcare, technology, and national security, it is essential that the methods used to analyze data are trustworthy and transparent. However, current data analysis tools often rely on the traditional assumption that data come from a specific form of probabilistic model—a practice that often fails to capture the complexity of modern data, leading to misleading conclusions and contributing to a growing crisis of scientific replication. This project studies a new framework called Predictability-Computability-Stability Inference (PCSI) for veridical data science (VDS) to help ensure that conclusions drawn from data are not only accurate but also stable, interpretable, and computationally practical. The research will also help train the next generation of data scientists, promote interdisciplinary collaboration, and support the responsible development of AI. By improving how uncertainty is measured and communicated, the project serves the national interest by strengthening scientific research integrity and public trust in data-driven decisions. The PCSI approach evaluates multiple predictive algorithms and filters out those with insufficient performance, avoiding dependence on any single model and focusing uncertainty assessment on those that are adequately predictive. It uses multiple bootstrap samplings to address uncertainty in an integrated manner with the new form of uncertainty in PCSI: stability over pred-checked algorithms. It also employs a novel multiplicative calibration technique to ensure valid prediction coverage, improving robustness to subgroup structures. The project specifically aims to advance the PCS framework for veridical data science (VDS) by developing PCSI methods for key areas of machine learning, including classification, deep learning, and ensemble learning. The research consists of three thrusts: (1) developing PCSI for classification to improve uncertainty quantification, robustness, and accuracy in both binary and multi-class settings; (2) designing PCSI methods for deep learning and large language models using computationally feasible perturbations and calibrations to enhance stability, interpretability, and performance in modern AI; and (3) establishing theoretical foundations for PCSI and PCS-guided ensemble learning, showing that even under model mis-specification, PCSI can remain valid and outperform existing methods such as conformal inference under reasonable conditions. These developments will result in statistically sound, computationally efficient tools, along with software, publications, and educational materials to broaden participation and ensure broad dissemination. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.