Collaborative Research: CyberTraining: Implementation: Small: Cybersecurity Training and Curriculum Design for Large Language Model Research Workforce

About This Grant

Large Language Models (LLMs) such as ChatGPT, Llama, Claude, and Gemini and their empowered applications (such as retrieval-augmented generation systems and autonomous agents) have been widely integrated into advanced cyberinfrastructure (CI) systems to enhance data management, collaboration, and scientific discovery by assisting with tasks such as large-scale text analysis, automated data classification, knowledge extraction, and domain-specific question answering. However, many research studies have shown that LLMs and their applications are vulnerable to various attacks, such as jailbreak, prompt injection, knowledge corruption, data poisoning, and privacy attacks. These attacks pose significant concerns for integrating LLMs into CI systems, as well as broad applications in security- and privacy-critical domains such as healthcare, finance, and law. Despite various research studies that have identified the cybersecurity risks associated with LLMs, there remains a huge training gap among many stakeholders. This gap stems from two factors: emphasis on utility and efficiency over security, and lack of expertise in LLM security. This training gap is particularly concerning as CI systems increasingly rely on LLMs for critical decision-making, code generation, and sensitive data analysis, which potentially exposes them to sophisticated cyber threats, especially for security-critical CI systems. This project aims to bridge this training gap. This project will develop a CyberTraining program to train undergraduate and graduate students across the nation to identify, analyze, and mitigate different attack vectors targeting LLM-empowered advanced CI systems. The program is centered on eight core training modules, which serve as its foundational framework. Based on these modules, a series of sustainable training activities are developed to prepare, nurture, and grow the workforce for supporting the development of LLM-empowered advanced CI systems. The training activities include 1) Hands-on exercises through an interactive learning platform that helps students gain practical experience in LLM security; 2) A two week onsite summer bootcamp designed to foster deeper engagement with faculty and industry mentors during the training modules while promoting professional development; and (3) Degree and curriculum development that selectively incorporates training modules into courses related to AI and Cybersecurity. The training modules and materials developed in this project will also be made publicly available for broad adoption. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.