Research: Developing and Piloting a Prompt Engineering Competency Framework for Software Engineering Education

About This Grant

As an award under the Research in the Formation of Engineers program, this project will investigate how to form software engineers for the era of Generative Artificial Intelligence (GenAI). We will study how generative AI tools (e.g., ChatGPT) are used by software engineers to understand what knowledge, skills, and dispositions are necessary to effectively use GenAI to develop software. GenAI is revolutionizing how software engineers do their work, but so far, there have been limited efforts to adapt software engineering education to account for these changes. The first step in this process is to understand how professional software engineers are using GenAI tools and identify the knowledge, skills, and dispositions they demonstrate, which we refer to collectively as prompt engineering competency. We will use two approaches to characterize prompt engineering competency in the context of software development. First, we will analyze a sample of prompts that are available through a public database of prompts from professional software engineers across industries and job roles. Second, we will conduct interviews and surveys will leaders in software engineering companies as well as practicing software engineers. By analyzing these data, we can understand what an effective prompt looks like and identify the skills necessary to craft such prompts. Our study will contribute to the formation of software engineers by developing a prompt engineering competency framework that can inform future efforts to develop software engineering courses and degree programs that prepare future engineers to use GenAI effectively in their software development processes. GenAI tools are being used by practicing software engineers in every industry and it is essential that software engineering education adapt to teach prompt engineering competency to improve the economic competitiveness of the United States. In this interdisciplinary project, we will characterize prompt engineering (PE) competency in the context of software engineering. We will address the following overarching research question: What knowledge, skills, and dispositions characterize prompt engineering competency in software engineering? We will build on our preliminary work, which found that these elements can be integrated into a PE competency framework using two existing frameworks: Socratic Questioning and the Goals-Operators-Methods-Selection Rules (GOMS). To build this novel PE competency framework, we will carry out three activities. First, we will use a hybrid coding approach to analyze an existing database of Developer-ChatGPT conversations to identify knowledge and skills needed for PE. Second, we will conduct a Delphi study with expert software engineers and educators to refine the PE competency framework. Our study will include multiple phases and collect perspectives via both interviews and surveys. Our project will contribute a PE competency framework grounded in theory and the research of software engineering practice. This framework can inform future research on PE practices and PE competency development in software engineering workplaces as well as in engineering education. This project is designed to deliver four societally relevant outcomes. Our PE framework, learning modules, and related rubric can be used to inform both (1) undergraduate STEM education, and (2) workforce development in software engineering. We also anticipate that our framework could be adapted or extended for use in other STEM disciplines for additional educational impact. The effective use of GenAI technologies has been shown to dramatically increase software engineering productivity, so our results will also (3) promote the economic competitiveness of the United States. Finally, (4) we expect increased academia-industry partnerships through the Delphi study and the new connections made via our advisory board. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.