Principles for Innovative Design and Educational Theory in Generative Artificial Intelligence Tool Creation

About This Grant

This NSF IUSE:EDU Level 1 project aims to serve the national interest by developing design principles by which educational tools powered by generative AI can support the goal of personalized learning in STEM education. Generative AI models, such as large language models, have presented educators with a tangible step toward addressing the National Academies of Engineering's grand challenge of personalized learning. In particular, chatbot-style AI tools (e.g., ChatGPT) have gained significant attention as mechanisms to enhance learning environments within and outside the classroom. Much effort has been placed into developing specific chatbots for disciplinary contexts, which lean heavily on the purported problem-solving abilities of modern large language models. This project plans to question design processes when creating these systems and aims to develop and test design principles for developing AI-empowered educational tools, focusing on providing personalized and humanized interactions and feedback grounded in theories of social presence and self-regulated learning. Computational thinking is recognized as a core 21st-century skill and a vital part of STEM education. As a context for testing the design principles, the project team intends to focus on courses aimed at developing computational thinking. The project team plans to build a generative AI tool called the "coachbot" utilizing the proposed education-informed design principles and will be tested in two complementary computing-focused courses at the University of Michigan and University of Cincinnati. The project aims to synthesize current literature to document how current educational AI tools are currently developed to extract preliminary design principles and gather insights related to their efficacy using a combination of learning analytics, student interviews, and pre-post surveys. The goal of this project is to develop design principles for AI-driven synchronous tutoring systems, focusing on computing education, by emphasizing broader, theory-based frameworks rather than niche, one-off chatbot solutions. Grounded in self-regulation theory and social presence theory, the initiative seeks to enhance students' metacognition and motivation in STEM while fostering acceptance of human-like digital tools. The resulting principles and AI model are intended to guide the development of future intelligent tutoring systems that prioritize educational outcomes beyond technological innovation. The project focuses on three research questions: (1) What design principles have been used to create educational AI tools? (2) How do students' experience interacting with the generative AI-powered tool called a “Coachbot” align (or not) with the elements of social presence? (3) How does the Coachbot support students' learning to solve computational problems? The project intends to address challenges in creating effective personalized educational experiences. To help to answer these questions, the project plans to: (1) conduct a scoping review to propose a set of evidence-based design principles for AI educational tool creation, (2) refine and implement the Coachbot as a specialized, personalized, principle-informed intelligent tutoring system for computationally focused courses, and conduct a case study centered on students’ use of the Coachbot of the proposed design principles from the perspective of students’ user experiences and learning outcomes; (3) revise the design principles based on the findings synthesized in Phase 2 and disseminate feedback from the greater design, AI, and education communities. The NSF IUSE: EDU Program supports research and development projects to improve the effectiveness of STEM education for all students. Through the Engaged Student Learning track, the program supports the creation, exploration, and implementation of promising practices and tools. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.