CAREER: Enhanced User Interface Engineering via Automated Semantic Screen Understanding

About This Grant

The ease of use afforded by software user interfaces (UIs) has expanded access to computing and driven the digitization of modern society. More aspects of daily life rely on software than at any point in human history, which necessitates the construction of well-designed UIs that allow end-users to easily carry out computing tasks. However, many modern UIs contain defects that make using the underlying software difficult or confusing, leading to wasted time and money for our computing-dependent society. One reason that bugs persist in UI-centric programs is that the process of building, testing, and maintaining high-quality software UIs, a process we refer to as UI Engineering, has long been documented to be uniquely challenging for software engineers. The difficulties in this process stem from the need to reason between the higher-level abstraction of a program UI, and the lower-level implementation details of program features. To address this fundamental challenge, this project will develop UI-specific machine learning techniques that will bridge the abstraction gap between UI and code and serve as the backbone of a set of developer-focused approaches that will improve activities related to UI engineering. The key premise that guides this project, which is supported by prior work, is that the tasks afforded by UIs are encoded into their visual pixel-based representations, and that these representations can be effectively learned by multimodal machine learning techniques. The results of this project will allow software engineers to carry out UI Engineering tasks related to documentation, testing, and debugging more effectively, leading to construction of higher-quality UIs with fewer defects. Organizations and society will benefit from software systems with higher-quality UIs that better support the seamless completion of a range of computing tasks, leading to lower costs in software development, testing, and maintenance. This project also aims to educate the next generation of software engineers by providing them with the knowledge and tools needed to construct high-quality UIs. The project will create reusable educational course content, integrate findings and tools resulting from the research into computer science curricula, and organize community building and outreach events. This project is centered around four goals. First, it will design and develop a new form of multimodal screen embedding that can capture the relationships of UI components, aiding in effective automated learning of UI semantics. Second, it will design and develop a set of novel automated multi-modal software documentation techniques that learn from both existing code and UI data to summarize code as it relates to app features. Third, it will design and develop approaches for automated repair of UI-related bugs by leveraging the relationship between faults that manifest through the UI and the patterns of corresponding patches. Fourth, it will design and develop an approach for agentic exploration of UI-centric apps using reinforcement learning. The proposed research will generate foundational knowledge related to designing natural language processing, computer vision, and program analysis techniques for automated understanding of UI screens, explore the potential of leveraging patterns in UIs to automate software engineering tasks, and develop new forms of UI-centric developer tools. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.