Combining category learning with machine vision models to test aesthetic valuation

About This Grant

People constantly make value judgments about visual objects and images, such as whether they like the design of a laptop, find a public space to be beautiful or are moved by AI-generated artwork. These perceptual valuations are surprisingly unique from person to person. Despite how important these judgments are for how people spend their time, how they feel, how they learn, and what they buy, very little is understood about how people make them. Using advanced machine learning and artificial intelligence (AI) techniques, this project tests the idea that perceptual valuations reflect how visual objects relate to the internal knowledge each person has about the visual world as a result of their personally lived experience. Is it familiar? Is it unique? This depends on what a person has been exposed to. Testing theories of perceptual valuation is important for understanding how and why people are curious to learn about the world around them, and for the development of the next generation of artificial intelligence systems. Perceptual valuation also has important translational science consequences for the design of public spaces, workplaces and hospitals, and for the use of the arts in therapies for mental and physical health. This project aims 1) to test whether “deep neural network” (DNN) machine learning models can serve as proxies for peoples’ internal models about objects in the visual world, 2) to use generative AI to create sets of novel images that can be used to probe these internal models in specific ways, and 3) to test the hypothesis that perceptual value is highest in a “zone of learning” that neighbors existing knowledge, for objects are similar to what is already known but also offer the opportunity to learn something new. Human observers participate in experiments during which they learn to categorize a training set of images (visual artworks) into two categories. In parallel, a set of DNNs are “tuned” using the same task and exposure, resulting in models that mimic the observers’ learning of new knowledge. Measurements from the DNNs are used as models of what observers know about the images, and are used to predict their perceptual value judgments about new images that are more or less similar to the training set. By using AI and machine learning to measure the similarity between visual images and to model individuals’ highly personal mental states and how they are changed by learning, this project develops tools for understanding state- and person-dependent cognition and contributes to the design of advanced self-supervised learning algorithms for AI. The project also includes training activities for students in how to use machine learning and artificial intelligence in the behavioral sciences, helping to create an AI workforce and spread AI literacy. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.