Visual Play: An Interactive Play Idea and AAC Grid Generator to Assist in Behavioral Interventions for Minimally Verbal Autistic Children

About This Grant

Abstract and Project Summary 1 in 36 children are autistic. Many have severe speech and language delays — at age 9, 20% of children with ASD use fewer than 5 words per day, while another 19% can say words but are unable to combine them into sentences. These outcomes can be improved through play-based behavioral therapies. In a 2014 study involving minimally verbal and non-verbal autistic children, Kasari et al added to such therapy a speech generating device (SGD), a form of augmentative and alternative communication (AAC) consisting of a tapping talkable grid of words relevant to the play activity. SGD use resulted in a significant increase in the number of socially communicative utterances, nearly all of which involved natural speech. Though their benefits are clear, SGDs are rarely used by speech and behavioral therapists due to the cognitive burden of programming SGDs and using them during their sessions. In this proposal, SKIES — an educational technology company — and Kasari’s autism intervention research group at UCLA are collaborating to build a new app called Visual Play (VP) that makes SGDs easier to use during therapy. During the session, the therapist takes pictures of the environment, which the app uses to suggest play ideas at the child’s level of play development. Selecting a play idea causes the app to generate a relevant SGD grid at the child’s level of language development. Auto-generating SGD grids from photographs is an active area of research driven by recent advances in computer vision and large language model technology. The key innovation of Visual Play over prior work is the explicit focus on supporting evidence-based therapy by including play ideas in an SGD, which is used to build grids with relevant nouns, verbs, adjectives, and prepositions. In this Phase 1 study, Aim 1 is optimizing an AI (GPT-4o LLM) so that it can generate play ideas on par with those generated by human experts, over a wide range of play levels and toy selections. Optimization will proceed via iterative prompt engineering and fine-tuning, with both human and AI-generated play ideas scored by experts in Kasari’s group on qualities of accuracy, engagement, and repeatability. Aim 2 is making a prototype of VP. Aim 3 is testing the feasibility and usability of the app to drive iterative app development. Testing will be performed with a cohort of 10 therapists receiving training in Joint Attention, Symbolic Play, Engagement, and Regulation (JASPER) therapy. Therapists will be selected who are actively working with preschool-aged minimally verbal autistic children. The following data will be collected: app usage per session, % interactions with Visual Play, and quantitative and qualitative usability feedback. The data will be collected during JASPER therapy sessions with children — both within an intensive week-long JASPER/VP training, and during a six week follow-up period where therapists use VP at least weekly with their actual clients. Phase I aims to bring the app’s usability to the point it is rated highly on an end-of-study questionnaire (≥5/7 average rating on a modified mHealth App Usability Questionnaire) by at least 80% of therapists; and used an average of >2 times per session by at least 50% of therapists.