SCH: Deep learning models of child development derived from social video game data

About This Grant

Autism Spectrum Disorder (ASD) affects about 1 in 31 children in the United States, yet many children are diagnosed too late to benefit from the most effective early interventions. Existing diagnostic approaches are often slow, resource-intensive, and reliant on limited specialist availability, creating barriers to timely care. Meanwhile, families increasingly use smartphones to capture everyday moments, presenting a unique opportunity to rethink autism detection. Our team’s innovative GuessWhat mobile game, designed to encourage natural play and interaction, has been used by over 500 families and produced a large, growing collection of over 5,000 short videos of young children, including nearly 3,000 videos from children with autism. These rich, real-world videos contain subtle behavioral cues that can be challenging for parents and clinicians to spot but can be harnessed by advanced artificial intelligence (AI) techniques. Our goal is to develop AI tools that automatically analyze these videos to provide accurate, early, and accessible autism risk assessments, ultimately empowering families and clinicians to act sooner and improve outcomes. From a technical perspective, this project will leverage the GuessWhat (GW) dataset to build and validate next-generation AI models for early autism detection in diverse children under 6 years old, eventually expanding to other learning conditions. In Aim 1, we will train specialized deep learning models, each focused on predicting a clinically relevant behavioral feature (e.g., eye contact, emotion), and then fuse these outputs using advanced machine learning approaches such as XGBoost and TabNet to form a comprehensive, interpretable diagnostic system expected to achieve at least 90% balanced accuracy. In Aim 2, we will develop multimodal self-supervised learning (SSL) models to learn directly from our large GW video library without relying on manual feature annotation. These SSL models, based on state-of-the-art VideoMAE and CAV-MAE architectures, will identify novel behavioral signals and enable robust autism predictions. Finally, in Aim 3 will build and implement test-time adaptation methods that incorporate important temporal patterns to ensure the models maintain high accuracy across diverse symptom presentations, recording conditions and device types, allowing for on-device, real-time performance and personalization. Together, these three aims will yield clinically robust, explainable, and scalable AI agents that can transform autism diagnosis, reduce wait times, and improve equitable access to early intervention worldwide. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.