HCC: Small: NSF-DST: GameSignAI: Using AI to Help Deaf Children Acquire Language Skills

About This Grant

Ninety-five percent of deaf children in the US are born to hearing parents, most of whom never learn sign language well enough to teach it to their children. This puts these children at risk for delayed communication development compared to hearing children or deaf children of deaf parents. These communication delays may lead to social isolation, mental health issues, and lowered quality of life. Using educational games to help bridge these gaps has been proposed, but little educational software is designed for American Sign Language (ASL) and even less for sign languages of other countries. This project's goal is to create a framework for mobile device developers to use on-device sign language recognition to develop educational software for Deaf children and their hearing parents. The plan is to develop educational software, tools, datasets and machine learning models and release them via open-source so the research can be replicated for other sign languages world-wide. To establish that the tools can generalize across languages, the project team will develop versions of the software for both ASL and for Indian Sign Language (ISL). ISL is both less studied than ASL and has a large user population, making it an excellent second language to work in. To ensure the work is well-suited to the needs of deaf people, the work will be done in close collaboration with the Deaf community, and with participation of Deaf student researchers. The main technical focus of the project is to develop a framework that allows the rapid development of educational software that (a) can target multiple sign languages and (b) can be practically deployed without specialized hardware or setup requirements. The concrete educational software to be developed is based on two educational games for sign language previously developed by the research team that use word- and phrase-level ASL recognition in their gameplay. One key part of the project is the creation of robust datasets of ASL and ISL use, including fingerspelling, word, and phrase-level expressions. These datasets will both drive the creation of new recognition models and support linguistic analysis of ASL use. A second key element is designing novel algorithms for both recognizing sign language that are accurate enough to be useful and cope with disfluencies, flexible enough to support both adult and children's signing, and efficient enough to run on tablets and smartphones. The third key element is developing game design principles for educational software for sign language: avoiding the need for English-based instruction, appropriately responding to both well-formed and disfluent signing, and ways to assess and improve both users' receptivity and expressiveness around sign language. The research team will assess the effectiveness of both the software itself, in terms of its effects on children's language development, and the framework, in terms of its ability to be useful to other languages, developers, and researchers. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.