ERI:CCSS:A Neural Network Assisted Sensing Approach For User Behavior based Biometric Identification In XR Applications

About This Grant

Extended Reality (XR) technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), offer users deeply immersive and interactive experiences, revolutionizing fields from entertainment and gaming to education and professional applications. As these technologies become integral to daily life, they increasingly collect vast amounts of sensitive personal data, such as biometrics, location, and confidential information. Thus, ensuring secure and continuous user identification becomes essential. Traditional security measures, such as passwords or two-factor authentication, are often insufficient or impractical for continuous security monitoring. Additionally, many advanced identification approaches, such as user task and iris scan, require intrusive procedures, external sensors, or high deployment costs, limiting their usability in practical AR/VR scenarios. To address these issues, this project introduces a novel approach to biometric user identification that leverages natural head movement patterns, which is a type of biometric signature inherently available in most commercial AR/VR devices, to authenticate users continuously and unobtrusively during regular device use. By embedding this behavior-based identification mechanism directly into AR/VR platforms, the research offers a new path to improving security, providing secure, continuous identification without additional hardware costs or interruptions to user experience. This initiative will enhance the security of XR applications broadly, protecting user privacy, safeguarding sensitive data, and promoting trust and wider adoption of XR technologies. The broader impacts also include future research on personalization and health care monitoring in immersive systems and expanding access to cybersecurity research and education through outreach activities involving K-12 students. The primary goal of this project is to implement an innovative biometric identification methodology based on user head movement data captured by sensors already embedded in XR head-mounted devices. By employing advanced neural network architectures, the solution could extract movement patterns for user identification. The project starts with a Bidirectional Recurrent Neural Network (Bi-LSTM) to capture temporal dependencies in user movement sequences. To better model the relative importance of each motion feature (e.g., rotational vs. translational motion), the system integrates Graph Neural Networks (GNNs) for feature-level aggregation and weighting. Further, attention mechanisms are applied between LSTM layers and output layers to enhance the model's ability to focus on critical time steps in the data. To address challenges related to user variability and limited labeled data, the project incorporates a Siamese Neural Network structure, enabling one-shot learning through feature similarity rather than multi-class classification, thereby improving generalizability and reducing model size. Additionally, the project incorporates federated learning frameworks to protect user privacy during the model training process, reinforcement learning strategies to ensure long-term adaptability and accuracy across diverse XR applications. In the end, the proposed project also explores model compression techniques to reduce computational overhead to make it possible to be implemented on real XR devices, which are mostly embedded platforms. The anticipated outcomes include a highly secure, user-friendly identification system that addresses critical XR security challenges and advances the state-of-the-art in biometric authentication technology. This project will involve more undergraduate students for research, such as XR user study. The collected data will be used to carry out a comprehensive evaluation. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.