Collaborative Research: Multi-Modal Sensing with Robust, Unified, and Scalable Diffractive Optical Neural Networks

About This Grant

Modern artificial intelligence (AI) and data-intensive applications, from language models to scientific simulations, demand ever-growing computational capability. Yet, conventional electronic-based computing systems face a critical bottleneck: the energy and time required to move data between processors and memory now exceed the costs of performing the calculations themselves. This imbalance has slowed progress in AI and deep learning, while contributing to rising energy demands. This project explores a transformative computing paradigm grounded in photonic-based, in contrast to the conventional electronic-based paradigm, enabling information processing through light. Using diffractive optical neural networks (DONNs), our team will develop specialized, ultrathin computing systems composed of engineered nanostructures that manipulate light waves. These DONN-based systems are capable of performing inference tasks at the speed of light while consuming significantly less energy than conventional electronic platforms. These breakthroughs will support future embedded and edge devices, from smart sensors to autonomous systems, enabling AI to scale sustainably. The project will expand the American workforce in AI and optical computing by integrating education and outreach. The PIs will connect students from local community colleges to research opportunities and build upon successful undergraduate and K-12 outreach programs at both NC State and the University of Minnesota. The team will introduce new AI- and photonics-focused courses into the undergraduate curriculum and engage students through REU programs, design projects, and public-facing events such as youth AI labs, science competitions, and museum activities. Together, these efforts will prepare students with expertise in optics, nanofabrication, and machine learning, ensuring they are equipped to lead future innovations in photonic computing. This project seeks to design, fabricate, and experimentally validate a new generation of metasurface-based diffractive optical neural networks (DONNs) that overcome key limitations of existing optical accelerators. The research targets three major advances: (1) Multi-modality: developing a unified DONN framework capable of processing diverse data types, including images, text, and graphs, and integrating programmable metasurface layers to significantly broaden its application scope. (2) Scalability: enabling deep, large-scale network inference using iterative train-prune-retrain, weight clustering, and tile-wise sparsification to minimize diffraction errors and optical crosstalk, combined with nonlinear activation reduction techniques such as low-degree polynomial network approximations for efficient, stable inference; and (3) Robustness to physical non-idealities: integrating fabrication-aware optimization and phase smoothing to mitigate meta-atom geometry variations, inter-element crosstalk, and environmental instabilities. The DONNs will employ multi-channel metasurface layers that leverage wavelength and polarization multiplexing for parallel, multi-task processing, and will incorporate architectural strategies such as shared diffractive layers, spatially distinct task routing, and optical skip connections to extend functionality to transformer and graph-style architectures. Fabrication will follow a staged cleanroom-to-foundry pipeline, with TiO2 nanofin metasurfaces prototyped at NC State and scaled through commercial foundry tape-outs to achieve device areas exceeding 250 μm2. The resulting systems will be experimentally benchmarked for accuracy, energy efficiency, and throughput across different workloads. By tightly integrating algorithmic innovation, photonic device engineering, and experimental validation, this work will establish the foundations for compact, energy-efficient, and adaptive optical AI processors, offering a pathway toward practical deployment in embedded and edge-computing applications. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.