Transferable polychromatic optical encoder for neural networks

Visual information plays a crucial role in human response, particularly in situations where reaction time is limited to a few tens to hundreds of milliseconds [1, 2]. Though the human brain has efficiency far exceeding that of any other human-made computing systems, it still cannot process the entire collected visual data due to its massive amount of information. Most likely, our brain performs early visual processing to extract essential features for efficient and rapid interpretation without handling the entire visual data [3, 4, 5].

With the dramatic development of artificial intelligence (AI), computers can process the visual information like human brain, thanks to artificial neural network (ANN), enabling computer/machine vision [6, 7, 8, 9, 10]. Despite impressive progress, real-time inference with limited computational resources remains very challenging even with more efficient algorithms. For example, in a flying object (i.e., habitat drones [11]) on-site data processing is plagued by severe heating, battery capacity and weight handling challenges. Utilizing cloud based systems poses challenges associated with data security and additional data transfer latency [12, 13].

Optical neural networks have emerged as a potential platform to circumvent these trade-offs, since an optical system can process multidimensional information with large spatio-temporal bandwidth [14]. Recently, integrated photonics and free-space or fiber optics have been employed to implement some parts of an ANN for image compression/encryption [15, 16] and classification [17, 18, 19, 20, 21]. However, most of them are highly restricted on solving a relatively simple gray-scale datasets (i.e., MNIST and fashion-MNIST) and only a couple of systems have shown their implementation for more complicated multichannel datasets (i.e., CIFAR-10 and ImageNet) [17, 19]. For these complex datasets, the optical system often become extremely large (with multiple stacks of the photonic circuits) [19], otherwise the classification accuracy remains low ($\sim 60\%$ accuracy for CIFAR-10 classification tasks) [17, 22, 23]. In addition, the most successful ANN architectures utilize nonlinear activation functions that are challenging to implement optically. Proposed solutions, including atomic vapor cells [24, 25] and image intensifiers [26], introduce significant experimental complexity, and additional power consumption.

To leverage the strengths of both optical and digital computing systems, an encoder-decoder inspired hybrid optical/digital architecture is a promising approach [8, 27, 10, 28]. Specifically, an analog linear optical frontend (denoted as the optical encoder) performs bulk of linear computational tasks, while the digital backend implements the nonlinear operations. One intriguing possibility is to employ a static optical frontend, which is data agnostic, whereas the backend is trained and reconfigured. This resolves usual issues of modulation speed, errors, and system size in all-optical systems. An optical encoder is particularly suitable for convolutional neural network (CNN) architectures, where convolutional layers act as feature extractors, encoding high-dimensional images into low-dimensional features [29]. In fact, every free-space optic inherently performs a two-dimensional convolution operation during the imaging under incoherent light. The captured image is a convolution of the scene and the optic’s incoherent point-spread-function (PSF) [30]. Thus, by engineering the PSF, an optical encoder can perform the desired convolution and replace the initial layers of a CNN.

Recently, PSF-engineered optical encoder has been employed to classify MNIST hand-written dataset and a reasonable classification accuracy with much less computational costs compared to the AlexNet is demonstrated [31]. We note that, however, MNIST images are monochrome, and is almost linearly separable ($0.84\%$ loss without any nonlinearity [32]). The monochrome nature of the images makes the PSF-engineering approach wavelength agnostic. On the other hand, datasets such as CIFAR-10 [33] or ImageNet subset (High-10) [34, 29] are not separable by linear layers. Moreover, they consist of colored images, where the actual color information is exploited in classification.

Here, we demonstrate a polychromatic optical encoder with PSF-engineered meta-optics to classify the CIFAR-10 dataset. We first compressed the architecture into a single convolutional layer and two fully-connected layers using Knowledge Distillation. Then, we physically realized the convolution layer using an array of metasurfaces, where each metasurface, thanks to the inherent chromaticity, performs a separate convolution for each color channel. As a result, the hybrid CNN with an optical encoder reduces the total number of multiply–accumulate (MAC) operations at the digital backend by an factor of $\sim 24,000$. The reduction of number of MAC operation directly corresponds to the computational costs, i.e., power and latency [35]. It is worth noting that we always require an imaging system (i.e., lens and camera) to capture the image under ambient illumination, before we deliver the image data to the computational backend. Hence, with a single meta-optical encoder, we are not adding any additional optics, but simply replacing a conventional lens with PSF-engineered meta-optics. This makes our optical system compact and fully compatible with conventional optical imaging systems, while the other systems such as, integrated-photonic systems require pre-processing of the data[19] and in-sensor computing needs a customized sensor design [36].

Furthermore, we adopt the same meta-optics (optical convolutional layer) which was optimized for CIFAR-10 dataset to High-10 dataset to explore the generality of optical encoders. In practice, a static optical encoder should be applicable for any scene. While, one approach is to employ reconfigurable frontend, e.g. based on non-volatile phase change materials [37] or liquid crystals [38], the performance of these reconfigurable front end in terms of individual pixel control, power consumption, and operating speed are still inferior for practical deployment. Remarkably, with the same passive optical encoder (optimized for CIFAR-10 dataset), we achieved a high classification accuracy (for High-10 dataset) by fine-tuning the digital backend with additional fully-connected layer (via transfer learning approach). This ability to generalize the frontend is crucial for any ANNs as it enhances their versatility, efficiency, and robustness. A network that generalizes well can be applied to different tasks without extensive re-training, saving time, reducing costs for meta-surface fabrications, and conserving computational resources for real-world applications.

Our optical encoder concept is described in Figure 1. The original CNN, i.e., AlexNet, has five convolutional layers and three max pooling layers at the front, followed by three fully-connected layers at the end, while nonlinear activation functions “ReLU” are placed in each layer. Replacing all individual five convolutional layers with five sequential optics is extremely difficult because of misalignment, large system size, lack of nonlinearity, and low signal-to-noise ratio, issues that compound with increasing number of optical elements. Therefore, we compressed AlexNet to a single convolutional layer and two fully-connected layers using knowledge distillation method [39], which reduces the complexity of the architecture with a minimal compromise in accuracy.

While, the compression of an original CNN is essential for realizing the optical encoder scheme, there are practical trade-offs to consider. On one hand, the physical size of the sensor and meta-optics limit the number of kernels and kernel size. On the other hand, small kernel size or number of kernels fail to classify the data effectively. We empirically searched for the optimal number and size of the kernels while compressing the original CNN. For CIFAR-10 classification task, we design 16 kernels of $7\times 7$ size (see details in Supplementary Materials). The training and testing accuracy from the compressed all-digital CNN are $76.24\pm 0.31\%$ and $75.90\pm 0.30\%$, respectively.

Since the CIFAR-10 images have three channel information $-$ corresponding to red (R), green (G), and blue (B) $-$ we have 16 individual $7\times 7$ kernels for each color, a total of 48 kernels. In meta-optics, it is possible to design a single meta-optics to produce three different PSFs (i.e., convolutional kernels) for RGB wavelengths (more details in Methods). It is difficult to create positive and negative weights on the camera in terms of light intensity at the same time, so we separate each kernel into its positive and negative parts and design an optic for each [40], for a total of 32 meta-optics in total corresponding to a 16 positive and negative polychromatic kernels.

Another important design parameter is to determine how many pixels on the camera represent one pixel of the PSF. We term this as the enlargement factor. For example, when the enlargement factor is $2$, the ground-truth PSF which is a $7\times 7$ matrix will correspond to $14\times 14$ pixels on the camera. While a large enlargement factor ensures less alignment error, the signal intensity on each camera pixel will be lower, resulting in low signal-to-noise ratio. To determine the optimal enlargement factor, we experimentally tested several meta-optics with different enlargement factors for a particular kernel (see details in Supplementary Materials), and obtained an optimal enlargement factor of $2$ for a meta-optic made of $3200\times 3200$ scatterers.

The metasurfaces were made of silicon nitride on a quartz substrate to ensure high transparency in the visible wavelength (Figure 2a). Figure 2b shows the transmission coefficients and phase shifts from silicon nitride pillars at RGB wavelengths as a function of the pillar width, $w$, at the fixed height of 800 nm, obtained by rigorous coupled-wave analysis (RCWA). We target wavelengths ($\lambda$) of $450$, $532$, and $635$ $nm$ for RGB colors based on the availability of the laser diodes.

In order to effectively model the wavelength-dependent effects of the meta-optics in a gradient descent-based optimization method, a fast and differentiable function is required to map between the pillar width and imparted phase. We define a proxy function inspired by the approximate phase shift of a dielectric waveguide with corrective factors that are fit to the RCWA simulation results. To calculate the phase shift ($f_{R}$, $f_{G}$, and $f_{B}$) for RGB wavelengths with respect to the pillar width, $w$, we define the proxy function as

The first term corresponds to a general phase shift from a dielectric waveguide, where $n_{eff}$ and $L$ are the effective refractive index and height of the silicon nitride pillars. The second term, only has the $w$ variance, corresponds to a correction term in the Gaussian shape with $A$, $B$, and $C$ as a fitting parameters. Lastly, $f_{0}$ corresponds to a phase shift offset, making the $f_{\lambda}(0)=0$. This proxy function does not model the resonance-induced phase variations; However, we do not want to use those phase variations due to reduced amplitude and these resonances are expected to be less prominent in the fabricated devices due to the sidewall roughness.

Figure 2c shows the design flow for the polychromatic RGB meta-optics that have optimized PSFs for individual RGB colors. For a meta-optic parameterized by an arbitrary two-dimensional pillar width map, $w(x,y)$, we extract three separate phase maps using the proxy functions $f_{\lambda}$. We then propagate the electromagnetic field using angular spectrum method [41] to simulate the PSFs at the focal plane, $2.4$ $mm$ away from the meta-optics. At the focal plane, we compare the computational ground-truth PSFs defined by the convolutional kernels obtained using knowledge distillation ($PSF_{GT,\lambda}$) and optically-simulated PSFs ($PSF_{sim,\lambda}$) at each RGB channels, where the channel-dependent losses are defined by the sum of squares of differences in each pixels:

We optimize the map of two-dimensional pillar width, i.e., meta-optics, for minimizing the net loss which defined as a root mean square of the losses at three different colors using the Adam optimizer in TensorFlow [42]:

Calculated losses for all the kernels at three different colors are shown in the Supplementary Materials.

An optical image of the fabricated chip is shown in Figure 3a. A single chip contains a total of 32 convolutional meta-optics (corresponding to 16 positive and 16 negative convolutional kernels) and additional 5 metalenses which are focusing light at the focal plane, to aid in the alignment (e.g., tilt, rotation, and distance) between the meta-optics and the camera. Figure 3b shows the schematic of the PSF measurement setup. By changing the laser diodes, we illuminate individual RGB coherent light onto the camera through the meta-optics and experimentally characterize the polychromatic PSFs. A pinhole of $25\mu m$ diameter creates an approximate point source and the positions of the optics, i.e., pinhole, meta-optics, and camera, remain the same while changing the laser diodes.

Figure 3c shows both the ground-truth PSFs and measured PSFs for a particular kernel for individual RGB wavelengths. To quantitatively analyze the difference between the ground-truth and experimentally measured PSFs, we define a cosine similarity ($\eta$) as:

where $A_{i}$ and $B_{i}$ are the ground-truth and measured intensity profiles of the PSF for RGB wavelengths, respectively. The calculated $\eta$ for RGB wavelengths are about 0.88, 0.56, and 0.81, respectively. The quntitative discrepancy can be attributed partially to the fabrication and measurement imperfections. Additionaly, not all the polychromatic PSFs are physically realizable as the phases at different wavelengths are not completely independent. Creating more physically-realizable PSFs via co-designing the optical frontend and computational backend, also termed as end-to-end design [43, 44], instead of replacing the convolutional layer with optics, may increase the $\eta$. However, including the meta–optical simulation in the end-to-end design may result in local optimum, and the fabrication/ measurement imperfections will still be present. As we will show later in the figure, the computational backend is robust against such discrepancy in the PSFs, and we can easily correct for these errors by introducing an additional fully-connected calibration layer in the digital backend.

Then, we test the polychromatic optical encoder for CIFAR-10 dataset. By replacing the pinhole with an organic light-emitting diode (OLED) display, we convolve the CIFAR-10 images with the characterized PSFs of the meta-optics (Figure 3d). The displayed image size is carefully adjusted according to the convolutional kernel size and the enlargement factor on the camera (more details in Supplementary Materials). Figure 3e shows the computationally and meta-optically convolved RGB images of one of the CIFAR-10 dataset. The meta-optically convolved image loses some of the high resolution components, likely due to the imperfect fabrication and alignment errors which are already recognisable from the PSF measurements as well as the spectral overlap between RGB color pixels of the camera. However, as we will show later in the figure, the computational backend is robust against such discrepancy as we do average pooling the convolved image into $6\times 6$ size. We added an additional fully-connected layer, called calibration layer, to address the weights of each kernels and colors, dealing with the discrepancy between optical/digital systems (e.g., normalization, scaling, translation, rotation, tilt, noise). This calibration layer allows us to use the pre-trained digital backend and incur minimal computational cost. Detailed explanations on the calibration layer is in the Methods and Supplementary Materials.

Figure 3f shows the confusion matrices of the classification accuracy of the CIFAR-10 data with an original CNN (AlexNet), a compressed CNN using knowledge distillation, and a hybrid optical/digital CNN using convolutional meta-optics after the compression. Even though there are slight differences between the optical and digital convolution results (Figure 3e), after introducing the calibration layer, we can achieve similar accuracy (less than 5% loss) for both training and testing dataset (Table 1). It is possible to improve the accuracy if we retrain the backend and the calibration layer; However, retraining the backend has no practical usages. Additionally, this hybrid approach significantly reduces computational costs which can be represented by the number of multiply-accumulate (MAC) operations. From the original CNN to compressed CNN, we can reduce the computational load, which is represented by a number of MAC operations, by a factor of $\sim 1,400$, while we can reduce further by a factor of $\sim 17$ after replacing a convolutional layer with meta-optics. Detailed calculation about number of MAC operations for each CNN architectures are described in Extended Data Table A1. The detailed information for network design and dimension selection is available in the Supplementary Materials.

To analyze the effectiveness of the meta-optical convolutional layer, we utilize principal component analysis (Figure 4). For the original CNN and compressed all-digital CNN, each class is well-separated (Figures 4 a and b), implying that we can extract out the key features of the CIFAR-10 image dataset after convolution. On the other hand, after the optical convolution using meta-optics, without calibration, different classes of the image were very difficult to distinguish (Figure 4c). Additionally, some clusters exhibit larger sizes and overlapping regions than Figure 4(a-b), e.g., the navy blue, brown, and red clusters (confidence ellipses). However, after introducing the calibration layer, the clustering regions become smaller, and the separations between classes increases. As shown in Figure 4d, each class becomes well-separated and distinguishable, similar to the compressed CNN. This critical role of the calibration layer is consistent with the classification accuracy without and with the calibration layer (Table 1). We note that the calibration layer can potentially be compressed into the pretrained digital backend via additional training and will not affect the number of MAC operation for inference (Extended Data Table A1).

Our convolutional meta-optics implements convolutional kernels which came from compressed CNN for CIFAR-10 data. Unlike computational neural networks, optical implementations are extremely difficult to modify once they are fabricated. This necessitates different convolutional meta-optics for different dataset. However, we found that the convolutional layer that we optimized for CIFAR-10 can be readily adapted to classify another dataset High-10, with a transfer learning process. We added an additional fully-connected layer, which we call a “transfer learning layer”, that is located in between the former fully-connected layers and convolutional layer. By training the transfer learning layer, we can fit the other dataset, i.e., High-10, to the CNN which is pre-optimized for a particular dataset, i.e., CIFAR-10, with only fine-tuning a small part of the original network without changing the former network structure (see details in Methods). The High-10 image dataset is polychromatic (RGB) and has a size of $224\times 224$ size. To use the former CNN optimized for CIFAR-10 data for High-10 data, we resize the High-10 images to $32\times 32$ size, same as the CIFAR-10 data.

Without applying a transfer learning method, the training and testing accuracy is rather low around 40%. However, after transfer learning, we achieve much higher training and testing accuracy ($\sim 67.43\%$ and $\sim 66.01\%$, respectively) on the High-10 data with the convolutional layer and two fully-connected layers. We further experimentally verified this approach works in our hybrid optical/digital CNN using the same convolutional meta-optics that we used for the CIFAR-10 data digital backend structure with one additional fully-connected layer. The average training and testing experiment accuracy of the High-10 data are similar (less than $5\%$ loss) to the compressed all-digital CNN, which is about the same of the CIFAR-10 case. The principal component analysis results of both compressed CNN and hybrid CNN for the High-10 data is shown in Supplementary Materials, where we can see how different classes are separable in their feature map. The detailed information for calibration layer design, number of calibration selection and PCA visualization is available in the supplementary materials.

The results in this work are strong evidence that optical frontend can significantly reduce the power consumption and latency of ANNs for computer vision tasks. Despite realistic fabrication and measurement errors arising from optical implementation, the approach achieves the state-of-the art classification accuracy in multichannel CIFAR-10 data with the addition of a calibration layer and trainable fully-connected layers. The use of a single meta-optical layer to perform complex, multi-channel convolutions highlights a unique applicability of meta-optics that cannot be accomplished using traditional optics. In addition, we address the lack of reconfigurability in existing optical implementations using transfer learning approach, and reconcile the optical frontend optimized for CIFAR-10 to the High-10 dataset. In this regard, we suggest that a hybrid approach comprised of an optical frontend and reconfigurable digital backend utilizes the key advantages of optics (i.e., no latency, no loss, large space-bandwidth) with robustness and reconfigurability provided by the backend.

Acknowledgements The research is supported by National Science Foundation (EFRI-BRAID-2223495). Part of this work was conducted at the Washington Nanofabrication Facility/ Molecular Analysis Facility, a National Nanotechnology Coordinated Infrastructure (NNCI) site at the University of Washington with partial support from the National Science Foundation via awards NNCI-1542101 and NNCI-2025489.

Conflict of interest A.M. is a co-founder of Tunoptix, which aims to commercialize meta-optics technology.

Code availability The code and data generated and/or analyzed are available from the corresponding author upon reasonable request.

Author contribution M.C. and J.X. contributed equally to this work. A.M. and E.S. conceived of the idea and provided funding. A.W.S. involved in developing the PSF-engineering approach. J.X. and E.S. developed the knowledge distillation approach. J.X. trained the neural network and analyzed the experiment data. M.C. developed the polychromatic PSF-engineering approach and designed the meta-optics. M.C. fabricated the meta-optics and conducted the optical experiments. S.H.B. advised on the result analysis. M.C., J.X., and A.M. wrote the manuscript with input from all authors.