Key frames assisted hybrid encoding for photorealistic compressive video sensing

The optical setup of the KH-CVS prototype built by us is shown in Fig. 2(a). A camera lens forms an image at the focal plane, where a DMD (ViALUX V-9001, 2560 × 1600 resolution, 7.6 um pitch size) is located. After modulated by the DMD, the image is then transferred to an image sensor (FLIR GS3-U3-123S6M-C, 4096 × 3000 resolution, 3.45 um pixel size) by a projection lens with a magnification rate of $0.45\times$ for a pixel-to-pixel mapping. Through the exposure of the image sensor, the measurements can be yielded and readout. The micromirrors on the DMD can be individually rotated $\pm 12^{\circ}$ to reflect the light to different directions, leading to an on-off state control. A total internal reflection (TIR) prism [36] is used to steerer the light from the camera lens, making it incident onto the DMD surface at $24^{\circ}$ to meet the angle condition of the DMD. Since it is difficult to directly measure the mismatch between the DMD and the image sensor, a Moire-fringe-based approach proposed by [37] is utilized to align these two elements. In the practice, to fit the spatial resolution of 256 $\times$ 256 in the deep model, we perform 6 $\times$ 6 binning, i.e., capture an region of interest (ROI) of 1536 $\times$ 1536 and then manually downsample by a factor of 6. In order to exclude the ambient light and increase the stability of the system, we package the optical system to build a prototype, of which a photograph is shown in Fig. 2(b).

Since KH-CVS needs to alternately capture short-exposure key frames without coding and long-exposure compressive frames with coding, it is required to design a timing scheme that differs from that used in standard SCI systems. As illustrated in Fig. 2(c), among the preloaded patterns on the DMD, there are $B$ coding patterns ($\mathbf{C}_{1}$ to $\mathbf{C}_{B}$) and an additional fully open pattern that all micromirrors are "on" (the DMD works as an ordinary reflective mirror with this pattern). We generate and synchronize the DMD and image sensor control signals through an arbitrary waveform generator (AWG, UNI-T UTG2000B). For the DMD, its trigger signal is a periodic rectangular wave with its period identical to the equivalent exposure time $t_{x}$, and the DMD refreshes the pattern at each rising edge [Fig. 2(d)]. For the image sensor, the trigger signal is an alternating short and long rectangular wave [Fig. 2(e)]. The short pulse width is $t_{x}$ and corresponds to a key frame. The duration of the long rectangular wave is $t_{y}$, and $B$ coding patterns will be consecutively refreshed on the DMD within this exposure, producing a compressive frame. In this manner, a sequence of alternate key frames and compressive frames is captured. A small temporal interval $t_{g}$ is set between two exposures of the image sensor for data readout and sensor cleaning.

The conceptual pipeline of KH-CVS is depicted in Fig. 2(f). With the aforementioned setup of DMD patterns and trigger signals, the scene video with high temporal dimension is encoded and captured as a sequence of key frames and compressive frames. Similar to standard SCI, $B$ frames can be recovered from each compressive, which is then fused with its neighboring two key frames, as indicated by the black dotted box. For compressive frame 1, key frame 1 and 2 are referred to as left and right key frame. Key frame 2 is also reused as the left key frame for compressive frame 2 in the gray dotted box. Thus, averaged 2 frames are capable of producing $B+1$ frames ($B$ reconstructed frames and 1 key frames itself) in KH-CVS, which leads to a compressive ratio as $2/(B+1)$ from a systematic viewpoint.

As illustrated in Fig. 3, in the algorithm, an intermediate reconstructed video is first made with a standard SCI method, and then each frame of it is fused with the two key frames by a deep fusion module (Fig. 4). The detailed methods are introduced as below.

In the mathematical model of standard video SCI, a high-speed dynamic scene video $\mathbf{X}\in\mathbb{R}^{n_{x}\times n_{y}\times B}$ containing $B$ frames $\left\{\mathbf{X}_{k}\right\}_{k=1}^{B}\in\mathbb{R}^{n_{x}\times n_{y}}$ with spatial resolution $n_{x}\times n_{y}$, is modulated by a coding cube $\mathbf{C}\in\mathbb{R}^{n_{x}\times n_{y}\times B}$, which contains $B$ encoding masks $\left\{\mathbf{C}_{k}\right\}_{k=1}^{B}\in\mathbb{R}^{n_{x}\times n_{y}}$. In general, pre-generated pseudo-random patterns are established for encoding in SCI systems. During the long exposure time $t_{y}$ of the image sensor, the coded frames are temporally integrated to produce a coded compressive frame $\mathbf{Y}\in\mathbb{R}^{n_{x}\times n_{y}}$, which is

where $\mathbf{G}\in\mathbb{R}^{n_{x}\times n_{y}}$ represents the measurement noise and $\odot$ denotes the Hadamard product, which can be expressed in detail as

where $i\in[1,2,\ldots,n_{x}]$ and $j\in[1,2,\ldots,n_{y}]$ denote spatial positions and $y$, $c$, $x$, $g$ are elements of $\mathbf{Y}$, $\mathbf{C}$, $\mathbf{X}$, $\mathbf{G}$, respectively. By taking the prior-known encoding masks $\mathbf{C}$ and the compressive frame $\mathbf{Y}$ as inputs, a reconstruction algorithm for standard SCI $\mathcal{R}$ is applied to estimate the scene video $\ddot{\mathbf{X}}\in\mathbb{R}^{n_{x}\times n_{y}\times B}$, which contains $B$ reconstructed frames $\left\{\ddot{\mathbf{X}}_{k}\right\}_{k=1}^{B}\in\mathbb{R}^{n_{x}\times n_{y}}$:

and the equivalent exposure time $t_{x}$ of each frame of the reconstruction video is $t_{y}/B$, thus the equivalent frame rate is boosted by $B$ times. For standard SCI, $\ddot{\mathbf{X}}$ is the output as the final results, but in KH-CVS, it acts as an intermediate reconstruction, which is then fused with the key frames to achieve enhanced visual quality.

In the proposed KH-CVS, the short-exposure key frames and the long-exposure encoded frames are alternatively sampled, as shown in Fig. 2(f). For each compressive frame $\mathbf{Y}$, its two neighboring short-exposured key frames $\mathbf{Z}_{l},\mathbf{Z}_{l}\in\mathbb{R}^{n_{x}\times n_{y}}$ are

To match the equivalent exposure time $t_{x}$, the exposure time of the key frames is set to $t_{z}=t_{x}$. As illustrated in Fig. 3, each frame $\ddot{\mathbf{X}}_{k}$ in $\ddot{\mathbf{X}}$ is sequentially fused with $\mathbf{Z}_{l}$ and $\mathbf{Z}_{r}$ to produce the corresponding output frame $\hat{\mathbf{X}}_{k}$ by the fusion module $\mathcal{H}$ shown in Fig. 4:

Although the key frames contain delicate visual details, there are movements between them and the intermediate frame, causing the features or contents of them to be spatially misaligned, posing a major difficulty for image fusion. To overcome this problem, we incorporate the optical flow [38] into the model, which is able to reveal the moving velocity and direction of each pixel between two frames. As a widely used technique, optical flow can be calculated with an off-the-shelf estimator, noted as $\mathcal{F}$. Then, the key frames can be spatially warped to align with the target intermediate frame for further fusion. To this end, the first step is to estimate the optical flow from the $k$th intermediate frame $\ddot{\mathbf{X}}_{k}$ to $\mathbf{Z}_{l}$ and $\mathbf{Z}_{r}$,

Then using a spatial warping function $\mathcal{W}$, which is an interpolation operation [38] based on the optical flow, the two key frames are warped to the target frame as

Due to the relatively low quality of intermediate frames, the optical flows and warped frames in this stage are relatively rough. Thus, a neural network $\mathcal{U}_{1}$ is used to refine optical flows by estimating their residuals. Moreover, intuitively, if the target frame is temporally near to the left keyframe, the contribution of the warped left keyframe to the final fused frame should be weighted more, and vice versa. Thus, a visible map is introduced to weight the pixel values taken from two warped keyframes, which is inspired by [39]. We use one output channel of $\mathcal{U}_{1}$ with an sigmoid function to yield a visible map $\mathbf{V}$ between [0,1] to weight left keyframe and use $(1-\mathbf{V})$ for right keyframe.

Thus, the refined optical flows are calculated by

which are used to warp the key frames as

Finally, another network $\mathcal{U}_{2}$ is used to make the final output frame by synthesizing the visible maps, warped key frames with optimal optical flows, and the intermediate frame:

Using the same method, all frames in $\ddot{\mathbf{X}}$ are processed and combined to form the final video $\hat{\mathbf{X}}$.

The KH-CVS is a flexible framework that allows for a variety choices of submodules. For the demonstration in this work, E2E-CNN [8] is adopted as the basic SCI reconstruction model $\mathcal{R}$. For optical flow calculation, taking both efficiency and precision into account, a well-established PWC-Net [40] is used as optical flow estimator $\mathcal{F}$. To improve the accuracy of the optical flow, instead of directly calculate $\mathbf{F}_{k\to l}$, we calculate $\mathbf{F}_{k\to k-1},\mathbf{F}_{k-1\to k-2},\dots,\mathbf{F}_{1\to l}$, and combine the relatively small movements revealed by this set of optical flows to form $\mathbf{F}_{k\to l}$. As illustrated in Fig. 4, the networks $\mathcal{U}_{1}$ and $\mathcal{U}_{2}$ are based on U-Net [41]. Inspired by [42, 32], the shallow layers in CNN are able to extract edges and textures as contextual information to help produce better results. Therefore, we use the first convolutional layer in a ResNet18 [43] to extract the feature maps of the frames and concatenate them together before feeding them into $\mathcal{U}_{2}$. In order to improve the ability of the network to handle spatial deformation, at the end of $\mathcal{U}_{2}$, a deformable convolutional layer [44] is applied to synthesize deep latent features to produce the final output.

The model is trained in a multi-stage manner. First, the SCI reconstruction model $\mathcal{R}$ is trained following the configuration of the corresponding SCI model (which is E2E-CNN here). For the loss function, the L1 distance is adopted here, which is more favorable than L2 distance in image restoration tasks [45]. Thus, the loss function of this stage is

Then, the optical flow refinement model $\mathcal{U}_{1}$ is trained. Given that $\mathcal{U}_{2}$ is not ready at this stage, we can use the aforementioned visible map to form a linear combination of two warped keyframes to supervise the training of $\mathcal{U}_{1}$,

where $\tau=k/(B+2)\in(0,1)$ is a time factor that indicates the relative temporal distance to the left and right key frames. To supervise both synthesized frames and warped frames, the loss function for training $\mathcal{U}_{1}$ is

Finally, $\mathcal{U}_{2}$ is trained using the loss calculated with the final output as

The proposed architecture is implemented in PyTorch with two NVIDIA RTX 3090 GPUs. Adam optimizer [46] is used to train the model, with its parameters $\beta_{1}$ and $\beta_{2}$ set to 0.5 and 0.999. For training of the basic SCI model $\mathcal{R}$, the learning rate is set to $10^{-4}$ for 100 epochs and then $10^{-5}$ for another 100 epochs. Then the trained $\mathcal{R}$ is fixed and $\mathcal{U}_{1}$ is trained with the same setup of learning rate and epoch number, with $\lambda_{1}$ and $\lambda_{2}$ set to 200 and 100, respectively. Next, $\mathcal{U}_{2}$ is connected to the model and trained 50 epochs with an initial learning rate of $10^{-5}$ and then $10^{-6}$ for another 50 epochs. Finally, the whole model is jointly trained for 50 epochs at a learning rate of $10^{-6}$. For model training, 5000 video clips are sampled from the ImageNet VID [47] training set and randomly cropped to 256 $\times$ 256, with horizontal flipping randomly for data augmentation. Each video clip has 16 continuous frames (thus $B=16$) with their two neighbored key frames to serve as network inputs.

A series of simulation experiments are conducted to investigate the performance of the proposed KH-CVS on simulated datasets from ImageNet VID [47] test set and NFS [48], including Road-sign, Vehicle, Fox, Grass, Indoor and Lizard. Multiple existing algorithms are used for comparison, including GAP-TV [12], PnP-FFDNet [16], PnP-TV-FastDVDNet [9], BIRNAT [20] and RevSCI [21]. According to the analysis in Sec.2, the compressive ratio is $2/(B+1)$, i.e., 11.67% for $B=16$. In order to conduct a fair comparison, the systematic compressive ratio of the competitive counterpart algorithms should not be lower than KH-CVS, ensuring sufficient data amount for the counterparts. Therefore, for the counterparts, the same 16 frames are reconstructed from two snapshot (each encodes 8 frames in standard SCI paradigm), which corresponds to a compressive ratio of 12.5%. As the basic SCI model used in KH-CVS, the E2E-CNN [8] is also compared. E2E-CNN-8 and E2E-CNN-16 represent the models trained with $B=8$ and $B=16$, respectively.

Fig. 5 shows the exemplary frames of the reconstructed videos, which validates that KH-CVS is capable of providing higher visual quality with more fine details. The baseline models GAP-TV results suffer from strong noise and the PnP-FFDNet produces unpleasant artifacts. Among the countpart methods, the state-of-the-art algorithm RevSCI shows the best performance, yet the details in the reconstruction are relatively blurry and distorted (like the characters in scene Road-sign and Vehicle). In the basic E2E-CNN-16 model of KH-CVS, Details are also blurred out, but rough outlines are reserved. By fusion of the information from the key frames, the details can be well compensated to obtain high-quality reconstructions.

To quantitatively evaluate reconstruction quality, multiple metrics are used, including widely-used peak signal to noise ratio (PSNR) [49], structural similarity index measure (SSIM) [49]. In addition to these classic metrics, the learned perceptual image patch similarity (LPIPS) [50] is also used to evaluate perceptual visual quality, which is calculated in the deep feature space as perceptual distance. Note that a lower LPIPS means the reconstructions are more similar to the ground-truths. Since the key frames are identical to their corresponding ground-truth frames in the simulation, metrics are calculated only on the reconstructed frames to prevent privilege of KH-CVS. Quantitative results are summarized in Tab. 1. It can be observed that the proposed KH-CVS achieves 1.05 dB improvement in PSNR and 0.0236 in SSIM on average, compared to the best results of counterparts, i.e., RevSCI. KH-CVS reduces the averaged LPIPS to about half of RevSCI, which reveals that the KH-CVS reconstruction frames are perceptually more closed to the ground-truth frames. It is also worth mentioning that, compared to its basic SCI model E2E-CNN-16, KH-CVS boosts PSNR by 5.11 dB and SSIM by 0.1416, and decreases LPIPS to about only 17.8% of itself, which shows that the information from the key frames can help significantly improve the performance of the basic SCI model.

The inputs and intermediate results of the model are visualized in Fig. 6. Here the 12th frame $\hat{\mathbf{X}}_{12}$ is used as an example. To visualize the optical flow field, we color code the optical flow field following [38], that is, using the color hue to represent the direction of motion and the saturation to represent the magnitude. In the scene of Road-sign, as the vehicle moves forward, the road-sign are from far to near, so from the left key frame $\mathbf{Z}_{l}$ to $\ddot{\mathbf{X}}_{12}$, the road signs are expanded outward, and the preliminarily estimated optical flow $\mathbf{F}_{12\to l}$ is consistent with this observation. Through this optical flow, $\mathbf{Z}_{l}$ can be warped to $\ddot{\mathbf{X}}_{12}$ as $\mathbf{W}_{12,l}$. Because $\mathbf{F}_{12\to l}$ is relatively rough, the warp results $\mathbf{W}_{12,l}$ also have errors (such as the position of the characters and the edge of the image). After refinement, the optical flow $\mathbf{F^{\prime}}_{12\to l}$ is more accurate and can distinguish the area of the road-sign with a displacement and the area of the relatively static background. At this time, the warped result $\mathbf{W}_{12,l}^{\prime}$ is also enhanced. The behavior of the right key frame $\mathbf{Z}_{r}$ is similar to that mentioned above. Since $\mathbf{Z}_{r}$ is temporally nearer to $\ddot{\mathbf{X}}_{12}$ here, it is supposed to provide more information than $\mathbf{Z}_{l}$, which is also consistent with that revealed by the visible map $\mathbf{V}$. It can be seen that, compared to the intermediate reconstructed frame $\ddot{\mathbf{X}}_{12}$, the final output frame $\hat{\mathbf{X}}_{12}$ has a significant improvement in the details of the scene, which is closer to the ground-truth ${\mathbf{X}}_{12}$.

In the process of capturing, the image sensor needs time to readout the image data and clear the sensor to prepare for next image, thus the frame gap (noted as $t_{g}$) exists in real systems. This gap also occurs between two snapshot in the standard SCI. Since KH-CVS fuses multiple frames to make reconstruction with considering temporal consistency of the scene, its robustness to the gap between measured frames is needed to be evaluated. In this subsection, a series of experiments on simulated datasets are conducted to evaluate the robustness to the frame gap. In the ideal case, the $B$ frames to be compressed and the two key frames are continuously sampled from the video dataset. In order to simulate the actual frame gap, we skip some frames between the key frames and $B$ compressive frames during this test. For example, when $t_{g}=2t_{x}$, it means that there are 2 skipped frames between the left key frame $\mathbf{Z}_{l}$ and the first compressive frame $\mathbf{X}_{0}$, and also 2 frames between $\mathbf{Z}_{r}$ and $\mathbf{X}_{B}$, thus the normalized frame gap $t_{g}/t_{y}=2/16=12\%$. In order to evaluate the effect of frame gap on reconstruction quality, we sweep the frame gap from 0 to 4 frames, corresponding to the $t_{g}/t_{y}$ changing from 0 to 25%, and the results are presented in Fig. 7. It can be seen from the results that with the increase of the $t_{g}/t_{y}$, the quality of reconstruction decreases, but the trend is relatively slight. From the ideal situation to $t_{g}/t_{y}=25\%$, the PSNR drops by 0.61 dB, the SSIM drops by 0.0092, and the LPIPS rises by 0.0066, while all metrics are still better than the best of the counterpart methods (marked by the black triangle). This shows that KH-CVS is robust to the frame gap. We experimentally found that setting $t_{g}$ as 300 $\mu s$ is enough for preventing missing the next trigger pulse and allows a continuously sampling. For the demonstration in this paper, $B=16$, $t_{y}$ is 33328 (see below) $\mu s$, resulting in $t_{g}/t_{y}$ less than 1%. Thus the reconstruction model is capable to deal with the frame gap in this system setup.

We used the KH-CVS prototype to record the actual scenes, and the experimental results are shown in Fig. 8. We compare KH-CVS ($B=16$) with standard SCI ($B=8$) in the same scene. The period of DMD trigger signal is set to 2083 $\mu s$, corresponding to an equivalent frame rate of about 480 frame per second (fps). In KH-CVS, the duration of short exposure $t_{x}$ is also 2083 $\mu s$ and that of long exposure $t_{y}$ is set to 33328 $\mu s$. For standard SCI, the exposure time of compressive frames is set to 16664 $\mu s$, which is approximately half to that of KH-CVS, so that the equivalent frame rates in the reconstructed videos of the two systems are kept equal. Note that due to the difference of exposure setups, the brightness of captures are different. In the experiments, the short exposed keyframes are darker than the compressive frames. Before entering the algorithms, they are normalized to keep the same mean value to the intermediate reconstruction frames. This strategy is a common operation that can be also seen in burst high-dynamic-range imaging [51]. For reconstruction, GAP-TV, PnP-TV-FastDVDNet, E2E-CNN-8 and RevSCI are used for standard SCI, and the KH-CVS results are reconstructed using the model proposed in this paper (the basic SCI model in KH-CVS is E2E-CNN-16). Based on this setup, first, a rotating fan, with letters marked on several blades of the it, serves as the target scene. The measurements and key frames [Fig. 8 (a)], the reconstructions [Fig. 8 (b)] of standard SCI methods and KH-CVS are shown. In all standard SCI results, the rough outline of the fan blade can be recovered, but the details are relatively blurred and severely distorted by strong artifacts. In the KH-CVS results, the stroke of the character can be clearly recovered. In another scene, we capture a moving toy dinosaur [Fig. 8 (c)-(d)]. Since the object is moved manually and the compressive frames of $B=8$ and $B=16$ are captured one after another, the location of the objects and the motion trajectories are not strictly identical. In the results of standard SCI, the texture of the eye of the toy dinosaur is difficult to resolve with its shape distorted, and the teeth are also blurred. In the results of KH-CVS, the surface texture and edges can be clearly seen with the oval shape of the eye well maintained, and details of the teeth can also be recovered, which also reveals that KH-CVS works well for both bright and dark areas. The results show that the KH-CVS proposed in this paper can achieve better visual quality than standard SCI in real high-speed scenes.

To further test the reconstruction quality of KH-CVS, we used a moving USAF 1951 resolution chart as a target. Note that the resolution chart here was printed by us, which is different from its original scale, but can still be used for relative resolution comparison. Measurements for standard SCI and KH-CVS and key frames are shown in Fig. 9(a). Exemplary reconstructed frames are shown in Fig. 9(b), along with line profiles and zoomed-in views. The blue line plots the horizontal part of group 2, elements 2 to 6, and the red line shows the profiles of group 2, element 1 (vertical parts). For the line profiles, the vertical axis is the grayscale pixel value (from 0 to 255), and the horizontal axis is the normalized spatial distance. The zoomed-in views show the details at group 1, element 5 and 6. In the standard SCI results obtained by the counterpart methods, the bars are affected by strong artifacts and distorted, and the digits are completely blurred. In KH-CVS, they can be clearly distinguished, with better contrast and almost no artifacts.