Improving Variational Autoencoder based Out-of-Distribution Detection for Embedded Real-time Applications

Recurrent neural networks (RNN) are a common choice of architecture for learning time series data. In (li2018anomaly, 21), a GAN-based anomaly detector was designed to learn inputs from sensors and actuators during the normal working conditions of a CPS. The sensors’ and actuators’ inputs are low dimensional. For high dimensional image video, an RNN needs to be huge based on its neuron count, rendering it less computationally feasible for real-time CPS on embedded systems.

Optical flow is a classical computer vision technique widely used to estimate objects’ actual movement in the physical world by computing changes in the neighboring pixels’ intensity over time. If we want robust OoD detection for unusual motion, optical flow is an ideal feature extraction mechanism due to its deterministic computational outcome. Let $I(x,t)$ denote pixel intensity in image frames and $\nabla x$ be the unknown flow velocity in 2D space. The optical flow constraint equation $I(x+\nabla x,t+1)=I(x,t)$ assumes that the pixel intensity in a local region remains constant over a very short time frame. Solving the partial differential Equation 2 gives us the flow velocity. $n$ is the order of the Taylor series.

$\nabla x$ is a 2D vector field that represents surrounding object motions between consecutive image frames. A type of flow field example of driving on an open road is shown in Figure 1, where far-away background semantics at the front view are filtered out, relative motions of the closer front road surface and lamp pole are transformed into motion vectors. We found that anomaly event detection in video surveillance (hu2004survey, 22) to be broadly relevant to our work. However, unlike in the surveillance setting, where camera locations are fixed, motion in video feeds from a driving scene moves faster. The unique challenge here is to discriminate optical flow vectors caused by genuine OoD motions from those caused by continually changing backgrounds and other ID motion patterns in the environment. Today, the use of a deep neural network probably is the most effective technique to tackle this challenge . (yu2021hmflow, 23, 24)

Our motion OoD detection problem is different from anomaly detection for autonomous vehicles. Our problem deals with monitoring the potentially unreliable ML-enabled components in autonomous vehicles, specifically, the machine perception system. Anomaly detection in autonomous vehicles is not limited to the vision system (i.e., a extremely high data dimension problem). For example, non-vision low-dimension data from vehicle sensors are utilized in (guo2019detecting, 25) to to detect anomalies from pair-wise data correlation, such as between the acceleration and wheel torque. The approach proposed by (ryan2020end, 26) is framed in the end-to-end autonomous driving context. However, this approach detects for anomalies in vehicle control (steering, braking, and accelerating) through training a CNN model to learn the correlation between input driving video and human driving behavior. Although both approaches tackle the safety challenge in autonomous driving, our OoD detection problem focuses on the cause of uncertainty arising from machine vision. In contrast, anomaly detection focuses on the unexpected outputs from the vehicle controller.

To this end, we propose to extract ID motion series from image video into 3D optical flow tensors. While the spatial dimension of 3D optical flow remains the same as input images, the signal complexity is dramatically reduced. Therefore, it is feasible to train the time series with significantly fewer 3D convolutional network blocks. Let $\nabla x_{h,t}$ and $\nabla x_{v,t}$ be the 3D optical flow tensors for the horizontal and vertical directions respectively, the two 3D convolutional VAE encoders individually learn the input manifold, forming two separate latent sub-spaces. The latent sub-spaces are then concatenated as one input and then fed to the VAE decoder. The objective of our VAE network is to minimize the loss as given in Equation 3a, where $\phi$ and $\theta$ are model parameters of the encoder and the decoder respectively. The second and third terms represent two encoders measuring the KL-divergence in the horizontal and vertical latent sub-spaces. The first term refers to the decoder measuring the likelihood between encoders’ input $\nabla x_{t}$ and the decoder’s reconstruction $\nabla\hat{x_{t}}$ given input from the latent sub-spaces. A high level diagram of the network architecture is depicted in Figure 1.

Bernoulli distribution, as proposed in (kingma2013auto, 6), is optimal for modeling black and white image data like MNIST because pixels value are either 0 or 1. Optical flow values and normalized color image pixel values are real numbers in a continuous domain. In practice, we adopt a common approach that modeling the training data with a Gaussian distribution with identity covariance. Hence a Gaussian decoder, where the negative log-likelihood term in Equation 3a is transformed into mean square error (MSE) in implementation, as shown in Equation 3b. See Appendix B for the derivation steps. M is the input space dimension, a multiplication of length, width, and depth of an input. OoD score is a summation of $\sum_{i=1}^{n}D_{KL}$ in horizontal and vertical latent sub-spaces. $n$ is the number of dimensions in the latent sub-space. Experiments with depth values of 6 and 1 showed that the proposed feature abstraction in spatiotemporal space leads to the robustness of OoD detection.

Can hazardous motion features still be robustly extracted when OoD factors that affect visibility, such as fog, co-occur? In Figure 2, we visualized the flow vectors of two OoD motion examples. Flow vectors shown in the bottom row are computed from the pair of image frames from the row above. Clean image sequences were obtained from (bengar2019temporal, 15). The fog was rendered over the clean sequences with dynamic effects through Autodesk Maya software. Both cases simulate that a car is driving ahead. Optical flow vectors in the front view are visualized as colored vectors. A cluster of red represents the prominent motion between two video frames. We can see that the fog almost has no impact in both cases. Qualitatively, we showed that a robust representation of motion around an autonomous vehicle can be extracted through optical flow under diverse visibility conditions. Since this representation is abstract, environmental background semantics will not be encoded into the latent space, which encourages model generalization and this better suits to driving environments that are different from the training data set. In related works (shreyas:emsoft2020wip, 17, 8), an informative subset of the latent space has to be experimentally selected for good OoD detection performance. A consequential benefit of this proposed method is that the entire latent space or a particular sub-space can be used for OoD detection.

Commonly, a simple prior $\mathcal{N}(0,\mathbf{I})$ is used to represent the true but unknown latent prior $p(z)$, and the KL-divergence is used to compute the distribution discrepancy between the encoder output and the prior, as given in Equation 4.

For robust OoD detection in the latent space, it is critical that the posterior accurately approximates underlying ID data properties. Intuitively, assuming that any underlying data follows a simple prior $\mathcal{N}(0,\mathbf{I})$ is not optimal. One way for improvement is to optimise the prior and not just the encoder and decoder (hoffman2016elbo, 27).

Substituting the simple prior with an approximation of aggregated posterior $q(z)=\frac{1}{N}\sum_{n=1}^{N}q(z\mid x_{n})$ was proposed by (tomczak2018vae, 28). A Student’s t-distribution prior allowed a more robust approximation of the underlying data in the presence of outliers (abiri2020variational, 29). A Dirichlet prior was chosen by (xiao2018dirichlet, 30) to model a latent representation of categorical distribution, specifically the class labels distribution, for a downstream object classification task. Since the input and latent spaces in our VAE proposal encode the same physical meaning but because the latent dimension is dramatically reduced, we decided to identify an optimal prior from model’s training data.

We select a random subset from training data and plot a histogram of optical flow values $\nabla x$, as shown in the left column of Figure 3. Distributions of flow values in both data sets approximate to Gaussian in horizontal and vertical directions but with very small deviations. In practice, we replace the variance of the simple prior $\mathcal{N}(0,\mathbf{I})$ with $\sigma_{p}$ approximated from a random partition of the training set $\nabla x$.

Since the $\sigma_{p}$ value is very small here, training with Equation 4 as the objective will be unstable. To overcome this, we adopt a novel autoencoder formulation from (tolstikhin2017wasserstein, 13), the Wasserstein Auto-Encoder (WAE). Motivated by the transportation theory in mathematics, the objective of WAE is to minimize the transportation cost between true data distribution $p(x)$ and a generative model, i.e., the decoder modeled after latent distribution $q(z\mid x)$, denoted as $c(x,z)$ in Equation 5. Notation $\mathcal{D}$ in the second term of the objective is a distance metric that encourages the encoded training distribution to match its prior $p(z)$. Both the cost and metric functions can be arbitrary, so WAE is a non-parametric probabilistic model. $\lambda>0$ is a hyperparameter. With WAE, we can use a metric other than KL-divergence as an OoD measure and any reconstruction error function to regularize the decoder.

With this new objective, we revise the objective in Equation 3 by substituting the $D_{KL}$ penalization with a 2-order Wasserstein metric $D_{W2}$, as given in Equation 6. The decoder is still regularized by MSE loss and $\lambda=1$. For isotropic Gaussian, covariance matrix $C_{p}=\sigma_{p}^{2}\mathbf{I}$ . A $D_{W2}$ penalization ensures the training is stable even when $\sigma_{p}$ is close to zero.

How will the optimal prior impact the OoD scores $D_{W2}$? Will the optimal prior lead to model overfitting? The box plots of $D_{KL}$ and $D_{W2}$ in Figure 3’s right column show that the difference between mean OoD scores of ID and OoD rain test samples increases in the model trained with an optimal prior. In experiments, we observed a consistent detection performance enhancement over test samples from the same or from different data sets, which indicates that the proposed optimal prior does not cause overfitting. Refer to Section 4.3 for a quantitative evaluation.

Section 3.1 explained that the motion abstraction with optical low is robust under diverse visibility conditions. However, we can’t rely to this abstraction when visibility is close to zero. To address such extreme visibility conditions, we propose an improvement to OoD detection in the input space.

In Section 2, we introduced the likelihood overlap problem and commented that most existing likelihood compensation approaches $\log p(\hat{x}\mid M)-c(x,\hat{x})$ are not suitable for real-time CPS application because of the extra computing cost incurred. There is an exception where authors of (serra2019input, 20) proposed to compensate with an image complexity ratio estimated by compression algorithms such as JPEG2000.

Instead of compression ratio, we use image entropy as a complexity measure to study the likelihood overlap problem. Information entropy is a widely used complexity measure not limited to just images. 2D image entropy measures the complexity in local neighborhoods of an image. From the histograms of image entropy in Figure 4, we can see that the fog factor does cause a considerable reduction of entropy while the rain factor almost has no impact. Also, images in the nuScenes data set are more complex than the benchmark data sets such as MNIST and KMNIST.

Inspired by related works, we propose to improve the detection of low visibility conditions by calibrating the likelihood with a data-driven baseline. The baseline is the mean entropy of model’s training data, denoted as $entropy(M)$ in Equation 7. $entropy(x)$ is the mean entropy of a test image. There does not exist ubiquitous background statistics in our problem, as in the genomics sequence study (ren2019likelihood, 19). So the likelihood is compensated only when a test image’s mean entropy is lower than the baseline. In experiments, we use $MSE(x,\hat{x})$ instead of likelihood $\log p(\hat{x}\mid M)$.

How effective is this metric compensation proposal? In Appendix A, we demonstrated that a model trained with KMNIST cannot differentiate MNIST images from KMNIST. Their likelihoods histogram almost overlap. Since most MNIST images can easily be separated from KMNIST images by entropy (see in Figure 4), calibrating likelihoods improves cross data-domain OoD detection performance from 0.504 to 0.939 in AUROC.

We shall note that this likelihood calibration is explicitly designed for robust detection of low visibility factors in an autonomous driving setting. Its effectiveness is evaluated in Section 4.2 using visibility factors such as fog and low light.

We implemented two of the most closely related OoD detection methods, (cai:iccps2020, 16) and (shreyas:emsoft2020wip, 17) from the autonomous CPS domain, for performance comparison. The performance advantages of our light-weighted spatiotemporal detection method were demonstrated with input sequences of different lengths in time. Our improvement to the likelihood overlap issue was validated. Experiment details are summarized in the below.

OoD factors and Experiment Data Sets We are interested in detecting factors that are potentially dangerous to safe driving. OoD factors evaluated were close hazardous motion, rain, fog, and low light. We selected a driving simulation data set Synthia (bengar2019temporal, 15) and a real-world driving data set nuScenes mini (caesar2020nuscenes, 31) to partition into training and test sets.

Synthia has 288 video scenes captured from a driving car in a virtual world at 25fps with a scene length between 10 to 30 seconds and a rich mixture of weather, landscape (i.e., city, town, and highway), and dynamic and static objects (car, pedestrian, bicycle, wheelchair, veneration, traffic light, etc.). We manually curated 94 video scenes from Synthia to form a training set of 27519 images. The nuScenes mini has 10 driving video scenes recorded in city environment. The first 48 frames of each nuScenes mini video were kept for testing, and the remainder of 1870 images were used for model training. Both training sets only include ID data.

Details of test sets are summarized in Table 1, where the source and sample numbers of images or video episodes for each class are listed in the third column. OoD classes starting with \saycomp were generated via rendering dynamic effects rain and fog over ID image sequences using Autodesk Maya software. Images in the darkness OoD class were generated by reducing ID images’ brightness in the HSV color space. OoD classes starting with \saysim are original images from the Synthia data set. In the sim.motion OoD class, each episode is 60 frames long with at least one motion OoD occurrence. Some examples of ID and OoD images are shown in Figure 5.

VAE networks To fairly compare the three detection methods, we used an identical CNN architecture for the encoders and decoders, because this network architecture was used by (cai:iccps2020, 16, 17). Their corresponding comparison studies will be carried out in Sections 4.2, 4.3, 4.5 and 4.7. All input images were resized to 120x160 in pixels. The encoder has four CNN blocks of 32/64/128/256 x (5x5) filters with exponential linear unit (ELU) activation and BatchNorm. The decoder is architecturally symmetric. The extra layers in (cai:iccps2020, 16, 17) networks are two MaxPool in the encoder and two MaxUnPool in the decoder, after the CNN blocks. Whereas our network does not have these pool layers as we use larger strides in the CNN blocks.

The dimension of latent space varies. Following the original proposals, the latent dimension is 1024 and 30 for (cai:iccps2020, 16) and (shreyas:emsoft2020wip, 17), respectively. The latent dimension in our method is 12 for each sub-space. The depth of 3D optical flow is 6, computed from 7 consecutive image frames. In sections 4.4 and 4.6, we show how different numbers of CNN layers, dimensions of latent space, and depth of input size influence the overall performance of the proposed OoD detection methods.

Hyperparameters All VAE models were trained with the Adam optimiser using PyTorch. For the Synthia training set, model (cai:iccps2020, 16) was trained with constant learning rates $10^{-4}$ and $10^{-5}$ for 200 and 150 epochs, respectively. Model (shreyas:emsoft2020wip, 17) was trained with a constant learning rate $10^{-5}$ for 100 epochs. These learning rates and training epochs were adopted from (cai:iccps2020, 16, 17). We tested a faster learning rate $10^{-3}$, however it degraded the model performance in these methods. Our VAE model was trained with a constant learning rate $10^{-4}$ for 100 epochs.

Increasing total training epochs did not enhance performance with the Synthia training set but was the opposite with the nuScenes-mini training set. After a quick search, all corresponding nuScenes models were trained with 600 epochs.

In (shreyas:emsoft2020wip, 17) method, we set the $\beta$ value to 1.4 and selected the nine latent variables that encode the most information, as proposed in the original paper. See Appendix D for the selection algorithm.

Evaluation metrics Equation 8 defines the OoD calculation formula used to analyze experiments in this paper. It is a summation of distribution discrepancies between a test sample and the latent priors in horizontal and vertical sub-spaces.

We used AUROC to compare the performances of OoD detection methods across different score thresholds. In true positive rate (TPR) 95%, i.e., tolerating 5% miss-out, we compared point performances of OoD detectors in precision and F1 score metrics. When TPR is fixed, precision, $TP/(TP+FP)$, reveals the false positive rate, while the F1 score conveys the balance between precision and sensitivity $TP/(TP+FN)$.

In this section, we first compared the performances of three VAE-based detection methods over the motion OoD group. Next, the benefit of optimizing prior distribution was analysed over the motion OoD group. Finally, the extra performance benefit from detecting OoD in a latent sub-space was shown. For simplicity, our detection method proposed in Section 3.1 is termed as bi3dof and its enhanced version proposed in Section 3.2 as bi3dof-optprior.

We believe that without an input space abstraction, the semantic meanings of environmental background will be encoded into the latent space, which discourages a model from generalizing to driving environments that differ from training. However, as pointed out in previous sections, related works only reported their detection performance on test sets partitioned from the same data set that was used for training.

In this section, we examined how the models in Section 4.2.1 behave if test sets are from a source different from a model’s training set. We denote such tests as cross-sets. An example of a cross-set test is to have the nuScenes test set evaluated against a model trained on Synthia data.

From Figure 8, we can see bi3dof models trained on the nuScenes set generalize perfectly on Synthia test sets across all OoD classes. As the motion ID patterns in the nuScenes training set are more complex than those in the Synthia set, we even observed a trivial performance increase for the Synthia comp.rain class. The maximum performance drop is 0.026 over the sim.motion class. Models trained on the Synthia set generalize slightly less perfectly, having a drop between 0.053 and 0.064. This observation confirms that for a robust ML model, it is vital that an application’s population data is well sampled into the model’s training set.

While models trained with bi3dof methods generalize very robustly, we can see models from both the related works failing badly. Six out of eight cross-set tests produced AUROC values around or below 0.5. An AUROC value less than 0.5 indicates a classifier’s performance is worse than a random guess, suggesting that the score over ID samples may be statistically higher than OoD samples in most of the cross-set tests here. A relatively good outcome was observed in (cai:iccps2020, 16) on the nuScenes cross-set test, where the AUROC value decreases by 0.151.

Besides the latent sub-spaces, another novelty in our detection method is enhancing OoD detection’s robustness through directly learning 3D optical flow with a sufficiently long temporal window. If a vehicle is moving at 50 km per hour and the video is captured at 25 frames per second, a depth value of 6 encompass motions spanning over 3.3 meters in 240 milliseconds. If the depth value is 1, each optical flow input data point would covers only motions over half a meter. Intuitively, it is less effective in exploiting the motions’ correlation over time. Using the nuScenes training set and $depth=1$, we trained a model with optimal prior (i.e., bi2dof-optprior) to validate the effectiveness of learning from spatiotemporal data points. The result in Table 4 shows, in both same and cross-set tests, 3D optical flow at depth 6 clearly improve detection performance by a big margin. Specifically, precision values of sim.rain and sim.motion OoD classes in cross-set tests almost double, which indicates false alarm rate dropped dramatically. The 2D optical flow requires less computing capacity, which is an advantage in resource-limited situations.

So far, our analysis focused on motion OoD detection in the latent space of VAE. This section discusses experimental outcome of enhanced input-space-based detection proposed in Section 3.3. We used (cai:iccps2020, 16)’s method as a baseline to compare OoD classes’ performances in the visibility group as given in Table 1.

Following the definition in Equation 7, each OoD score is compensated by mean image entropy ratio between the model’s training set $entropy(M)$ and the individual test sample. We used VAE models from Section 4.2.1 to compute a raw OoD score $MSE(x,\hat{x})$ of a test sample. The baseline entropy, $entropy(M)$, for Synthia and nuScenes models are 4.657 and 5.364, respectively. The difference in $entropy(M)$ values indicates images in the nuScenes data set are averagely more complex than images in the Synthia data set.

Since there are night scenes in the nuScenes training set, darkness detection cannot be tested on nuScenes as dark scenes are considered ID. From Table 5, we can see the entropy compensated OoD metric improves average detection capacity in both Synthia and nuScenes data sets, with negligible impact on the motion OoD classes. Higher precision values over the comp.fog class in nuScenes and darkness class in Synthia indicate that the false alarm rate was reduced. However, the new metric does not improve the F1 score and precision over the Synthia comp-fog class. A pair of comp.fog images in Figure 5. We think this could be a simple mean entropy (in Equation 7) is not effective for very bright images. A statistically more sophisticated entropy score might worth looking into in future work.

In our comparison studies, all encoders and decoders have four conv-blocks, where each conv-block comprises of one convolutional neural layer, one ELU activation layer, and one BatchNorm layer. In this section, we investigate whether this network choice is optimal for our light-weighted spatiotemporal and entropy-compensated OoD detectors. We experimented with narrower or wider latent dimensions and reduced conv-blocks. The results is summarized in Figure 9. For the bi3dof-optprior detection method, we can see increasing the dimension of latent space leads to higher performance. When the size of latent space is the same, a network of four conv-blocks consistently outperforms the network with three conv-blocks. Based on the performance curve, four conv-blocks with 12 dimensions for latent sub-space is a good design choice that balances between performance and computational cost. A choice of three conv-blocks with 18 dimensions for latent sub-space is reasonable if one trades performance for less computational cost. For the entropy-compensated OoD detection method in VAE input space, one more conv-block contributes significantly to detection performance. However, when the latent space dimension is too high, detection performance drops. This can be understood as the information bottleneck required to distill proper abstract representation from underlying data was not formed. A network of four conv-blocks with 1024 dimensions is the optimal choice among all the network parameter space searched.

The execution time of an detector matters for OoD detection in real-time, as well as its memory footprint. The average execution times of each detector on a workstation with an Intel Xeon 3.50GHz is shown in Table 6. The pre-process computation includes image resizing and optical flow operations in the case of bi3dof. The VAE model inference in (cai:iccps2020, 16) takes the longest execution time because the decoder computation is required. For the same reason, (cai:iccps2020, 16)’s memory footprint is the largest. Our method has the lowest number of dimensions in the latent space, hence the smallest memory footprint. Its execution time is equivalent to (shreyas:emsoft2020wip, 17). With one execution below 10 milliseconds, all three detection methods can efficiently do real-time OoD detection on general purpose desktop computers. If the neighboring depth-6 3D optical flow results cannot be reused, for example, a frame drop happened while streaming the input video, the total execution time will take 26.3 milliseconds.

The RPI-3B+ has a ARMv8 CPU at 1.4Hz. Averagely, one optical flow operation takes 31.7 milliseconds, and one model inference takes 117.7 milliseconds. If all RPI-3B+ computing resource is dedicated to the OoD detector, to achieve zero frame drop in real-time means the video streaming has to be set at 6 frames per second (FPS). In the meantime, the bi3dof model used in Section 4 experiments requires 7 consecutive frames from the input video stream to do one OoD detection process. This will incur a long time delay between the image frames captured and when an OoD result is produced, which does not make it suitable for real-time applications on common off-the-shelf embedded devices.

To adapt to the RPI-3B+ CPU, we used the bi2dof-optprior model trained on the nuScenes training set. After reducing the input space dimension, the model takes about 200 millisecond to produce an OoD result on a RPI-3B+ mini-computer. However, the model inference time of bi2dof is still very long, compared to 7.6-26.3 milliseconds of the bi3dof model running on a workstation. After pushing the model onto the Edge TPU, the VAE model inference time can be reduced on average over four times. In the rest of this section, we discuss the pros and cons of applying model compression, including its impact to the model’s detection capability and compare with related works in Section 4.

DNN models rely on millions or even billions of parameters to achieve superb prediction abilities. For example, the famous AlexNet has over 60 million parameters that require floating point operations. Our floating-point CPU model has 2.2 million parameters. Graphics processing units (GPUs) have been the horsepower for training and inference of DNN models. In recent years, Tensor Processing Units (TPUs), together with various DNN model compression and acceleration techniques, are overcoming challenges in deploying DNN models to embedded devices with limited resources. Parameter quantization is a model compression technique that reduces the number of bits required to store each parameter. With a minimal loss in classification accuracy, train and infer in 8-bit quantized integers, a running time reduction of 50% was reported on the open-source TensorFlow (jacob2018quantization, 34).

Instead of training a new model in 8-bit directly, we took the 32-bit floating point PyTorch model (bi2dof-optprior) from Section 4.5 and applied a full integer post-training quantization. The quantized model is then compiled into the Google Edge TPU’s supported format. The conversion workflow is outlined in Figure 10. Applying the same procedure, we compressed our implementation of (shreyas:emsoft2020wip, 17) in Section 4 and a variant implementation of (cai:iccps2020, 16). Model graphs of all final outputs and intermediate output of our model are shown in Appendix C.

A variant implementation is needed because as (cai:iccps2020, 16) does OoD detection in the input space, its decoder is required and needs to be compressed. However, the OpenVINO model optimizer doesn’t support the MaxPool layers in the encoder to pass on pooling indices to the MaxUnPool layers in the decoder. As such, we removed all MaxPool and MaxUnPool layers in the variant implementation. This ended up increasing a significant number of neurons to the linear layer after pooling in the variant model.

The results of computational cost are summarised in Table 7. The compressed 8-bit-integer bi2dof-optprior, is 3.6 times smaller and 2.6 times faster than its floating-point model. The compressed 8-bit-integer version of (shreyas:emsoft2020wip, 17) is 3.8 times smaller and 11 times faster than its floating-point model. A lower end-to-end reduction ratio of execution time in bi2dof-optprior is because two model inferences are invoked in the integer version, one for the horizontal and one for the vertical branch, including an extra pre-process computing (optical flow). The 8-bit-integer version of (cai:iccps2020, 16)-variant is 3.9 times smaller than its floating-point model because most of the decoder’s layer computation could not to be pushed onto the Edge TPU, as shown in Appendix C, Figure 15. The execution time however, increased by 42% instead, which implies that the time savings from the computation of the encoder branch on Edge TPU was wiped out by the time spent transmitting the decoder’s intermediate result back to the host via USB 2.0.

Quantization approximates floating-point values into integers with $int8\_value=real\_value/scale+zero\_point$. It has been reported that its impact on image classification models’ performance is minimal. For example, the ImageNet trained Inception_V4 top-1 accuracy is 80.1%, as reported in the TensorFlow guideline. Its corresponding quantized model top-1 accuracy is 79.5%. In classification tasks, the integer vector output can be used directly for class prediction.

In our case, integer-8 bi2dof-optprior outputs are mean and variance values to be plugged in to Equation 8 to derive an OoD score. We used the scale and zero_point values shown in Appendix C, Figure 13 for converting integer values to floating values and vice versa. The process requires a typical data set for calibrating the range of floating-point input tensors. We used bi2dof’s training set as the typical data set. a same compression workflow was applied to (shreyas:emsoft2020wip, 17) and (cai:iccps2020, 16)-variant.

Comparing detection performance of 8-integer models in Figure 11 with corresponding floating-point versions, we can see that the quantized model’s performance dropped a little across all three. While precision is tired up, the F1 score of integer-8 bi2dof-optprior remains higher than the rest. Performance-wise the 8-integer (cai:iccps2020, 16)-variant is closer to our method, but it is not deployable on the RPI + Edge TPU platform. Because its end-to-end execution time is 492 milliseconds.

The result from integer-8 bi2dof-optprior is comparable with its corresponding 32-bit floating-point model, although its performance loss is not minimal as compared to well-known classification tasks. The conversion of integer outputs to float values for OoD score calculation could be one reason for losing more precision in our case.

We explored the resource challenges of deploying a DNN-based OoD detector onto off-the-shelf embedded hardware platforms for real-time detection. It showed a compressed version of our detector completed one detection at 58.3 milliseconds at an accuracy comparable with its corresponding floating-point model. That is, given an embedded hardware platform with RPI-3B+ and Google Edge TPU dedicated for OoD detection, our compressed integer-8 bi2dof-optprior, can do one execution without any frame-drop when streaming real-time video at 17 FPS. On the other hand, we showed that reducing input data point in the temporal dimension caused the OoD detection to be less robust. One optical flow operation on the aforementioned CPU takes 31.7 milliseconds (see in Table 7). Handling input data points with a larger temporal dimension on off-the-shelf embedded hardware platforms such as Rasberry Pi is challenging. One option for future implementation and deployment of our OoD detector onto a similar embedded platform is to design a FIFO buffer to cache the previous optical flow computation results for a bi3dof OoD detector to reuse. In this feasibility study, we established a bottom line on performance and computational cost.