Single-Frame based Deep View Synchronization for Unsynchronized Multi-Camera Surveillance

Without loss in generality, we choose one view (denoted as view $0$) as the reference view, and other views are to be synchronized to this reference view. We first assume that synchronized frame pairs are available in the training stage. The frames are $I_{0}^{t_{0}}$ from reference view $0$ captured at reference time $t_{0}$, and $I_{i}^{t_{0}}$ and $I_{i}^{t_{i}}$ from view $i$ ($i\in\{1,2,..,n-1\}$) taken at times $t_{0}$ and $t_{i}$. Note that frames $(I_{0}^{t_{0}},I_{i}^{t_{0}})$ are synchronized, while $(I_{0}^{t_{0}},I_{i}^{t_{i}})$ are not.

The synchronization module consists of the following stages. First, camera frame feature maps $(F_{0}^{t_{0}},F_{i}^{t_{0}},F_{i}^{t_{i}})$ (both synchronized and unsynchronized frames) are extracted and projected to the 3D world space, resulting in the projected feature maps $({\cal F}_{0}^{t_{0}},{\cal F}_{i}^{t_{0}},{\cal F}_{i}^{t_{i}})$. Second, synchronization is performed between the reference view $0$ and each other view $i$. The projected feature map ${\cal F}_{0}^{t_{0}}$ from the reference view is concatenated with the projected feature map ${\cal F}_{i}^{t_{i}}$ from view $i$, and then fed into a motion flow estimation network ${\cal M}_{s}$ to predict the scene-level motion flow $w_{i}$ between view $i$ at time $t_{i}$ and the reference view at time $t_{0}$:

$\displaystyle w_{i}={{\cal M}_{s}}(\mathrm{Cat}({\cal F}_{0}^{t_{0}},{\cal F}_{i}^{t_{i}})),i\in\{1,...,n-1\},$

(4)

where $\mathrm{Cat}$ is the concatenation operation. The ${\cal F}_{i}^{t_{i}}$ from view $i$ is then synchronized with the reference view at time $t_{0}$ using a warping transformation ${\cal W}$ guided by $w_{i}$, ${\cal W}(w_{i},{\cal F}_{i}^{t_{i}})$,

$\displaystyle\hat{{\cal F}}_{i}^{t_{0}}={\cal W}(w_{i},{\cal F}_{i}^{t_{i}}),i\in\{1,...,n-1\},$

(5)

where $\hat{{\cal F}}_{i}^{t_{0}}$ are the warped projected features of the $i$-th view synchronized to time $t_{0}$. Note that the warping $\cal W$ only applies spatial shifting to the feature map ${\cal F}_{i}^{t_{i}}$, i.e., it only changes the feature locations and does not change the feature values. Finally, the reference view features ${\cal F}_{0}^{t_{0}}$ and estimated warped features of the other views $\{\hat{{\cal F}}_{i}^{t_{0}}\}$ are fused and decoded to obtain the final scene-level prediction $V_{p}$:

	$\displaystyle V_{p}$	$\displaystyle=D(U({\cal F}_{0}^{t_{0}},\hat{{\cal F}}_{1}^{t_{0}},\ldots,\hat{{\cal F}}_{n-1}^{t_{0}}))$		(6)
		$\displaystyle=D(U({\cal F}_{0}^{t_{0}},{\cal W}(w_{1},{\cal F}_{1}^{t_{1}}),\ldots,{{\cal W}}(w_{n-1},{{\cal F}}_{n-1}^{t_{n-1}}))).$		(7)

In the testing stage, only unsynchronized frames $(I_{0}^{t_{0}},I_{i}^{t_{i}})$ are available and the forward operations related to frame $I_{i}^{t_{0}}$ are removed from the network.

Two losses are used in the training stage. The first loss is a task-specific prediction loss $\ell_{p}$ between the scene-level prediction $V_{p}$ and the ground-truth $V_{gt}$. For example, for multi-view crowd counting $\ell_{p}$ is the mean-square error, and $V_{p},V_{gt}$ are the predicted and ground-truth scene-level density maps. The second loss is on the multi-view feature synchronization in the multi-view fusion stage. Since the synced frame pairs are available during training, the feature warping loss $\ell_{{\cal W}}$ encourages the warped features to be similar to the features of the original synced frame of view $i$,

	$\displaystyle\ell_{{\cal W}}(w_{i},{\cal F}_{i}^{t_{0}},{\cal F}_{i}^{t_{i}})$	$\displaystyle=\mathrm{mse}({\cal F}_{i}^{t_{0}},\hat{{\cal F}}_{i}^{t_{0}}),$		(8)
		$\displaystyle=\mathrm{mse}({\cal F}_{i}^{t_{0}},{\cal W}(w_{i},{\cal F}_{i}^{t_{i}})),$

where $\mathrm{mse}$ is the mean-square error loss. Note that the warping $\cal W$ only applies spatial shifting, and thus the minimization of the warping loss $\ell_{\cal W}$ in (8) will be based on the feature alignment via scene-level motion flow $w_{i}$ and not global feature value changes (e.g., color correction). Finally, the training loss combines the task loss and the warping loss summed over all non-reference views,

$\displaystyle\ell$

$\displaystyle=\ell_{p}(V_{p},V_{gt})+\gamma\sum_{i=1}^{n-1}\ell_{{\cal W}}(w_{i},{\cal F}_{i}^{t_{0}},{\cal F}_{i}^{t_{i}}),$

(9)

where $\gamma$ is a hyperparameter.

The view synchronization model is applied to each view separately. The camera view features $(F_{0}^{t_{0}},F_{i}^{t_{i}})$ from the unsynchronized reference view and view $i$ are first passed through a matching module (see below) and then fed into the motion flow estimation network ${\cal M}_{c}$ to predict the camera-view motion flow $w_{i}$ for view $i$. The warping transformation $W$ guided by $w_{i}$ then warps the camera-view features $F_{i}^{t_{i}}$ from view $i$ to be synchronized with the reference view at time $t_{0}$,

$\displaystyle\hat{F}_{i}^{t_{0}}=W(w_{i},F_{i}^{t_{i}}),i\in\{1,...,n-1\},$

(10)

where $\hat{F}_{i}^{t_{0}}$ is the warped camera-view features of view $i$ captured at time $t_{i}$, which is synchronized to reference view $0$ captured at time $t_{0}$. Finally, the reference and warped camera views are projected

$\displaystyle{\cal F}_{0}^{t_{0}}={\cal P}(F_{0}^{t_{0}}),\hat{{\cal F}}_{i}^{t_{0}}={\cal P}(\hat{F}_{i}^{t_{0}}),i\in\{1,...,n-1\},$

(11)

and then fused and decoded to obtain the scene-level prediction $V_{p}$:

	$\displaystyle V_{p}$	$\displaystyle=D(U({\cal F}_{0}^{t_{0}},\hat{{\cal F}}_{1}^{t_{0}},\ldots,\hat{{\cal F}}_{n-1}^{t_{0}}))$		(12)
		$\displaystyle=D(U({\cal P}(F_{0}^{t_{0}}),{\cal P}(\hat{F}_{1}^{t_{0}}),\ldots,{\cal P}(\hat{F}_{n-1}^{t_{0}}))).$		(13)

In the testing stage, only unsynchronized frames $(I_{0}^{t_{0}},I_{i}^{t_{i}})$ are available and the forward operations related to frame $I_{i}^{t_{0}}$ are removed from the network.

We propose 3 methods to match features to predict the view-level motion flow. The first method concatenates the features $(F_{0}^{t_{0}},F_{i}^{t_{i}})$ and then feeds them into the motion flow estimation network ${\cal M}_{c}$ to predict the motion flow $w_{i}$:

$\displaystyle w_{i}={\cal M}_{c}(\mathrm{Cat}(F_{0}^{t_{0}},F_{i}^{t_{i}})),i\in\{1,...,n-1\}.$

(14)

The second method builds a correlation map $C_{i}$ between features from each pair of spatial locations in $F_{0}^{t_{0}}$ and $F_{i}^{t_{i}}$,

$\displaystyle C_{i}((x,y),(x^{\prime},y^{\prime}))=F_{0}^{t_{0}}(x,y)^{T}F_{i}^{t_{i}}(x^{\prime},y^{\prime}),$

(15)

which is then fed into the motion flow estimation network ${\cal M}_{c}$ to predict the motion flow $w_{i}$:

$\displaystyle w_{i}={\cal M}_{c}(C_{i}),i\in\{1,...,n-1\}.$

(16)

The third method incorporates camera geometry information into the correlation map to suppress false matches. If both cameras are synchronized at $t_{0}$, then according the multi-view geometry, each spatial location in view $0$ must match a location in view $i$ on its corresponding epipolar line (Fig. 3a). Thus in the synchronized setting, detected matches that are not on the epipolar line can be rejected as false matches. For our unsynchronized setting, the matched location in view $i$ remains on the epipolar line only when its corresponding feature/object does not move between times $t_{0}$ and $t_{i}$. To handle the case where the feature moves, we assume that a matched feature in view $i$ moves according to a Gaussian motion model with standard deviation $\sigma$ (Fig. 3b). With the epipolar line and motion model, we then build a weighting mask, with high weights on locations with high probability of containing the matched features, and vice versa. Specifically, we set the mask $M_{i}((x,y),(x^{\prime},y^{\prime}))=1$ if $(x^{\prime},y^{\prime})$ is on the epipolar line induced by $(x,y)$, and 0 otherwise, and then convolve it with a 2D Gaussian with standard deviation $\sigma$ (Fig. 3c). We then apply the weight mask $M_{i}$ on the correlation map $\tilde{C}_{i}=M_{i}\odot C_{i}$, which will suppress false matches that are not consistent with the scene and motion model. Thus, the motion flow $w_{i}$ is

$\displaystyle w_{i}={\cal M}_{c}(\tilde{C}_{i})={\cal M}_{c}(M_{i}\odot C_{i}),i\in\{1,...,n-1\}.$

(17)

Multi-scale feature extractors are used in multi-camera tasks like crowd counting [9] or to refine the final prediction via multi-scale prediction fusion [51, 50]. Therefore, we next show how to incorporate multi-scale feature extractors with our camera-level synchronization model.¹¹1No extra steps are needed to incorporate multi-scale features with scene-level synchronization because the synchronization occurs after the feature projection. Instead of performing the view synchronization in each scale separately, the motion flow estimate of neighbor scales is fused to refine the current scale’s estimate (see Fig. 4). In particular, let there be $m$ scales in the multi-scale architecture and $j$ denotes one scale in the scale range $\{1,2,...,m\}$, with $m$ the largest scale. The multi-scale predicted motion flow are fused as follows.

• •

  When $j=1$ (the smallest scale), the correlation map $C^{(1)}_{i}$ of scale $1$ is fed into the motion flow estimation net to predict the motion flow $w^{(1)}_{i}$ for scale $1$.

• •

  For scales $j>1$, first the difference between the correlation map $C^{(j)}_{i}$ and the upsampled correlation map of the previous scale $\mathrm{up}(C^{(j-1)}_{i})$ is fed into the motion flow estimation net to predict the residual of the motion flow between two scales, denoted as $\tilde{w}^{(j)}_{i}$.

• •

  The refined motion flow of scale $j$ is

  $\displaystyle w^{(j)}_{i}=\mathrm{up}(w^{(j-1)}_{i})+\tilde{w}^{(j)}_{i}.$

  (18)

Similar to scene-level synchronization, a combination of two losses (scene-level prediction and feature synchronization) is used in the training stage. The scene-level prediction loss is the same as before. The feature synchronization loss encourages the warped camera-view features at each scale to match the features of the original synchronized frame,

	$\displaystyle\ell_{W}$	$\displaystyle=\mathrm{mse}(F_{i}^{t_{0},(j)},\hat{F}_{i}^{t_{0},(j)})$		(19)
		$\displaystyle=\mathrm{mse}(F_{i}^{t_{0},(j)},W(w_{i}^{(j)},F_{i}^{t_{i},(j)})).$		(20)

Similar to scene-level synchronization, the warping function $W$ only applies spatial shifting, and thus the minimization of $\ell_{W}$ in (20) will be based on feature alignment rather than feature value changes. Finally, the training loss is the combination of the prediction loss and the synchronization loss summed over all non-reference views and scales,

$\displaystyle\ell$

$\displaystyle=\ell_{p}(V_{p},V_{gt})+\gamma\sum_{i=1}^{n-1}\sum_{j=1}^{m}\ell_{W}(w_{i}^{(j)},{\cal F}_{i}^{t_{0},(j)},{\cal F}_{i}^{t_{i},(j)}),$

(21)

where $\gamma$ is a hyperparameter.

Two multi-view counting datasets used in [9], PETS2009 [56] and CityStreet [9], are selected and desynchronized for the experiments.

PETS2009 contains 3 views (camera 1, 2 and 3), and the first camera view is chosen as the reference view. The input image resolution ($w\!\times\!h$) is $384\!\times\!288$ and the ground-truth scene-level density map resolution is $152\!\times\!177$. There are 825 multi-view frames for training and 514 frames for testing. The frame rate of PETS2009 is 7 fps ($\Delta t=1/7s$). For constant frame latency, $\tau_{i}\in\{5s,-5s\}$ is used for cameras 2 and 3, and $\kappa_{i}=5s$ for random latency.

CityStreet proposed in [9] consists of 3 views (camera 1, 3 and 4), and camera 1 is chosen as the reference view. The input image resolution is $676\!\times\!380$ and the ground-truth density map resolution is $160\!\times\!192$. There are 500 multi-view frames, and the first 300 are used for training and the remaining 200 for testing. The frame rate of CityStreet is 1 fps ($\Delta t=1s$)²²2We obtained the higher fps version from the dataset authors.. For constant latency, $\tau_{i}\in\{3s,-3s\}$ for cameras 3 and 4, and $\kappa_{i}=3s$ for random latency.

Following [9], the mean absolute error (MAE) and normalized absolute error (NAE) of the predicted counts on the test set are used as the evaluation metric:

$\displaystyle\mathrm{MAE}=\tfrac{1}{N}\sum\nolimits_{i=1}^{N}|\hat{c}_{i}-c_{i}|,$

(23)

$\displaystyle\mathrm{NAE}=\tfrac{1}{N}\sum\nolimits_{i=1}^{N}|\hat{c}_{i}-c_{i}|/c_{i},$

(24)

where $c_{i}$ is the ground truth count and $\hat{c}_{i}$ is the predicted count, and $N$ is the number of testing images.

The experiment results using training with synchronized and unsynchronized frames are shown in Table III. The hyperparameter $\gamma=1$ is used for feature warping loss. On both datasets, our camera-level synchronization methods, CLS-cor and CLS-epi, perform better than other methods, including the baselines, demonstrating the efficicacy of our approach. Scene-level synchronization (SLS) performs worse than camera-level synchronization methods (CLS), due to the ambiguity of the projected features from multi-views. Furthermore after projection to the ground-plane, the crowd movement between frames $I^{t_{0}}_{i}$ and $I^{t_{i}}_{i}$ on the ground-plane is less salient due to the low resolution of the ground-plane feature map. CLS-cat performs worse among the CLS methods because simple concatenation of features cannot capture the image correspondence between different views to estimate the motion flow. Finally, the two baselines (BaseS and BaseSU) perform badly on CityStreet because of the larger scene with larger crowd movement between neighboring frames (due to lower frame rate).

The experiment results by training with only unsynchronized frames (which is a more practical real-world case) are shown in Table IV. Since the synchronized frames are not available, the MVMS model weights are trained from scratch using only unsynchronized data. Our models are trained with the similarity loss $\ell_{s}$ (with hyperparameter $\gamma=1000$), which encourages alignment of the projected multi-view features. Generally, without the synchronized frames in the training stage, the counting error increases for each method. Nonetheless, the proposed camera-level synchronization models CLS-cor and CLS-epi performs much better than the baseline BaseU. CLS-cor and CLS-epi trained on only unsynchronized data also performs better (on CityStreet) or on par with (on PETS2009) the baseline BaseSU, which uses both synchronized and unsynchronized training data. These two results demonstrate the efficacy of our synchronization model when only unsynchronized training data is available. Finally, the error for almost all synchronization models increases on both datasets when training without the similarity loss ($\ell_{p}$ in Table IV). This demonstrates the effectiveness of using $\ell_{s}$ to align the multi-view features in training.

In the previous experiments (training with only unsynchronized frames, see Sec. IV-C3), the ground-truth is corresponded (synchronized) to the frames of the reference view. We also perform experiments when the ground-truth scene-level density maps are calculated from the unsynchronized multi-view images. Specifically, we project the same person’s image coordinates of each unsynchronized view to the world plane and the average of the projection results is used as the ground-truth person location on the ground. Then, we use the obtained person location map to generate the scene-level density map.

The results for training with ground-truth from unsynchronized multi-view images and only unsynchronized frames can be seen in Table V. From the table, we can also find that the proposed method CLS-cor / CLS-epi can achieve better performance than other methods and CLS-epi achieves the best performance, and the performance can be further improved by adding similarity loss $\ell_{s}$.

We next present an ablation study on the multi-scale architecture for the multi-view counting in Table VI. Generally, the multi-scale architecture performs better than single-scale architecture models, and the proposed CLS-cor/CLS-epi can perform better than SLS or CLS-cat under both single-scale or multi-scale architecture, and under both training paradigms (sync and unsynced, or only unsynced).

The feature warping module only applies spatial shifting on the features of the unsynced views, i.e., it does not change the values (e.g., color) of the unsynced features (see Eqs. 5 and 10). To demonstrate this, we calculate the average statistics (mean and variance) of the feature maps before and after feature warping of Views 2 and 3 of CityStreet, and present the results in Table VII. The statistics of the feature maps do not change much after performing feature warping, and thus the performance improvement of the feature warping module is not due to color correction (feature value changes).

We further perform an ablation study to show that image color correction by itself cannot solve the frame desychronization problem. On the CityStreet dataset, in the baseline model (MVMS [9]), we add a learnable “color correction” layer, comprising an extra 1$\times$1 convolution layer (32 channels) in the branches of the other camera views before the projection and fusion step. The results are denoted as “color correction” in Table VIII. The error for using “color correction” is worse than the proposed SLS, CLS-cor and CLS-epi. The reason is that the desychronization issue comes from the capture time difference between camera views, which is better solved by spatial shifting of features rather than color correction (changing feature values).

We present the model size (number of parameters) and running speed of the baseline methods and the proposed SLS, CLS-cat, CLS-cor and CLS-epi in Table IX. The input resolution for the correlation step of the camera-view synchronization module is $160\times 95$. All models are tested on the CityStreet dataset with a Nvidia 1080Ti GPU. The baseline methods (BaseS, BaseSU and BaseU) do not use view synchronization modules, so their model sizes are smaller and running speeds are faster. The proposed CLS-cor and CLS-epi methods have the correlation module, and thus have more parameters than SLS or CLS-cat. CLS-epi is slower than CLS-cor due to the extra multiplication step with the epipolar weights.

Example results are shown in Fig. 5. Generally, the proposed synchronization methods CLS-epi and CLS-cor can predict better quality density maps, such as in the red box regions in the figure, where comparison methods tend to over-count these regions due to the same person being counted multiple times in unsynchronized frames. Furthermore, we also observe that the predicted density map is with better quality when synchronized frames are available compared to training with only unsynchronized frames. Finally, the prediction results are improved if similarity loss is enforced when training with only unsynchronized frames, such as the methods CLS-epi and CLS-cor on PETS2009.

We use the Human3.6M [57] dataset, which consists of 3.6 million frames from 4 synchronized 50 Hz digital cameras along with the 3D pose annotations. We follow the preprocessing step³³3https://github.com/anibali/h36m-fetch. Accessed: Oct. 10, 2019. recommended in [57], and sample one of every 64 frames ($\Delta t=64/50$) for the testing set, and sample one of every 4 frames ($\Delta t=4/50$) as the training set. The first camera view is always used as the reference view (if the first camera view is missing, the second one is used). We test desynchronization via random frame latency, with $\kappa_{i}\in\{8/50,32/50,64/50\}$ seconds, and only use unsynchronized data for training. Following [2], Mean Per Point Position Error (MPJPE) and absolute position MPJPE are used as the metric for evaluation. In training, the single-view backbone uses the pretrained weights from the original 3D pose estimation model. Baseline methods BaseS, BaseSU and BaseU are compared with our proposed camera-view synchronization models CLS-cor and CLS-epi.

The experiments results are presented in Table X. The original 3D pose estimation method (BaseS, BaseSU and BaseU) cannot perform well under the unsynchronized test condition, especially under large latencies (e.g., 64/50s). Our camera-view synchronization methods performs better than the baseline methods, with the performance gap increasing as the latency increases. Using similarity loss $\ell_{s}$ improves the performance of our models, and adding epipolar-guided weights can suppress false matches and further reduces the error. The detailed performance for each pose type under different frame latency settings is shown in Table XI, Table XII and Table XIII. From the tables, we can find that the proposed methods can perform especially better on the poses with larger movement between unsynchronized frames, e.g., Walking, WalkingDogs and WalkingTogether.

The ablation study on hyperparameter $\gamma$ for the method CLS-epi for 3D pose estimation is presented in Table XIV. In general, $\gamma=0.01$ achieves better performance than other weights.

We present the model sizes and running speed comparisons of our proposed models and the baselines for 3D pose estimation in Table XV. The input resolution for the correlation step of the camera-view synchronization module is $48\times 48$. As the original synchronized 3D pose estimation model [2] is already very large, the running speed of the proposed models CLS-cor and CLS-epi is similar to the baseline methods BaseS/BaseSU/BaseU.

Visualization results of unsynchronized 3D pose estimation are presented in Figs. 6 and 7. In the figures, the first row shows the input unsynchronized multi-view frames, and the top labels indicate the unsynchronized frame latency. Rows 2-8 show the 2D key-joints projected from 3D poses of Ground-truth, BaseS, BaseSU, BaseU, CLS-cor ($\gamma=0$), CLS-cor and CLS-epi, respectively, where synchronized frames are displayed for better visualization effect. In Fig. 6, BaseU fails on the unsynchronized input, and CLS-epi achieves the best performance among all methods, especially the prediction of the arms in view 1. In Fig. 7, the CLS-epi also achieves the best performance among all comparison methods.