Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Inter-frame affinity. We first measure inter-frame affinities between two frames $t-\delta$ and $t$. The input backbone features from frames $t-\delta$ and $t$ are resized into the stride of 16 of the image resolution and then concatenated across each level of the feature pyramid. We represent these processed features as $\mathbf{F}_{t}$ and $\mathbf{F}_{t-\delta}$ for frames $t$ and $t-\delta$, respectively. By using these features, we compute the transition matrix to measure inter-frame feature affinity between each spatial location. The inter-frame affinity matrix $\mathbf{W}_{t-\delta\rightarrow t}\in\mathbb{R}^{HW\times HW}$ is computed as

where $\circ$ is a matrix multiplication operator. Each element of the inter-frame affinity matrix represents the affinity between corresponding two locations in $t$ and $t-\delta$ frames. We normalize the affinity matrix to make the sum of each row to be 1.

Attention estimation via object propagation. Similar to TVOS [7], we aim to propagate masks using the transition matrix. However, instead of propagating pixel-level segmentation masks, we use a binary object map, which serves as a loose estimate for the pixel-level mask. The input binary object map for frame $t-\delta$ is generated by marking all pixels within the instance-specific bounding box. In addition to the object map, we also generate a binary map for the background by inverting the object’s binary map to take into account the background information when computing inter-frame attentions. We vectorize both binary object and background maps for matrix multiplication. Using the transition matrix, the binary object map, $\mathbf{b}_{t-\delta}$, is propagated to the current frame $t$ by

Since we propagate both object and background maps, we apply softmax to find the regions of the object. We represent the softmax output of the propagated object map as $\hat{\mathbf{a}}_{t}$ and use it as an attention map. Unlike existing attention-based algorithms which allow the machine to prioritize important features, we explicitly supervise the attention module to learn where to focus by propagating temporal information. This allows us to maximize the temporal context.

Attention-based mask prediction. Our next aim is to predict a mask using the attention map, $\hat{\mathbf{a}}_{t}$, and the features from frame $t$, $\mathbf{F}_{t}$. We first apply the attention map to the features by conducting element-wise multiplication to each spatial location. We feed the attention-guided features to a mask prediction module, which consists of four convolution modules (a combination of a convolution layer and ReLU), one deconvolution module, and prediction module (a combination of a convolution layer and a sigmoid layer).

Loss functions. The propagation loss $L_{prop}$ consists of two terms, the mask propagation loss $L_{prop}^{m}$ and the attention loss $L_{prop}^{a}$. $L_{prop}^{m}$ is computed identically to the mask head in Mask R-CNN [4]. The attention loss, $L_{prop}^{a}$, is computed as follows:

where $y_{ij}$ is the value in the ground-truth attention map at location $(i,j)$, and $\tilde{y}_{ij}$ is the predicted attention value.

Training. We use Mask R-CNN [4] with ResNet-50 backbone pre-trained on COCO dataset. Our model is trained on 4 Tesla V100 GPUs. We use a batch size of 24, with 6 images on each GPU. We train our model for 12 epochs using SGD optimizer with a learning rate of 0.005, which is decayed by a factor of 10 at 8 and 11 epochs.

Inference. During inference, our framework is completely online and does not require any future frames. We store the output predictions for previous frames in memory and propagate instance masks to the current frames. Using mask propagation, we are able to alleviate the temporal instability problem. When a bounding box is missing from the current frame, we propagate an instance mask from the frame history to the current frame to segment the missing object instance. Therefore, we use mask propagation as an empty instance filling mechanism.