Towards On-Board Panoptic Segmentation of Multispectral Satellite Images

The PASTIS dataset is composed of 2,433 multi-spectral image patches, each with 10 spectral bands and of size 128 $\times$ 128 pixels, at 10m/pixel resolution. To the best of our knowledge, this is the only publicly available multi-spectral satellite image dataset with panoptic annotation. This data has been captured using the Sentinel-2 platform and four Sentinel tiles have been used. The authors have generated a time-series dataset by stacking all the available observations for each patch between September 2018 and November 2019.

However, the temporal sampling of the data is irregular as the orbital schedule of Sentinel-2 means patches are observed a different numbers of times and at different intervals. Furthermore, data providers have not processed the images if there is more than 90% cloud coverage, which leads to irregular length time-series. The temporal length of the observations ranges from 33 to 61 samples per sequence.

The dataset authors have used the publicly available French Land Parcel Identification System to retrieve the extents and the content of each parcel when annotating the patches. Within the annotations, each pixel has a semantic label that represents the crop type or the background class. Non-agricultural land is classified as the background class. In total there are 18 crop types, the background class and a void class, resulting in 20 semantic classes. The void class denotes agricultural parcels that contain crops that are not classified. The authors have also proposed a 5-fold cross validation scheme that divides the 2,433 patches into 5 splits. When splitting the data, care was taken to ensure that each fold has patches from all 4 Sentinel tiles and each split contains comparable class distributions. Furthermore, cross-contamination of data caused by adjacent patches falling into different splits was avoided.

In [10] the authors have observed that even after filtering out predominantly cloudy acquisitions, there still exists partially or completely obstructed patches. Furthermore, they highlight the utility of time-series data for resolving those obstructions and boundary ambiguities due to clouds and other atmospheric noise. For instance, in Fig. 2 we visualise the image sequence of a randomly chosen patch. Within the frames it can be seen that there exists boundary ambiguities which are indistinguishable even for humans. The multiple temporal observations can be used to eradicate these ambiguities in ground-based processing, however, in an on-board setting an AI algorithm does not have access to such historical data. In particular, when benchmarking the algorithms presented in [10] in a non-temporal (single frame) setting we illustrate how state-of-the-art methods struggle to generate accurate predictions (See Sec. IV-B). Hence we propose an on-line knowledge distillation strategy to augment a light-weight model that observes only a randomly chosen single frame of the time-series, allowing it to imagine the corresponding features from the time-series and achieve more accurate panoptic segmentation.

PASTIS-R is an extension of the PASTIS dataset where the authors have combined the Sentinel-2 multi-spectral patches with the corresponding Sentinel-1 observations. Specifically, they have used Sentinel-1 data in ground range detected format and assembled these into 3-channel images (vertical polarization, horizontal polarization and the ratio of vertical over horizontal polarization). Separate observations were captured in both ascending and descending orbits, generating two 3-channel images for each observation. This multi-modal dataset has rich spectral information coming from the Sentinel-2 multi-spectral observations, while the C-band radar of Sentinel-1 captures useful geometric information as it’s immune to cloud cover [29].

Hence, this multi-modal dataset can be seen as containing complementary information to be leveraged when views are obstructed due to cloud and other illumination conditions, however, it is illogical to directly fuse the data in an on-board setting as data is captured from separate satellites. Therefore, we propose a knowledge distillation strategy where a heavy-weight multi-modal teacher guides a uni-modal single frame student such that it can imagine the unseen features which are only visible to the teacher.

In this section we illustrate how the proposed cross-modality attention backbone is formulated. The approach takes inspiration from the U-TAE backbone from [10], though we incorporate specific augmentations to increase its capabilities in a multi-modal setting. The spatio-temporal encoding structure is based on [10], however, prior to the decoding phase we employ cross-modality attention to extract task specific, salient information that compliments each modality.

The inputs to the teacher network, $\dot{X}$ and $\ddot{X}$, are spatio-temporal inputs with shapes $T\times\dot{C}\times W\times H$ and $T\times\ddot{C}\times W\times H$, respectively, where $T$ denotes the length of the time-series (i.e number of frames), $W$ indicates the width of the image and $H$ denotes its height. $\dot{C}$ and $\ddot{C}$ represents the number of channels in the multi-spectral and radar modalities, respectively. Each spectral encoder, $\dot{f}_{e}^{l}$ and $\ddot{f}_{e}^{l}$ ($l\in[1,\ldots,L]$), has $L$ levels where each level is comprised of convolutions, Relu and normalisation layers. The encoder $f_{e}^{l}$ ($f_{e}^{l}\in[\dot{f}_{e}^{l}$, $\ddot{f}_{e}^{l}]$) at level $l$ receives the output $\dot{e}^{l-1}_{t}$ ($t\in[1,\ldots,T]$) of the immediately proceeding layer and generates a feature map $\dot{e}^{l}_{t}$. At each level the encoder maps are compressed using strided convolutions such that the output height $H^{l}=\frac{H}{2^{l-1}}$, and width $W^{l}=\frac{W}{2^{l-1}}$. All frames in the sequence are simultaneously processed by the encoders.

To perform temporal encoding, similar to [10], we use a Light-weight Temporal Attention Encoder (L-TAE) [30] which is a multi-head self-attention network with $G$ heads. The feature maps $\dot{e}^{l}$ and $\ddot{e}^{l}$ of the last level (i.e $L$) are used and the L-TAE networks, $\dot{f}_{L-TAE}$ and $\ddot{f}_{L-TAE}$ generate $G$ attention vectors, each of shape $T\times W_{L}\times H_{L}$ as,

To apply these attention maps to all encoded maps from the U-Net, interpolation is performed, where each attention vector, $\dot{a}^{l,g}$ and $\ddot{a}^{l,g}$ in $g$ is interpolated to size $H_{l}$ and $W_{l}$ for level $l\in[1,\ldots,L]$. The the respective channels $\dot{C}$ and $\ddot{C}$ of the multi-spectral and radar modalities are split into $G$ consecutive groups (i.e. $[\dot{e}^{l,1},\ldots,\dot{e}^{l,G}]$, each of shape $T\times\frac{\dot{C}_{l}}{G}\times W_{l}\times H_{l}$; and $[\ddot{e}^{l,1},\ldots,\ddot{e}^{l,G}]$, each of shape $T\times\frac{\ddot{C}_{l}}{G}\times W_{l}\times H_{l}$).

Then we multiply each embedding by it’s respective attention map, average the temporal sequence, and pass the resultant feature vector through a $1\times 1$ convolution layer,

where $[.]$ denotes concatenation along the channel dimension.

In [29], the authors have introduced several fusion strategies for the PASTIS-R dataset. Their evaluations using early, late, decision, and mid level fusion indicated that early fusion is the best strategy for panoptic segmentation. However, leveraging the generated attention vectors, $\dot{a}^{l,g}$ and $\ddot{a}^{l,g}$, we propose fusion of the multi-spectral and radar modalities using cross-modality attention. Specifically, we generate attention maps, $\dot{a}^{l}$ and $\ddot{a}^{l}$ for each modality such that,

These attention maps denote how salient features are spatially arranged in a particular modality. Hence, they are important for capturing complimentary information from a second mode. We interpolate these 2D maps to match the channel dimension of the complementary mode, and generate an augmented feature such that,

where $[,]$ denotes concatenation in the channel dimension. Then the augmented feature $\hat{h}^{l}$ is used by the spatial decoder.

Following the conventional U-Net network structure, the encoder map from level $l$, $\hat{h}^{l}$, is concatenated with the decoded feature map, $\hat{d}^{l-1}$, from level $l-1$. Each decoder block is composed of strided transposed convolutions to up sample the feature maps from the previous level. The decoder output is formally denoted as $\hat{d}=[\hat{d}^{1},\ldots,\hat{d}^{L}]$.

The student backbone receives the input, $\tilde{X}$, of shape $\dot{C}\times W\times H$, that corresponds to a single frame from the multi-spectral modality. Similar to the teacher backbone we pass it through a spatial encoder $\tilde{f}_{e}^{l}$ with $L$ levels and generate the feature map $e^{l}$. However, in contrast to the teacher backbone, $\tilde{e}^{l}$ is of dimension $C_{l}\times W_{l}\times H_{l}$. Despite the absence of the temporal dimension, the L-TAE is used in the student backbone with the motivation of capturing the channel-wise relationships with the aid of a multi-head attention formulation such that,

however, when applying the attention no temporal summarisation is performed. Specifically,

where $[.]$ denotes concatenation along the channel dimension. Then, similar to the teacher backbone, the spatial decoder of the student up samples these features to generate the multi-scale output features, $\tilde{d}=[\tilde{d}^{1},\ldots,\tilde{d}^{L}]$.

The panoptic classifier receives multi-scale feature maps (either from the teacher or student backbones) and produces predictions over parcels. We used the same architecture as the panoptic classifier network from [10], where it makes four predictions for each parcel: a center point, a bounding box, a binary instance mask and a class. We refer the readers to [10] for further details regarding this network’s architecture.

We use a single instance of this panoptic classifier to evaluate the predictions of both student and teacher backbones, allowing it to identify the similarities and correspondences between the feature vectors $\hat{d}$ and $\tilde{d}$. Note that in our formulation, the dimensions of the decoded maps from the student and teacher backbones at each level of the U-Net are identical.

As discussed in Sec. III-B3, the panoptic classifier makes four predictions, a centerness heat map, bounding boxes, instance masks and classes, and as such [10] used four losses to supervise these predictions.

Formally, let $\breve{m}\in[0,1]^{W\times H}$ denote the ground truth centerness heat map, let $m$ be the predicted centerness heat map, and let there be $P$ ground truth parcels. Then $L_{\mathrm{center}}$ is defined as,

where $\beta$ is a hyper-parameter and following [10] it is set to 4.

Then $P^{\prime}$ is defined as a set of detected parcels and $k^{\prime}(p)$ denotes the predicted class of the parcel $p\in P^{\prime}$, while $k^{\prime\prime}(p)$ indicates its ground truth class. Then the class loss of parcel $p$ is defined as,

Let $h^{\prime}(p)$ and $h^{\prime\prime}(p)$ denote the predicted and ground truth heights of the bounding boxes of parcel $p$, and $w^{\prime}(p)$ and $w^{\prime\prime}(p)$ indicate its respective width. Then,

defines the size loss. In addition a pixel-wise Binary Cross Entropy (BCE) loss is computed between the predicted shape, $l^{\prime}(p)$ and the corresponding ground truth instance $s^{\prime\prime}(p)$ mask when it is cropped using the predicted bounding box. This can be defined as,

Then the complete loss function in [10] can be written as,

To facilitate an on-line student-teacher knowledge distillation framework we perform the following augmentations. Let $f_{\mathrm{panoptic}}$ denote the output of the panoptic classifier introduced in Sec. III-B3 and let $L$ denote the complete loss defined in Eq. 10. Then,

where $[\hat{d}_{i}]^{\mathrm{detach}}$ denotes detaching the tensor from gradient computation. Then the final loss of our framework is defined as,

where $\lambda_{1}$ and $\lambda_{2}$ are hyper-parameters controlling the contributions of the respective loss terms.