End-to-End Segmentation via Patch-wise Polygons Prediction

Accurate segmentation in dense environments is a challenging computer vision task. Many applications, such as autonomous driving [33], drone navigation [25], and human-robot interaction [24], rely on segmentation for performing scene understanding. Therefore, the accuracy of the segmentation method should be high as possible.

In recent years, fully convolutional networks based on encoder-decoder architectures [21], in which the encoder is pre-trained on ImageNet, have become the standard tool in the field. Several techniques were developed to better leverage the capacity of the architectures. For example, skip connections between the encoder and the decoder were added to overcome the loss of spatial information and to better propagate the training signal through the network [27]. What all these methods have in common is the representation of the output segmentation map as a binary multi-channel image, where the number of channels is the number of classes $C$. In our work, we add a second type of output representation, in which, for every patch in the image and for every class, a polygon comprised of $k$ points represents the binary mask.

Alternative image segmentation architectures are presented in Fig. 1. The most common is the encoder-decoder one. The active contour methods, similar to our work, employ a polygon representation. Such methods, however, are iterative, and produce a single polygon, lacking the flexibility to cope with occlusions. Transformer based network employ patch-wise self-attention maps to calculate a spatial representation that attends globally to multiple parts of the image. The output mask is then generated by an MLP decoder. Our method differs from these approaches and produces multiple polygons, which are rendered by a neural renderer to produce the final segmentation mask.

For optimization purposes, we use a neural renderer to translate the polygons to a binary raster-graphics mask. The neural renderer provides a gradient for the polygon vertices and enables us to maximize the overlap between the output masks and the ground truth. Since there are polygons for every patch and every class, whether the object appears there or not, we multiply the binary mask with a low-resolution binary map that indicates the existence of each object.

The method achieves state-of-the-art results on four segmentation benchmarks, Cityscapes, CUB-200-2011, Nucleus segmentation challenges (MoNuSeg) and on the Vaihingen building segmentation benchmark.

Most modern approaches to image segmentation perform image-to-image mapping with a Fully Convolutional Network [21]. Many encoder-decoder architectures were proposed, e.g., [12, 32, 30, 14], most of which employ U-Net based skip connections between parallel blocks of the encoder and the decoder [27]. Improved generalization is obtained by reducing the gap between the encoder and decoder semantic maps [41]. Another contribution adds residual connections for each encoder and decoder block and divides the image into patches with a calculated weighted map for each patch as input to the model. [37]. Wang et al. add attention maps to each feature map in the encoder-decoder block in order to enlarge the representation between far areas in the image with good computational efficiency [36]. This is extended by adding scalar gates to the attention layers [34]. The same contribution also adds a Local-Global training strategy (LoGo), which further improves performance. Dilated convolutions [38] are used by many methods to preserve the spatial size of the feature map and to enlarge the receptive field [5, 40, 39]. The receptive field can also be enlarged by using larger kernels [26] and enriched with contextual information, by using kernels of multiple scales [6]. Hardnet [4] employs a Harmonic Densely Connected Network that is shown to be highly effective in many real-time tasks, such as classification and object detection.

A recent contribution [19], refines the output instance mask of pretrained Mask-RCNN algorithm by learning to correct the coordinates of the extracted polygons. This algorithm obtains good performances for the cityscapes instance segmentation benchmark but uses different settings than the rest of the literature. For example, the entire image is used instead of a specified bounding box, and the bounding box expansion differs from what is conventionally used in published work. Since no code was published, we cannot perform a direct comparison. From the algorithmic perspective, the methods greatly differ in the type of polygons used (boundary vs. local), the type of optimization (Chamfer Distance loss [13] on polygon coordinates vs. using a neural renderer), and almost every other aspect.

The input to our method is an RGB image $I$, of dimensions ${3\times H\times W}$. The image is divided into patches of size $s\times s$ and a polygon with $k$ vertices is predicted for each patch. In addition, each patch has an associate scalar that determines the probability of the object being present in the patch.

This mask representation is illustrated in Fig. 2. The low-res probability map in panel (a) assigns one value per grid cell, where we denote higher probabilities by a brighter value. The local polygons in panel (b) determine a local segmentation mask in each grid cell. When the grid cell is completely enclosed by the segmented object, the polygon covers the entire cell. At object boundaries, it takes the form of the local boundary. When out of the object, it can degenerate to a zero area polygon or an arbitrary one. The final segmentation map, depicted in Fig. 2(c) is obtained by multiplying the low-resolution probability map with the local patch rastered for each polygon.

The learned network we employ is given as $f=f_{2}\circ f_{1}$, where $f_{1}:\mathbb{R}^{C\times{H}\times{W}}\rightarrow\mathbb{R}^{C\times\frac{H}{32}\times\frac{W}{32}}$ is the backbone (encoder) network, and $f_{2}$ is the decoder networks. The output space of $f_{2}$ is a tensor that can be viewed as a field with $\frac{H}{s}\times\frac{W}{s}$ vectors $\mathbb{R}^{2k+1}$, each representing a single polygon that is associated with a single image patch. This representation is the concatenation of the 2D coordinates of the $k$ vertices together with the probability. The image coordinates are normalized, such that their range is [-1,1].

A fixed triangulation $T$ is obtained by performing Delaunay triangulation to a regular polygon of $k$ vertices. It is used to convert all polygons to a raster image, assuming that locally the polygons are almost always close to being convex. We set the z-axis to be constant and the perspective to be zero, in order to use a 3D renderer for 2D objects.

The output $f(I)$ is divided into two parts, see Fig. 3. We denote by $P=\{P^{i}\}$ the set of all $\frac{H}{s}\times\frac{W}{s}$ polygons, and by $M$ the map of size $\frac{H}{s}\times\frac{W}{s}$ that contains all the probabilities. The neural renderer $r$, given the vertices $P$ and the fixed triangulation $T$, returns the polygon shape as a local image patch. In this raster image, all pixels inside the polygon defined by $P$ through the triangulation $T$ are assigned a value of one, zero for pixels outside, and in-between values for pixels that the polygon divides:

$${M^{i}_{p}}=r(P^{i},T)\in\mathbb{R}^{s\times s}\,,$$

(1)

The aggregation layer collecting all local patches $M^{i}_{p}$, each in its associated patch location, we obtain the map $R_{p}\in\mathbb{R}^{H\times W}$. By performing nearest neighbor upsampling to the probability map $M$ by a factor of $s$ in each dimension, and obtain a probability maps of the same size as $R_{p}$. The final mask is the pixel-wise product between these two maps.

$${M}_{o}={\uparrow}M\odot{R_{p}}\,,$$

(2)

where $\odot$ denotes elementwise multiplication and $\uparrow$ denotes upsampling. This way, the low-resolution mask serves as a gating mechanism that determines whether the polygon exists within an object of a certain class in the output segmentation map.

To maximize the overlap between the output mask (before and after incorporating the polygons) and the ground truth segmentation map $Y$, a soft dice loss [31]:

$$\mathcal{L}_{dice}(y,\hat{y})=1-\frac{2TP(y,\hat{y})+1}{2TP(y,\hat{y})+FN(y,\hat{y})+FP(y,\hat{y})+1}$$

(3)

Where TP is the true positive between the ground truth $y$ and output mask $\hat{y}$, FN is a false negative and FP is a false positive. In addition, a binary cross-entropy is used for $M$:

$$\mathcal{L}_{BCE}(y,\hat{y})=-y\log(\hat{y})+(1-y)\log(1-\hat{y})$$

(4)

$$\mathcal{L}=\mathcal{L}_{BCE}({\uparrow}M,Y)+\mathcal{L}_{dice}(M_{o},Y)$$

(5)

This loss appears without weighting, in order to avoid an additional parameter. During inference, we employ only $M_{o}$ and the second term can be seen as auxiliary.

We present our results on three segmentation benchmarks and compare our performance with the state-of-the-art methods. We further perform a study on the effect of each parameter of the proposed algorithm, and on the effect of the underlying network architecture. We put an added emphasis on the subset of test images that depict an occlusion of the segmented object.

The Citycapes dataset [9] for instance segmentation includes eight object classes with $5,000$ fine-annotated images, divided into $2,975$ images for training, $500$ for validation, and $1,525$ for the test set.

The Cityscapes dataset contains urban scenes that are characterized by dense and multiple objects. In order to fully understand the scene, the segmentation method is required to be robust to occlusions of objects, which causes the segmentation task to be more challenging. Following [20], we train and evaluate given the bounding box of the object, such that the network outputs a single mask for the entire instance.

To analyze the images with occlusions, the cases where the ground truth contains at least two contours are identified. for this purpose, we define an occlusion distance $d$ per image as:

$$d=\max_{C_{i},C_{j}\in C}\min_{p\in C_{i},q\in C_{j}}\|p-q\|\,,$$

(6)

where $C$ is the set of all contours of a specific class defined for a specific image, each contour being a set of points. The distance $d$, which degenerates to zero if there is only one contour, is then used to stratify the test images by the number of occlusions. Tab. 1 contains the number of images in both the validation set of the entire Cityscapes dataset, as well as for the subset of which the occlusion distance $d$ is larger than one. As can be seen, occlusions are fairly common for many of the objects.

The baseline methods we compare to on the Cityscapes dataset are PSPDeepLab[6], Polygon-RNN++[1], Curve-GCN[20] and Deep active contours [10]. Another baseline, denoted as “FCN-HarDNet-85” employs a fully-convolutional network with the same backbone [4] we used for our own method.

We note that our method employs 17M parameters in its decoder, while the “FCN-HarDNet-85” employs a decoder with 22M parameters.

The Vaihingen [28] dataset consists of 168 aerial images, at a resolution of $256\times 256$, from this German city. The task is to segment the central building in a very dense and challenging environment, which includes other structures, streets, trees, and cars. The dataset is divided into 100 buildings for training, and the remaining 68 for testing.

For building segmentation, we use an encoder-decoder architecture based on the HarDNet-85 [4] backbone and produce $M$ at 1/8 resolution ($s=8$).

The baseline methods we compare to on the Vaihingen buildings dataset are “DSAC”[23], “DarNet”[7], “TDAC”[11] and Deep active contours [10]. We also present results for “FCN-UNET” [27], “FCN-ResNet-34” and “FCN-HarDNet-85”, all based on fully-convolutional network decoders.

The MoNuSeg dataset [18] contains a training set with 30 microscopic images from seven organs with annotations of 21,623 individual nuclei, the test dataset contains 14 similar images. Following previous work, we resized the images into a resolution of $512\times 512$ [34]. An encoder-decoder architecture based on the HarDNet-85 [4] backbone. Since the recovered elements are very small, our method employs a map $M$ at 1/4 resolution ($s=4$), and not lower resolutions as in other datasets.

The baseline methods we compare to on the Monu dataset are “FCN”[2], “UNET”[27], “UNET++”[41], Res-Unet[37], Axial attention Unet[36] and Medical transformer [34].

Finally, we report results on CUB-200-2011 [35]. The dataset contains 5994 images of different birds for training, and 5794 for validation. We trained the model for the same setting of cityscapes where the input image size is $256^{2}$ and the evaluation is on the original image size.

Following [10], we used for the Cityscapes images a set of augmentations that includes: (i) scaling by a random scale factor in the range of $[0.75,1.25]$ and then cropping an image of size $256\times 256$, (ii) color jitter with the parameters of brightness sampled uniformly between $[0,0.4]$, a contrast in the range $[0,0.4]$, saturation in the range $[0,0.4]$, and hue in the range $[0,0.1]$, (iii) a random horizontal flip with a probability of 0.5.

For the Vaihingen dataset, also following [10], we used (i) a random rotation augmentation of 360 degrees and a scale range of $[0.75,1.5]$, (ii) a random vertical and horizontal flip with a probability of 0.5, (iii) random color jitter with a maximal value of 0.6 for brightness, 0.5 for contrast, 0.4 for saturation, and 0.025 for hue.

For the MoNu dataset, we used (i) a random rotation augmentation of ±20 degrees and a scale range of $[0.75,1.25]$, (ii) a random horizontal flip with a probability of 0.5, (iii) random color jitter with a maximal value of 0.4 for brightness, 0.4 for contrast, 0.4 for saturation, and 0.1 for hue.

Fig. 4 presents sample results and compares our method to the leading active contour method (“Deep contour”) and to the FCN-HarDNet-85 baseline. As can be seen, the active contour method struggles to model objects that deviate considerably from their convex hull, and cannot handle occlusions well. In comparison to the FCN baseline, our method provides more accurate details (consider the bicycle fork in the first row), but the overall shape is the same.

The results for images stratified by the level of occlusion (the measurement $d$) are presented in Fig. 5. All methods present a decrease in performance as the amount of occlusion increases. The FCN-HarDNet-85 maintains a stable performance level below our method, and the active contour method suffers the most from occlusion, as expected.

The results for the Vaihingen building dataset are presented in Tab. 3. Our method outperforms both the fully connected segmentation methods, as well as the active-contour based methods. Fig. 6 presents typical samples, demonstrating the refinement that is obtained by the polygons over the initial segmentation mask.

The results for MoNu dataset are reported in Tab. 4. We outperform all baselines for both the Dice score and mean-IoU. Our algorithm also obtains better performances in comparison to the fully convolutional segmentation network with the same backbone Hardnet-85. The improvement from low resolution mask ($M$) to the output mask ($M_{o}$) is from 49.13% IoU to 66.82%. The full resolution FCN-Hardnet85 obtains a lower result of 66.06 IoU. Fig. 7 presents the output mask of our algorithm for samples from the test set.

The results for CUB-200-2011 are reported in Tab. 5. We obtain better results than both FCN-Hardnet85 and Deep Contour on all metrics. Fig. 8 present the output mask of our algorithm for samples from CUB-200-2011.

Fig. 9 shows our network performances for building segmentation, when varying the two parameters. As can be seen, for small patches, for instance, $4\times 4$ or $8\times 8$, the number of nodes that optimizes performance is five. As may be expected, for larger patches, the method performs better with a higher number of vertices.

We also checked our performances for different Hardnet backbones - HarDNnet39DS, HarDNnet68DS, HarDNnet68, HarDNnet85 (DS stands for for depth-wise-separable, see [4]). The results appear in Tab. 6 for the building dataset. As expected, the performance improves with the increase in the number of parameters in the network. It can also be observed that the low- resolution mask after upsampling $M$ and the refined mask $M_{o}$ have a performance gap of 4% in terms of mean IoU, in favor of the latter, across all of the backbones.

The ability to work with polygons allows our network to produce segmentation maps at a resolution that is limited only by machine precision. This can be seen in Fig. 2(b), where the polygons have fractional coordinates. These polygons can be rasterized at any resolution.

The property of practically-infinite resolution may be of use in applications that require an adaptive resolution, depending on the object or its specific regions. An example is foreground segmentation for background replacement (virtual “green screens”), in which hair regions require finer resolutions. Such scenarios would raise research questions about training networks at a resolution that is higher than the resolution of the ground truth data, and on the proper evaluation of the obtained results.

We note that while we do not enforce the polygons of nearby patches to be compatible at the edges, this happens naturally, as can be seen in Fig. 2(b). Earlier during development, we have attempted to use loss terms that encourage such compatibility but did not observe an improvement.

Further inspection of the obtained polygons reveals that polygons with $k>4$ often employ overlapping vertices in order to match the square cell edges. In empty regions, polygons tend to become of zero-area by sticking to one of the boundaries. These behaviors emerge without being directly enforced. In some cases, phantom objects are detected by single polygons, as in the top right part of Fig. 2(b). These polygons are removed by the gating process that multiplies with $M$, as can be seen in Fig. 2(c). We do not observe such polygons in the output images.

The pixel grid representation is commonly used by deep segmentation networks. In this work, we present an alternative approach that encodes the local segmentation maps as polygons. The method employs a neural renderer to backprop the rasterization process. A direct comparison to a method that employs the same backbone but without the polygonal representation reveals a significant improvement in performance on multiple benchmarks, despite using fewer training parameters. Moreover, our method also outperforms all other segmentation methods.

This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation programme (grant ERC CoG 725974).