3D Fully Convolutional Neural Networks
With Intersection Over Union Loss
for crop mapping from multi-temporal satellite images

Multi-temporal remote sensing images are being generated at an unprecedented scale and rate from resources such as Sentinel-2 (5 days frequency), Landsat (16 days frequency), and PlanetScope (daily). In the light of this, there have been many scientific efforts towards converting huge quantities of multi-temporal remote sensing images into useful information. One of these scientific efforts is automatic crop mapping [1, 2, 3, 4], being an active research area in remote sensing.

A decisive factor towards the goal of crop classification from multi-temporal images is developing methods that can learn temporal relationships in image time series. Traditional approaches for temporal feature representation such as Multi layer Perceptron, Random Forest, Support Vector Machine, and Decision Tree [5, 6, 7, 8, 9] have been developed for single-date images and are not able to explicitly consider the sequential relationship of multi-temporal data.

Recently with the success of deep neural networks in learning high-level task-specific features, CNN and LSTM-based methods have drawn increasing attention and achieved promising results in the field of crop classification from multi-temporal images [3, 2, 10, 11, 1, 12]. While most deep learning-based methods for crop mapping use pixel-by-pixel approach, in this paper, we will design a Fully Convolutional Neural Network (FCN) and use it for crop mapping. FCNs have been widely used in semantic segmentation, salient object detection, as well as brain tumor segmentation [13, 14, 15, 16]. They are capable of generating the segmentation map of the whole input image at once and thus are more efficient computationally. In addition, the spatial relationship between adjacent pixels is taken into account by using FCNs in contrast to pixel-by-pixel approaches, which take individual pixels as input. To fit the need of crop mapping, i.e. learning the sequential relationship of multi-temporal remote sensing data, we use 3D convolution operators as the building blocks of this FCN. They allow both the spatial and the temporal features to be extracted simultaneously. This would be beneficial to crop mapping since both the spatial and temporal relationships in multi-temporal remote sensing data are important for accurate crop mapping.

To learn different crop types, most deep learning-based crop mapping methods use the cross-entropy loss [3, 11, 12, 1, 10, 17]. They achieved promising results, but we hypothesize that there is still room for improvement by using a loss function better suited for cop mapping than the cross-entropy loss. To guide the network to generate more accurate prediction for crop types, we propose to learn the crop types by increasing the overlap between the prediction map and ground reference mask directly rather than using the cross-entropy loss that only focuses on per-pixel prediction. To the best of our knowledge, this is the first attempt to use this loss function in crop mapping.

In summary, the main contribution of this paper is to learn to identify different crop types by increasing the overlap between the prediction map and ground reference mask for each crop type, which would result in a rethink of the loss functions used to train deep neural networks for crop mapping. In conjunction with this loss function, we design a 3D FCN to simultaneously take into account the spatial and the temporal relationships in multi-temporal remote sensing data.

In this section, we explain our designed 3D FCN and Intersection Over Union (IOU) loss function, which is used to train the network. This network (Fig. 1) is composed of an encoder-decoder network. It learns to generate the segmentation map of crop lands from the input images. One important component of our proposed FCN is the 3D convolutional operator that is more beneficial than 2D convolutional operator for multi-temporal crop mapping since it also extracts the temporal features in addition to the spatial features. In the 3D FCN architecture, the Encoder extracts features at four different levels, each of which has different recognition information from each other. At lower levels, the Encoder captures spatial and local information due to the small receptive field, whereas it captures semantic and global information at higher levels because of the large receptive field. To take advantage of the both high level global contexts and low level details, features of different levels are merged in the Decoder through concatenation as shown in Fig. 1. In conjunction with the 3D FCN, we propose to use Intersection Over Union (IOU) loss to guide the FCN to output accurate segmentation maps.

In contrast to most deep learning-based crop mapping methods that use cross-entropy loss to learn the crop types, we propose a better loss function to guide the network to learn each crop type more accurately. We propose to use a loss function that tries to increase the overlap between the prediction map and ground reference mask directly. This loss function is more suited to crop mapping than the cross-entropy loss that only focuses on per-pixel prediction. Therefore, to increase the overlap between the prediction map and ground reference, we maximize the Intersection Over Union (IOU) for each crop type by adopting the following loss function:

$$L_{IOU}=\sum_{k=1}^{C}(1-IOU_{k})$$

(1)

where C denotes the number of classes, i.e. number of crop types and $IOU_{k}$ is defined as:

$$IOU_{k}=\frac{1}{M}{\sum_{m=1}^{M}{\frac{\sum_{i=1}^{N}p^{k}_{i,m}\cdot g^{k}_{i,m}}{\sum_{i=1}^{N}[p^{k}_{i,m}+g^{k}_{i,m}-p^{k}_{i,m}\cdot g^{k}_{i,m}]}}}$$

(2)

where M, N, p, and g denote total number of examples, total number of pixels in each example, prediction map, and ground reference mask respectively.

In the Experimental Results section we will show that using this loss function for learning the crop types results in a boost in the performance compared to using the cross-entropy loss.

In this study, a 3D FCN with an IOU loss function has been successfully applied to map soybean and corn crops in the US corn belt. The experimental results show that the adopted IOU loss function, which maximizes the overlap between the prediction map and ground reference mask for each crop type, is able to increase the performance compared to using the widely used cross-entropy loss. Therefore, using the IOU loss function is a better choice to learn individual crop type. For future work, we plan to apply the method to the regions that include more crop types to see to what extent our method can improve the performance.

This work was supported by Microsoft under the AI for Earth Microsoft Azure Compute Grant.