Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types

The first step in the pipeline is to gather the latitude and longitude coordinates of candidate fields and their corresponding roadside image along all roads in the country. We started with Open Street Map (OSM), a worldwide open geographic database with high coverage and completeness (Haklay and Weber 2008), and used Overpass API to query all OSM ways within Thailand. In order to maximize recall of GSV field images, we generated equidistant points at 10m steps along OSM ways. Examples of points sampled from OSM street nodes in a chosen area are shown in Figure 4.

Next, we used existing land cover maps to filter for street points near farmland and remove field points that are not visible from the street due to obstructions (Figure 1). In particular, we used the European Space Agency’s WorldCover 10m v100, which classifies global land cover at 10 meter resolution for the year 2020. The map contains 11 land cover classes, including tree cover, shrubland, grassland, built-up, and cropland. We accessed WorldCover and filtered candidate points using functions available in Google Earth Engine.

Starting from the OSM-derived equidistant points, we removed points containing no cropland within a 10m radius. An inspection of 200 remaining points revealed that, in 30% of GSV images, crop fields were still blocked by trees and other vegetation next to the road. We therefore also removed points containing any tree cover within the same 10m radius.

Finally, we are only interested in roadside images taken during the growing season, when crop types may be visible from the road. For this work, only GSV images captured during the wet growing season in Thailand (May 2022–October 2022) were considered. The above approach can be modified to any region and growing season worldwide by changing the date range and shapefile of the region of interest.

Due to the grid layout of streets in the US Midwest, prior work using GSV for crop type mapping obtained roadside images facing crop fields and points within crop fields by extrapolating street points in north/south and east/west directions (Yan and Ryu 2021). Since street layouts are more complex in smallholder regions, we first found the street bearing $\theta$ using the haversine formula (see Section A.2), and then computed the heading for the camera to face the two fields on either side of the street as $\theta\pm 90$ degrees (Figure 4). We empirically determined that 30m was the best distance to extrapolate street points to crop field interior points (Table 6).

After filtering out points near trees and finding the appropriate camera heading, we found that 58% of GSV images still had an obstructed view of a field due to small bushes between the road and field not detected by satellite land cover maps (Table 3). We therefore labelled 2986 images into {field, not field} and trained a binary classifier using a ResNet-18 pre-trained on ImageNet. Hyperparameters used include an Adam optimizer and learning rate of $0.001$ for 15 epochs. The model classified field images from non-field images with 95% recall and 98% precision. Candidate street-view images were downloaded ($n=224,000$) and classified by the field/not-field CNN. Those labelled as field ($n=89,000$) were used as input in Section 2.3 to be classified into crop types. Although this work utilized 89,000 street-view images to generate ground references, there are approximately 800,000 GSV field images available in Thailand, offering substantial potential for the dataset to expand.

Trained models were used to classify street-view images ($n=89,000$) into crop types. We took sliding windows across each image, with a window of $300\times 300$ pixels and a stride of $50$. We used the mode of high probabilities (MHP) to select the predicted class from all the sliding windows in an image. For each window, if p is the vector of softmax probabilities and $c$ is the crop type with the highest probability, then the window casts one vote for class $c$ if $\textbf{p}_{c}$ exceeds some threshold $\tau$. If none of its probabilities exceed $\tau$, then the window casts no votes. The final prediction for the image is the crop type with the most votes across all windows.

Each clean street-view image was classified by the 5 models with sliding window and MHP and paired with its field coordinate to generate the ground reference training sets.

Consistent with existing state-of-the-art methods (Inglada et al. 2015; Jin et al. 2019), we used Sentinel-2 satellite time series for country-wide crop type classification. Sentinel-2 images capture 13 optical bands at 10-60m resolution on a 5-day cycle, and their time series contain information on crop phenology that allows different crop types to be distinguished.

We used Google Earth Engine to export Sentinel-2 L2A time series in Thailand from May 1 – October 31, 2022. Four spectral bands were used: Red Edge 4, SWIR 1, SWIR 2, and NIR. We added the green chlorophyll vegetation index ($\text{GCVI}=\text{NIR}/\text{GREEN}-1$), as prior work showed it to be a valuable feature for crop type classification. We used the Cloud Probability band to remove cloudy days before extracting time-series for each ground reference.

We used the harmonic regression to extract frequency-domain features from time series of varying lengths for input to machine learning (Figure 9). Equivalent to the discrete Fourier Transform, harmonic regression generates robust features for crop type classification (Shumway, Stoffer, and Stoffer 2000; Jin et al. 2019; Ghazaryan et al. 2018; Jakubauskas et al. 2001; Wang, Azzari, and Lobell 2019; Wang et al. 2020). We applied a 3^rd order harmonic regression to each band independently to arrive at a total of 35 features.

We trained random forests (with $500$ trees) on crop type classification, with the harmonic coefficients as input and the crop type labels generated by the street-view CNN as output. Random forests (Breiman 2001) are frequently used in remote sensing applications for their high accuracy and computational efficiency (Gislason, Benediktsson, and Sveinsson 2006; Azzari and Lobell 2017; Ok, Akar, and Gungor 2012).

The crop type ground references ($n=81,000$, Section 2.3) were used to train the remote sensing crop type classifier. We trained 5 random forests, one for each ground reference set (Table 4). A sixth model was trained only on the expert-labeled dataset ($n=1,153$). Compared to automatically-labeled points, expert-labeled points are more accurate but almost 100 times smaller in quantity.

Our experiments on the 6 datasets showed that weak supervision with online images can successfully classify crop type in street-view images (Table 4). WebCC ($n=3,266$) on its own achieved 82% overall accuracy, similar to iNaturalist ($n=9,646$) at 81%. For comparison, a baseline that classifies all samples as the most common class (rice) would achieve 67% accuracy. Maize had the lowest F1 score under both WebCC and iNaturalist (62% and 68%, respectively), followed by sugarcane (62% and 76%). The low performance is likely due to the more visual similarities between the two crops at the early-growth stage as well as their similar height. Meanwhile, rice has short and bright green leaves and cassava has a palmate leaf structure and is grown more separately, both more visually distinctive from the street.

When merged together, iNaturalist + WebCC ($n=12,912$) overall accuracy improved to 85%. F1 scores for cassava, maize, sugarcane and other also substantially improved. This complementarity could be due to iNaturalist having higher label accuracy but more out-of-domain images (e.g. closeups), while WebCC has lower label accuracy but considerably more street-view-like images.

GPT-4V demonstrated consistently high zero-shot classification performance in all categories, with the highest accuracy (95%). Upon inspecting its responses, it showed a clear understanding of the distinctive phenotypes of the plants to support its decision.

The CNN trained on the expert-labeled dataset ($n=1,153$) achieved 93% accuracy, nearly identical to the Combined model ($n=12,339,93\%$ accuracy). The comparable performance of the CNN trained on GPT-4V labels ($n=1,153)$ shows that today’s multimodal LLMs can already help minimize the need for expert labeling.

The MHP approach to remove low-confidence classifications (Section 2.3) proved to increase the accuracy and F1 score across all models for thresholds set to 0.7 or above. The top performing model trained on the Expert labeled dataset achieved 82% accuracy when directly classifying the whole image, 90% with sliding windows but no MHP threshold, and 93% with a 0.90 MHP threshold. Improvements were similar for the other training sets.

We note that a higher MHP threshold $\tau$ leads to some images being dropped from the inference set, because sometimes no sliding windows have softmax probabilities exceeding $\tau$. Despite the slight loss of data, we observed that F1 score increased monotonically with $\tau$ all the way up to $\tau=0.95$.

We found that training a random forest on large CNN-generated GSV ground references ($n=81,000$) resulted in crop type maps that were significantly more accurate than training on small expert-labeled ground references ($n=984$), despite the CNN-based labels being imperfect (Table 5). Training on expert-labeled ground references achieved 69% overall accuracy, which is modest given the 65% accuracy of a baseline model that classifies everything as the most common class (rice). In comparison, the random forest trained on 81,000 GSV samples whose crop types were predicted by the street-view CNN trained on expert labels achieved an accuracy of 93%. This suggests that the sample volume generated by leveraging GSV outweighs the noise that predicted crop type labels can carry from misclassification.

Among the 6 automatically-generated GSV ground reference datasets, the datasets generated by CNNs trained on expert labels or GPT-4V labels proved more accurate than those generated by CNNs trained on weak labels from the web. Consistent with the street-view classification results in Table 4, the lowest-performing datasets were those created by the WebCC CNN and the iNaturalist CNN; the remote sensing-based crop type classifier trained using their predictions as labels achieved accuracies of 80% and 78%, respectively. The crop type classifer trained on iNaturalist + WebCC CNN predictions performed better at 82% accuracy, but suffered from low cassava and maize F1 scores (68% and 69%). The three highest-performing crop type classifiers were trained on outputs of the GPT-4V, expert-labeled, and Combined CNNs, achieving accuracies of 92%, 93%, and 92% and higher F1 scores across all five crop classes.

For the individual crop types, rice—the most abundant crop—was classified most accurately, with F1 scores of 95% for the top 3 models. Sugarcane was also classified accurately, with F1 scores ranging from 90% to 91% for the top 3 models. Maize and cassava were consistently the most difficult crop types to classify across all models.

The weakly-supervised CNNs struggled to classify the “Other” class, but the GPT-4V and expert-labeled models performed better. We observed in experiments that including the “Other” class enhanced the performance of the four main crop type classes.

Lastly, we investigated the relationship between the number of ground references and the remote sensing-based crop type classification performance. We found that, even at 81,000 ground references, performance had not yet saturated as a function of training set size; the overall F1 score continued to increase linearly as more street-view points were added (Figure 7).