AutoLTS: Automating Cycling Stress Assessment via Contrastive Learning and Spatial Post-processing

Training and testing our model requires three datasets: i) road network topology, ii) ground-truth LTS labels for all road segments, and iii) street-view images that clearly present the road segments. We collect all the data in Toronto, Canada via a collaboration with the City of Toronto. Data sources and pre-processing steps are summarized below.

Road network topology: We retrieve the centerline road network from City of Toronto (2020). Geospatial coordinates of both ends of each road segment are presented. We exclude roads where cycling is legally prohibited, e.g., expressways. The final network has 59,554 road segments.

LTS label. The LTS calculation requires detailed road network data. For each road segment in Toronto, we collect road features as summarized in Table 1 and calculate its LTS label following Furth, Mekuria, and Nixon (2016) and Imani, Miller, and Saxe (2019) (detailed in Appendix B).

Street-view image. We collect street-view images using the Google StreetView API. We opt not to collect images for road segments that are shorter than 50 meters because a significant portion of those images typically present adjacent road segments that may have different LTS labels. For each of the remaining road segments, we collect one image using the geospatial coordinate of its mid-point. We manually examine the collected images to ensure that they clearly present the associated road segments. If an image fails the human screening, we manually recollect the image when possible. Images are missing for roads where driving is prohibited, such as trails and narrow local passageways.

Our final image dataset consists of 39,153 high-quality street-view images, with 49.0%, 34.5%, 6.9%, and 9.7% of them labeled as LTS 1, 2, 3 and 4, respectively.

We propose a contrastive learning approach to train the image encoder. The novelty lies in the development of a new contrastive loss that considers the ordinal relationship among LTS labels. We adopt a contrastive learning framework (Figure 3) similar to MoCo (He et al. 2020) to train the image encoder $f$ on a pretext task where the encoder learns to pull together “similar” images in the embedding space. Given a batch of $n$ road segments indexed by $\mathcal{N}$, let $\mathbf{x}_{i}$ and $y_{i}$ denote the street view image and the label of segment $i\in\mathcal{N}$, respectively. We assume $y_{i}\in[m]$ are discrete and ordered for all $i\in\mathcal{N}$. We create $l$ virtual labels $(y_{i}^{1},y_{i}^{2},\ldots,y_{i}^{l})$ for each image $\mathbf{x}_{i}$ where $y_{i}^{u}=\lceil y_{i}/u\rceil$ for all $u\in[l]$. In words, these virtual labels are created by grouping the “neighboring” real labels at different granularities. Consequently, images with “similar” real labels have more overlapping virtual labels. We create two views $\bar{\mathbf{x}}_{i}$ and $\tilde{\mathbf{x}}_{i}$ of each image $\mathbf{x}_{i}$ by applying a random augmentation module twice. We create a momentum encoder $g$ that has the same structure as $f$ and whose parameters are updated using the momentum update function (He et al. 2020) as we train $f$. The image views $\{\bar{\mathbf{x}}_{i}\}_{i\in[n]}$ and $\{\tilde{\mathbf{x}}_{k}\}_{k\in\mathcal{K}}$ are encoded by $f$ and $g$, respectively, where $\mathcal{K}$ is a fixed-length queue that stores previously generated image views. Let $\bar{\mathbf{z}}_{i}=f(\bar{\mathbf{x}}_{i})$ and $\tilde{\mathbf{z}}_{i}=g(\tilde{\mathbf{x}}_{i})$ denote the embedding generated by these two encoders. During training, these embeddings are further fed into a projection layer, which is discarded during inference following Khosla et al. (2020) and Chen et al. (2020). The encoder network $f$ is trained to minimize the following loss that applies to the projected embeddings:

where $\mathcal{K}^{u}_{i}=\{k\in\mathcal{K}:y^{u}_{i}=y^{u}_{k}\}$ for all $u\in[l]$, $w^{u}$ is a constant weight assigned to the $u^{\textrm{th}}$ virtual label, $\tau$ is a temperature hyper-parameter, and p is the projection function.

Comparison to other loss functions. Compared to MoCo (He et al. 2020), our OrdCon takes advantage of label information. Consequently, as illustrated in Figure 3, our image embeddings form clusters that correspond to their image labels. Compared to the SupCon (Khosla et al. 2020), our OrdCon considers the ordinal relationship among image labels by aggregating the real label at different granularities. As a result, the relative positions of our embedding clusters reflect the similarity between their corresponding labels. OrdCon recovers the SupCon when $l=1$ and $w^{1}=1$.

Several LTS-related road features, e.g., motor traffic speed, have strong spatial correlations, meaning that the values associated with adjacent road segments are highly correlated. Such structure can be useful in regulating road feature predictions, which may lead to improved LTS predictions. However, it is often not obvious how spatial smoothness should be enforced. For example, consider a case where the motor traffic speeds of five consecutive road segments are predicted as 60, 40, 60, 40, and 60 km/h, respectively. It is likely that two of them are wrong, yet it is unclear if we should change the 40s to 60 or 60s to 40. In this section, we propose a principled way to address this problem.

Evaluation scenarios. We evaluate AutoLTS and baseline methods in three data-availability scenarios, each under four train-test-validation splits, totaling 12 sets of experiments.

For data availability, we consider LTS based on

1. 1.

   Street view image

2. 2.

   Street view image, road and cycling infrastructure types

3. 3.

   Street view image, number of lanes, and speed limit.

The design of these scenarios is informed by the real data collection challenges we encountered in Toronto. The number of lanes and the speed limit of each road segment are accessed via Open Data Canada (Government of Canada 2020). Road type and the location of cycling infrastructure are available via Open Data Toronto (City of Toronto 2020). However, as the two data platforms use different base maps, combining data from these two sources requires considerable manual effort, echoing the data collection challenges in many other cities.

For the train-test-validation split, we consider

1. 1.

   A random 70/15/15 train-test-validation split across all road segments in Toronto.

2. 2.

   Three spatial splits, which use road segments in an area as the test set and performs a random 80/20 train-validation split for other road segments. As shown in Figure 5, we consider using road segments in three of Toronto’s amalgamated cities, York, Etobicoke, and Scarborough as the test sets. These three areas have very different LTS distributions, which allows us to exam the generalization ability of AutoLTS in real-world settings.

The random split mimics the situation where we use AutoLTS to extrapolate manual LTS assessment or to update the LTS assessment in the city where the model is trained. The spatial split mimics the situation where we apply AutoLTS trained in one city to an unseen city.

Evaluation Metrics.

• •

  LTS Prediction Accuracy

  $$\textrm{Acc}=\frac{1}{{N_{\textrm{test}}}}\sum_{i=1}^{N_{\textrm{test}}}\mathds{1}[y_{i}=\hat{y}_{i}].$$

  (3)

• •

  High/Low-Stress Prediction Accuracy

  $$\textrm{HLA}=\frac{1}{N_{\textrm{test}}}\sum_{i=1}^{N_{\textrm{test}}}\mathds{1}[h(y_{i})=h(\hat{y}_{i})]$$

  (4)

  where $h$ is a function that takes a value of $1$ if the input LTS label is low-stress (LTS1/2) and takes $0$ if the LTS label is high-stress (LTS3/4).

• •

  Average False High/Low-Stress Rate

  $$\textrm{AFR}=\frac{1}{2}\left\{\frac{\sum_{i=1}^{N_{\textrm{test}}}\mathds{1}[h(\hat{y}_{i})=1]}{n^{l}_{\textrm{test}}}+\frac{\sum_{i=1}^{N_{\textrm{test}}}\mathds{1}[h(\hat{y}_{i})=0]}{n^{h}_{\textrm{test}}}\right\}$$

  (5)

  where $n^{l}_{\textrm{test}}$ and $n^{h}_{\textrm{test}}$ denote, respectively, the numbers of test road segments that are low- and high-stress.

Acc and HLA measure the overall prediction performance, while AFR considers the fact that the dataset is imbalanced with a higher portion being low-stress. Ideally, we want a model that achieves high Acc and HLA and low AFR.

Baselines. To demonstrate the value of image data, in scenarios where road features are available, we use a classification and regression tree (CART) that predicts LTS based on available road features as a baseline. CART is selected because the LTS calculation scheme (Furth, Mekuria, and Nixon 2016) can be summarized by a decision tree. We also compare AutoLTS with image-based supervised and contrastive learning methods. For supervised learning, we consider Res-50 (He et al. 2016) trained using the cross-entropy loss. For contrastive learning, we consider supervised contrastive learning (MoCo) (He et al. 2020) and self-supervised contrastive learning (SupCon) (Khosla et al. 2020), both implemented with the MoCo trick (He et al. 2020). Baselines are detailed in Appendix E.

Model details. We use ResNet-50 (He et al. 2016) as the image encoder. The normalized ReLU activations of the final pooling layer are used as the image embedding ($\xi$=2,048). We follow He et al. (2020) to set $\tau=0.07$ and use the SimCLR augmentation (Chen et al. 2020) for training. We train one ResNet-50 to predict each missing road feature. All road features are discretized using the thresholds defined in the LTS calculation scheme (Furth, Mekuria, and Nixon 2016) (Appendix C). In the LTS prediction module, we first train a CART model to predict a road segment’s LTS based on its predicted and available road features. We then use the LTS distribution in the leaf node that a road segment is assigned to as its road feature embedding, which is mapped to a $\xi$-dimensional space via a linear layer and averaged with the image embedding. Finally, a linear classifier predicts the road segment’s LTS based on the averaged embedding. Training details are summarized in Appendix D.

The performance of AutoLTS and baselines are shown in Table 2. We summarize our findings below.

The value of image data for cycling stress assessment. AutoLTS achieves LTS prediction accuracy of 62.31%–73.41% and high/low-stress accuracy of 93.50%–94.69% only using street-view images. Such a model can be useful for cycling infrastructure planning and route recommendation tools that do not require the granularity of four LTS categories and focus solely on the difference between high- (LTS3/4) and low-stress (LTS1/2) road segments. In data-availability scenarios where partial road features are available (scenarios two and three), incorporating street-view images leads to increases of 0.43–32.39 percentage points in Acc with little to no increases in AFR. The improvements are particularly large in scenario two where the average increase in Acc due to the usage of street-view images is 23.10 percentage points across all train-test-validation splits considered. By combining street-view images with the speed limit and the number of lanes (scenario 3), the Acc is over 90% under all splits. These numbers demonstrate that street view images are valuable for cycling stress assessment with and without partial road features.

The performance of AutoLTS and other image-based methods. Overall, AutoLTS achieves the highest Acc, which is of primary interest, in all evaluation scenarios. Due to the limited sample size, unsupervised contrastive learning (MoCo) generally falls around 10% behind SupCon. SupCon outperforms the simple image classification formulation (Cross-Entropy) in 8 out of 12 scenarios yet is inferior to AutoLTS in all scenarios. However, we observe that when there is a significant domain shift from training to test data (spatial splits), all methods including AutoLTS are more prone to overfitting the training data, and thus have worse out-of-sample performance than in random splits.

Next, we present ablation studies using data-availability scenario one to demonstrate the values of our two-step learning framework, ordinal contrastive learning loss, and the post-processing module, The results are summarized in Table 3.

The value of the two-step learning framework. We compare AutoLTS with an alternative approach that replaces the LTS prediction module with the exact LTS calculation scheme (2Step-Exact and 2Step-Spatial-Exact). This change leads to reductions of 6.16–40.83 percentage points in Acc due to the compounded errors from the first step, highlighting the importance of second-step learning. Moreover, AutoLTS outperforms all baselines that predict LTS based only on image (MoCo-, SupCon-, and OrdCon-NN), demonstrating the value of incorporating road feature predictions.

The value of ordinal contrastive learning. We compare the three contrastive learning methods using the AutoLTS framework and the linear classification protocol (He et al. 2020). When used to predict LTS without road features (MoCo-, SupCon-, and OrdCon-NN), OrdCon and SupCon are competitive in Acc, yet OrdCon constantly achieves higher HLA and lower AFR because it considers the relationship among LTS labels. This is practically important because the ability to distinguish between low- and high-stress roads plays a vital role in most adults’ cycling decision makings (Furth, Mekuria, and Nixon 2016). When combined with road features, all contrastive learning methods perform reasonably well. Nevertheless, Auto-OrdCon consistently outperforms others by a meaningful margin.

The value of spatial post-processing. Applying the spatial post-processing technique to road feature predictions generally leads to an increase of around 1% in road feature prediction accuracy (presented in Appendix C) which can be translated into improvements in LTS prediction Acc (2Step-Exact versus 2Step-Spatial-Exact). While the improvement seems to be limited, it corresponds to correctly assessing the LTS of 21–162 road segments in the studied area, which can have a significant impact on the routing and cycling infrastructure planning decisions derived based on the assessment.