Mapping suburban bicycle lanes using street scene images and deep learning

The official Australian design standards for bicycle lanes specify that bicycle lane markings should appear within 15m of each intersection [32]. Therefore, intersections along known bicycle lane routes are locations where example images are likely to be found.

Training and evaluation of candidate models was conducted on local infrastructure, as described in Appendix C.1. TensorFlow was configured for GPU-accelerated development, using CUDA and cuDNN libraries from NVIDIA.

The dataset that was collected in section 3.1.1.2 and labelled in section 3.1.1.3 was randomly split into “training” and “test” datasets according to an 80:20 ratio. There were 205 images in the “training” dataset, and 51 images in the “test” dataset.

The Jupyter Notebook described in appendix A.1.6 was used to download a candidate pre-trained model from the TensorFlow 2 Object Garden and configure it to search for bicycle lane markings based on the collected dataset. The Jupyter Notebook printed out commands that could be used from the command line, to train the model for a specified number of training steps against the “training” dataset, evaluate its performance against the “test” dataset, and create a “frozen” copy of the model as-at the current level of training.

Three significant factors were found to influence the performance of the model: The size of the dataset available for “training” and “test” purposes; the number of “epochs” or “training steps” spent training the model on the data; and the “confidence score ” threshold, from 0% to 100%, at which the model declares that a bicycle lane marking has been detected.

The initial size of the dataset was set based on previous experience in other papers such as Campbell et al., 2019 [30], where models had been successfully trained with around 500 images. An initial dataset of 256 images was gathered, and further images were added later in response to model performance.

During the training process, the TensorFlow 2 training script reported “loss” performance metrics every 100 training steps. This performance metric typically gradually improved as more training was conducted over the dataset, until a point of diminishing returns was reached. Every 2000 training steps, the training was paused, and an evaluation script was run to test the performance of the model against the independent “test” dataset that had been withheld from the training process. When training reached the point that further training steps did not significantly improve performance, training was halted.

The “confidence” threshold was adjusted manually, by reviewing the false positives and false negatives produced by the model, and adjusting it to provide a balance between over-confidence and conservatism. As part of the evaluation process, copies of each image were written to disk with an overlay showing any detected bounding box and its confidence score. These outputs were reviewed as part of the threshold tuning process. The boundary boxes of detections were also checked to ensure that the results were sensible.

Multiple pre-trained models from the TensorFlow 2 Model Garden were trialled, with selections guided by their benchmark “mAP” (mean Average Precision) metrics for processing of the “COCO 17” dataset. The process of building a map of bicycle lanes for an area can be considered a “batch” process, with no fixed time constraints, therefore mean Average Precision was prioritized over benchmark “speed”.

The TensorFlow 2 Model Garden provided an efficient way to test multiple models via a single framework. There are popular object detection models, such as the “YOLO” series of models, that are not supported by it. For the purposes of this research, it was not considered necessary to perform an exhaustive search of all possible models across multiple frameworks. A different model (e.g. YOLOv3) or a different framework (e.g. PyTorch) could be substituted if required.

Please refer to Appendix A.1.6 for instructions on how to run the training process.

In section 3.1.1.2, it was noted that Google charges up to $7.00 USD per 1,000 Google Street View images, and that the most likely place to find a bicycle lane marker is in the area immediately surrounding an intersection. To reduce costs, a strategy of sampling only intersections and their immediate vicinity was adopted. Samples could also be taken at regular intervals along each road, if initial results suggested that this was necessary to compensate for missed detections.

The Jupyter Notebook described in appendix A.1.8 was used to load an OpenStreetMap XML extract file for the survey area into memory, and then traverse it to produce a batch file with the coordinates of every point within 20 metres of an intersection, at intervals of 10 metres. The length of the batch file was inspected, to assess potential costs, and then the required images were downloaded via the Google Street View API, if they were not found in the local cache.

It was discovered that some intersections would be missing on roads at the very margins of the survey area, if the intersecting road existed entirely outside the bounds of the OpenStreetMap XML extract for the survey area. A second extract, capturing bounding box with an extra margin of 200 metres around the survey area, was used to rectify the missing intersection issue. The 200 metres margin was found to be sufficient to ensure that the adjoining road was included in the extract by the osmium tool. Roads in the second extract were not considered part of the survey, but data about them was taken into account to ensure that intersections at the margin of the survey area were not missed.

Please see appendix A.1.9 and A.1.10 for detailed instructions on how to operate the tools that were used to generate a list of sample locations in a survey area.

The Jupyter Notebook described in appendix A.1.10 was also used to apply the detection model to the sampled Google Street View images, and produce an output CSV with the coordinates of every point where a bicycle lane marking was detected. If one or more bicycle lane markings were detected in an image, a copy of the image was saved to a folder of “hits” with a bounding box and confidence score overlay. An example image with detection overlay is shown in figure 3.2. If no bicycle lane marking was found, a copy of the image was instead placed into a “miss” folder. Partitioning the result images into “hits” and “misses” allows the operator to quickly browse the images to check for any false positives or false negatives.

The result of the batch process in section 3.2.2 is a list of locations where a bicycle lane marking was found, linked to the OpenStreetMap “way” IDs and “node” IDs that they were sampled from. The next step is to use those detection points to infer contiguous routes that can be drawn as lines on a map.

Four Google Street View images were sampled at each sample point in the survey, providing a 360 degree view. An approximate heading was calculated for each road at each intersection, but occasionally that did not appear to match the retrieved images. Therefore, if a bicycle lane marking was detected at an intersection, it could be difficult to accurately decide which road it belonged to.

The proposed solution was to infer routes from the detection points based on the assumption that if markings were detected at two or more consecutive intersections along a named road, there is a bicycle lane along that segment. A configuration option was provided to allow for one or more intersections with “missing” detections before assuming that the bicycle lane route has been interrupted.

If a street shared an intersection with a road that had a bicycle lane, it would not usually be reported as having a bicycle lane unless it had another intersection with a bicycle lane at the next intersection.

A continuous road may be represented in OpenStreetMap as multiple “way” segments if any of its characteristics change from one segment to the next, such as a change of speed limit. Before inferring bicycle lane routes along a road from the detection points, it was necessary to link adjacent “way” segments from a single named road back up into a single chain. Otherwise, the option to allow for intersections with “missing” detections would not always work as intended.

Each inferred route was drawn as a line in a geojson file, which can be drawn on a map or measured as per section 3.2.4 below.

This approach was intended to draw a map of bicycle lane routes irrespective of which side of the road the bicycle lane is on. If additional information is required about roads where bicycle lanes are only present on one side of the road, the process would need to be refined to take into account the heading of the road, the heading of the camera angle, and exactly where in the frame the bicycle lane marking was detected.

Once a geojson file has been produced to describe the detected bicycle lane routes, it can be drawn on a map in a Jupyter Notebook using the Python “ipyleaflet” library. See appendix A.1.11.

The OpenStreetMap XML extract for the survey area can be filtered down to “ways” that are roads with the “cycleway” tag, and then converted into an equivalent geojson file. The detected routes and the routes that are tagged in OpenStreetMap can be viewed side-by-side on a map.

To allow more detailed comparisons, the geojson files were drawn simulaneously, in a single process. For each road where a bicycle lane marking was detected, or tagged in OpenStreetMap, a process “walks” down “nodes” in order. If the nodes are part of a “way” that has been tagged as a “cycleway”, their coordinates are added to a line in the “OpenStreetMap” geojson. If a detection point is encountered at two intersections in a row, then the intersection nodes and the nodes in between them are added to a line in the “detected routes” geojson. At the same time, additional geojson files are drawn to show the route sections where both sources agree that there is a bicycle lane, and route sections where one source registers a bicycle lane and the other does not.

The total length of all routes in a geojson file can be calculated by using the Python “geographiclib” library to measure the distance between each pair of points along the route. Therefore, the geojson files can not only be used to compare routes on a map, but to measure the similarities and differences in metres.

The “Principal Bicycle Network” dataset is available in a geojson format. This allows a quick visual comparison with OpenStreetMap and the detected routes. It is possible to align the “Principal Bicycle Network” dataset to the OpenStreetMap data to perform a detailed comparison with measurements, however this was not attempted. The “Principal Bicycle Network” dataset had significant gaps in the survey areas, so significant that quantifying the differences in metres would not have been very meaningful. Please see the results section 4.2.

For these experiments, the “Navman MiVue 1100 Sensor XL DC Dual Dash Cam” was self-installed inside a sedan, at a cost of $529 AUD. This model was selected because it provided location metadata in NMEA files accompanying the MP4 video recordings. For each minute of footage recorded, the device produced an MP4 file and a corresponding NMEA file. The NMEA file provided the latitude, longitude, heading, and altitude at one second intervals.

Video footage was recorded at 60 frames per second, at 1920x1080 resolution. Only images from the front-facing camera were used.

The camera features a “wide angle” lens to capture a good view of the road and its general surroundings, however optical specifications such as focal length were not available from the manufacturer’s website.

On the 3rd of October, 2021, Melbourne set the record for the longest “COVID lockdown” in the world [46]. Severe restrictions were placed on residents being more than 5km from home, gradually easing to 10km and then 15km for a period of many months during the course of the research[47]. Permitted reasons to leave home were restricted. Therefore it was only possible to experiment with dash camera footage from a confined area.

Approximately 100 minutes of footage was gathered by driving within a local area around the outer metropolitan suburbs of Mount Eliza, Frankston, Langwarrin and Baxter in Melbourne, Victoria, Australia. Not every road in the area was covered. Roads were selected for inclusion in the footage to include all known bicycle lanes in the area, based on local knowledge and a review of the OpenStreetMap data. A variety of other roads were included, to cover roads with and without a paved shoulder, major divided roads and small local roads, and a mix of residential and commercial areas.

The dash camera video footage and corresponding NMEA files were transferred to a computer and processed using the Jupyter Notebook described in appendix A.1.12.

For each pair of files, the NMEA file was loaded into memory, then the video file was split into a series of images. The 60 frame per second footage was reduced to 5 frames per second by only sampling every 12th image. Sampling images at a higher frequency than this did not appear to be necessary, due to the high similarity between adjacent frames. Each extracted frame was assigned a location by extrapolating the readings from the two nearest measurements in the NMEA file. An entry was then added to an output “batch” CSV file with the path to the image file, and its location metadata.

The dash camera footage was initially processed using the original model that was trained exclusively on Google Street View images in section 3.1, without any retraining based on dash camera images. The original model was able to detect every bicycle lane marking in the test footage, however further training and refinement was required to address some performance issues:

• •

  Although the Google Street View-trained model was able to detect every marking during initial trials, it typically only did so when the marking was very close to the camera vehicle. Further training was apparently needed on footage from the dash camera in order to allow it to detect markings that were further away, but still close enough to be clearly visible.

• •

  There were also issues with false positives that had not been apparent when using the Google Street View model on Google Street View images. The model would sometimes yield false positives in the following situations:

  • –

    Other white markings on the road, such as turning arrows, diagonal stripes to represent a traffic island, “keep clear” signs, or dashed lines advising traffic to “give way” at an intersection

  • –

    Pale road defects or other anomalies such as manhole covers

  • –

    Reflections from the bonnet of the camera vehicle

  • –

    Clouds or random objects well outside the boundary of the road

See figure 3.4 for examples.

A decision was made that the detection model needed to be trained with additional data from the dash camera footage. The footage was divided up into a “training area” consisting of footage captured in the Frankston, Langwarrin, and Baxter area, and “test area” footage from Mount Eliza.

The Jupyter Notebook documented in appendix A.1.16 was used to process the dash camera footage from the “training area” with the initial detection model that had been trained exclusively on Google Street View images. Every image that yielded a detection – whether a true positive or a false positive – was then labelled and allocated to the “training dataset” and “test dataset” according to an 80:20 ratio. A total of 342 images were added to the datasets, supplementing the existing Google Street View images.

The original Google Street View training images were labelled with a single class “BikeLaneMarker”. The new training images were labelled with the following additional classes, to encourage the detection model to think of many of the sources of false positives as something other than a bicycle lane marking: “ArrowMarker”, “IslandMarker”, “RoadWriting”, “GiveWayMarker”, and “RoadDefect”.

The Google Street View images that had already been included in the dataset were not re-labelled to consider the additional classes, though this was considered as an option if required to improve performance.

To reduce false positives due to reflections from the bonnet of the camera vehicle, or clouds and other random objects well away from the road, a detection mask was defined so that bicycle lane markings would only count as detections if they fell into an area on the left hand side of the road, in front of the bonnet of the car. See figure 3.5 for a visualization of this mask in an example image.

The new images for the enhanced “training” and “test” datasets were gathered and labelled via the Jupyter Notebook documented in appendix A.1.13 and the “labelImg” tool. A new model was then trained and evaluated via the Jupyter Notebook documented in appendix A.1.14, and its predictions for the “training” and “test” datasets were analysed in detail using the Jupyter Notebook documented in appendix A.1.15. Finally, the selected model was applied to the dash camera footage from the “test” area of Mount Eliza via the Jupyter Notebook documented in appendix A.1.16.

Section 3.2.4 described the process that was used to produce geojson files to compare routes that were detected from Google Street View images to the routes that are recorded in OpenStreetMap. When sampling images from Google Street View, each sample location can be traced back to a “way” and a “node” in OpenStreetMap, where the latitude and longitude details came from. Therefore any detections are already precisely aligned to points in the OpenStreetMap extract.

Images that were collected from dash camera footage take their coordinates from the dash camera’s GPS receiver, and they are sampled from locations anywhere along the road. To infer routes from bicycle lane markings detected in dash camera images, each detection location is mapped to the nearest intersection “node” in the OpenStreetMap data. Once this has been done, the rest of the process is the same as for detections from Google Street View images.

To find the nearest intersection node to a point, the Python “shapely” library is first used to find the nearest “way” to the point. The “shapely” library appears to use bounding boxes to accelerate this process: It was significantly faster than a brute-force search of all nodes in the OpenStreetMap data. Once the closest “way” is found, it is traversed to find the nearest “node” that is an intersection. At the same time, the process records the nearest “node” of any type, to help with traceability in case the detection occurred a significant distance away from the nearest intersection.

Wherever OpenStreetMap represented a highway or other divided road as multiple parallel ways, the above process was able to distinguish the side of the road that was closest to the camera position. It is important to gather footage from both sides of a divided road when measuring the differences between the generated map and OpenStreetMap. Otherwise, OpenStreetMap might count a bicycle lane on two “ways” for the divided road, where the generated map only counts one.

Lens distortion is where the optical properties of a camera’s lens causes straight lines to appear curved in an image. The “OpenCV” library provides a method to correct distortion for a specific camera [48]. The camera is used to capture a range of images where a special calibration tool appears in frame. See figure 3.6 for some example images. An image of a “chessboard” pattern of known dimensions is held in front of the camera in different positions.

After the calibration images are collected, they are processed by OpenCV to create a mathematical model of the camera’s lens distortion. With this model, OpenCV can apply a transformation to images from the same camera, to correct for lens distortion.

When searching for paved shoulders in a survey area, we are looking for lines in an image that are often straight. The dash camera being used to collect images was therefore calibrated, and in this exercise, all images were processed to correct for distortion. See figure 3.7 for an example raw image from the dash camera, and a corresponding corrected version.

There was not much difference between the corrected and uncorrected images. The most obvious sign that the correction has been applied in figure 3.7 is that the top edge of the corrected image is bent downwards slightly, and the location/speed/time information that usually appears in the bottom right corner has been shifted. Although the operation to correct for lens distortion did not appear to have a significant impact when it came to detecting paved shoulders, it was retained in the pipeline to support any future photogrammetry work to estimate the widths of the lanes.

A common solution to the problem of detecting lane markings involves applying a combination of the Canny edge detection algorithm proposed by Canny, 1987 [35] and the Hough transformation proposed by Hough, 1972 [36]. This general approach has been followed in many papers such as Li et al, 2016 [49] and Chai et al, 2014 [50], and it is frequently used in capstone projects in the self-driving car domain. The approach does not involve any machine learning or deep learning techniques.

The Canny-Hough approach typically works as follows:

1. a.

   Correct the image for lens distortion.

2. b.

   Apply Canny edge detection to detect edges in an image [35].

3. c.

   Apply a mask to exclude edges outside a triangle representing the area immediately in front of the vehicle.

4. d.

   Apply a Hough transform [36] to convert these edges into lines, each line having a slope and an intercept.

5. e.

   Partition the lines into two groups: Lines that slope upwards when scanning from left to right, and lines that slope in the opposite direction. For each group of lines, take the average slope and intercept. These represent the assumed lane boundaries. Optionally redraw the image with the detected lines drawn as an overlay, to visualize them.

Please see figure 3.8 for example images to demonstrate the process.

The “OpenCV” library provides convenient tools to implement the required transformations.

Typically, the Canny-Hough process is used to detect the camera vehicle’s own lane. In order to detect a paved shoulder, we first want to detect the boundaries of our vehicle’s own lane, then perform a second pass at the image where the mask has shifted to focus exclusively on areas further to the edge of the road.

A paved shoulder will be defined by two lines, each with a “slope” and an “intercept”. One line will shared with a boundary of the camera’s own lane. The other line will be the edge of the road. To detect this extra line, we apply a mask to the Canny edge detection image just to the left of the camera’s own lane boundary that was found in the first step. We then apply the same Hough transformation to detect lines, and average the ones with the expected “left” slope, to find a slope and intercept for the outer boundary of the paved shoulder. See figure 3.9 for an example.

If the Canny-Hough approach cannot find another line to the left of the camera vehicle’s own lane, it is a strong sign that there is no clearly defined paved shoulder on the road at that location. If a line is found, it could be a paved shoulder, or it could be a false positive caused by “noise” from something on the side of the road, but within the area included by the mask.

Please see figure 3.10 below for example images where lines were detected for a possible paved shoulder. In each of the example figures, two horizontal lines have been drawn. The lower horizontal line shows where the detected lines intersect a horizontal line just above the bonnet of the car, the nearest point in the frame with a clear view. The upper horizontal line shows where they intersect at an arbitrary point further into the distance. Vertical lines are also drawn to highlight the intersection points. Together, the horizontal and vertical lines are a visual aid to help assess the width and stability of any detected lanes, as the operator views the images quickly in sequence.

Figure 3.10(a) shows a true positive match, where the boundaries of a paved shoulder have been correctly identified. Figure 3.10(b) shows a situation where a concrete gutter has resulted in a very narrow area being detected. This potential for this type of false positive can be mitigated by requiring a minimum width for any detected lane. Figure 3.10(c) shows a false positive match, where “noise” from a low wooden fence parallel to the road has caused the average of lines to the left of the camera’s old lane to appear to be a paved shoulder, off-road and in the grass. This phantom line is not “stable” and only appears for a few frames. This type of false positive can be reduced using metrics that require a paved shoulder to be consistently detected across most images in a road segment, and/or requiring that the slope and position of the line not vary too much from frame to frame.

The Canny-Hough lane detection method can be applied to all sample frames from the survey area, using a similar batch process to the bicycle lane detection process in section 3.3 and substituting a different detection model. The challenge is setting robust criteria for flagging a “detection” that will result in a sensible map. The simplest criteria would be to look for frames where lines were detected for both boundaries of a paved shoulder, but when it was applied to the training area footage, the results were very inconsistent. On roads where there was a paved shoulder or bicycle lane, the boundaries of that lane remained relatively stable from frame to frame, and the width between the boundaries of the detected area was relatively wide. There were few “skipped” frames where the boundaries were not detected. In contrast, on roads without a paved shoulder or bicycle lane, there may be many “skipped” frames, the slopes of the detected boundaries were inconsistent from frame-to-frame, and the detected boundary lines often had an intersection point much closer to the bottom of the frame.

The following solution was proposed:

• •

  Each frame would be linked to its two nearest intersection “nodes” in the OpenStreetMap data. All frames associated with a single stretch of road between two intersections would be assessed as a group.

• •

  If the model fails to find both boundaries of a paved shoulder in more than a threshold percentage of frames along a stretch of road, then it is assumed that there isn’t one.

• •

  Frames from locations within 30 metres of an intersection were excluded from consideration, to account for routine tapering of paved shoulders at intersections.

• •

  A horizontal line is drawn across the frame towards a “horizon” as per figure 3.10. The width of the space between the boundary lines is measured at this height within the frame, in pixels. The mean width is calculated across all frames in the stretch of road where boundary lines were detected. If the mean is less than a threshold number of pixels, then the paved shoulder is too narrow, and it is assumed not to exist. This accounts for very narrow margins such a gutter that are of no use to a cyclist.

• •

  If the model finds both left and right boundary lines for a possible paved shoulder, use the slopes and intercepts to calculate an (x, y) coordinate relative to the bitmap image where those two boundary lines intercept. Calculate the standard deviation of the x and y intersection values across all frames in the stretch of road where both boundary lines are defined. If the standard deviation in either the x or y dimension is greater than a threshold number of pixels, then the boundary lines are considered to be too inconsistent across images. The stretch of road is assumed not to have a paved shoulder.

The thresholds described above were tuned by analysing the dash camera footage from the “training” area and producing a CSV file with one record per frame, in chronological order. Each frame was mapped to the nearest “way” and the two nearest intersection “nodes” in the OpenStreetMap data, to associate it with a particular stretch of road, from intersection-to-intersection. It was also labelled with the name of the road, for traceability. The required summary statistics were calculated for each group of frames associated with a stretch of road, and these were attached to the CSV file as additional columns. Finally the footage was reviewed, and using the road names, changes of heading and “node” IDs as a guide, each record was labelled to record whether there was really a paved shoulder there or not.

A manual tuning process was performed, filtering the rows based on the column recording the “ground truth”, to find thresholds that appeared to correctly predict the outcome as often as possible.

The model was then applied to the footage from the Mount Eliza “testing” area to see how it performed with the selected thresholds.

To construct a map of detected paved shoulders, the dataset was summarised to a list of distinct combinations of “way” ID and intersection “node” IDs for each road section where a paved shoulder was predicted. This information was cross-referenced to the OpenStreetMap data for each “way” ID, and a line was “drawn” in a geojson file for all “nodes” on the way between the two intersection nodes, including the intersection nodes themselves. The geojson file could then be drawn on map in a Jupyter Notebook using the Python “ipyleafelet” library as per sections 3.2 and 3.3.

In this exercise, lane detection was attempted from the dash camera footage only, as a demonstration of how the techniques developed to address the first three research questions could be re-used to gather other information visually. It could be applied to Google Street View images, however it was only attempted with the dash camera footage, where it is easier to ensure that each image is taken with a heading that is consistent with the direction of the road.

The model that was selected in section 4.1 was applied to a sample survey area in the vicinity of Mount Eliza. This area was chosen due to its accessibility: It was one of very few areas with bicycle lanes that was reachable within the COVID-19 lockdown restrictions that were in place at the time [46] [47]. It was therefore possible to validate the results via an in-person survey, in addition to comparing the results to OpenStreetMap and the “Principal Bicycle Network” dataset. It also allowed for comparison with results from dash camera footage in section 4.3.

A total of 1,113 locations were sampled from the survey area, at points within 20 metres of every intersection. Four images were downloaded via the Google Street View API at each location, a total of 4,452 images at a cost of approximately $31 USD. The exact survey area within Mount Eliza was chosen to include a mix of roads with and without bicycle lanes, ranging from no-through roads to a highway.

Figure 4.2 is a screenshot from an interactive map in the Jupyter Notebook described in appendix A.1.11, showing existing and planned bicycle routes in the survey area according to the “Principal Bicycle Network” dataset. The blue bounding box shows the survey area. The red lines show “existing” routes, and the blue lines show “planned” routes. The “Principal Bicycle Network” dataset is out of date. It correctly records that there is an existing route on the Nepean Highway, in red. However, the long, straight “planned” route, Humphries Road, has existed for at least several years, and so has the route to the north-east of it, on Baden Powell Drive. The planned route on Baden Powell Drive to the west of Humphries Road does not yet exist.

In section 4.1, the models were trained for 2,000 training steps, beyond which further training did not appear to yield any improvements when the model was evaluated against the small 51 image “test” dataset. The Faster R-CNN model appeared to be the most promising. However, when that model was applied to Google Street View images from the Mount Eliza survey area, the initial results were poor. Precision was significantly lower than it had been in the initial evaluation, with many false positive detections for random objects such as bushes, or leaves on the road. The models received further training on the original “training” dataset, up to 30,000 training steps, and the CenterNet model had the best performance, producing zero false positive detections after 30,000 training steps. It was therefore used to construct maps from the Google Street View images in the survey areas. See figure 4.3 for training statistics.

Figure 4.4(a) shows the individual points that were detected by the model from Google Street View images sampled near intersections in the survey area. Figure 4.4(b) shows the bicycle lane routes that were inferred from those detections, according to the method described in section 3.2.3. The routes are colour-coded to highlight areas of agreement and disagreement between the detection model and OpenStreetMap. Where both the detection model and OpenStreetMap agree that there is a bicycle lane, routes are coloured green. They agree that there is a bicycle lane on Humphries Road. Routes that were detected by the model but not recorded with a “cycleway” tag in OpenStreetMap are coloured blue. The OpenStreetMap data is missing the cycleway on the Nepean Highway, and the fragment of the Baden Powell Drive cycleway that was detected near the boundary of the survey area. Routes that are recorded in OpenStreetMap but were not detected would be coloured red, however there were no such routes in this survey.

Based on an in-person survey of the area, and inspection of the Google Street View images that were downloaded, the detection model correctly detected all bicycle lanes within the survey area, with no false positives. It identified two routes that had not been recorded in OpenStreetMap.

After surveying Mount Eliza with the Google Street View approach, the exercise was repeated for a survey area in Heidelberg, Victoria. The purpose of this exercise was to assess how well the detection models generalised to a more heavily built-up area, closer to the city. It included both residential and commercial areas. See figure 4.5 for the results.

The red lines in the map again represent routes that are recorded in OpenStreetMap but not detected. Any red lines outside the bounding box describing the survey area can be ignored. The Google Street View images for the red lines within the survey area were examined manually, to assess whether the model had produced sensible results based on the input. Zooming into the red diagonal line on Studley Road near the Heidelberg train station, it can be seen that the model did correctly predict that there was a bicycle lane, but it only marked it on one side of the road, whereas it is tagged separately on both sides of a divided road in OpenStreetMap. The remainder of the red lines are routes where there is no bicycle lane visible in the Google Street View images, and the model has produced a sensible result. See figure 4.6 for example images from Cape Street and Stradbroke Street. It is possible that this is caused by out-of-date Google Street View images, however the images for Stradbroke Street were only six months old, and the streets did not seem wide enough to easily accommodate the addition of a bicycle lane. These routes might be errors in the OpenStreetMap data. An on-site survey by a local resident with access to the area confirmed this.

The blue lines near the Austin Hospital and Heidelberg Station are false positives. They are caused by the road being close enough to see a bicycle lane marking on a nearby road.

There is a very short segment where OpenStreetMap records a cycleway on Lower Heidelberg Road near the intersection of Banksia Street. This route would be barely larger than the nearby intersection, and it is almost certainly an error in OpenStreetMap.

A comparison between the improved detection model results and OpenStreetMap in terms of route lengths can be found in table 4.4.

The model was able to identify all bicycle lane routes that were recorded in OpenStreetMap and visible in the Google Street View images. It produced 469 metres worth of false positives, due to bicycle lane markings on one road being visible from multiple points on a nearby road. An approach using dash camera footage could help to address this issue, by ensuring that the camera is always correctly oriented to face the direction of the road, thereby allowing detection masks to be used to eliminate bicycle lane markings from nearby roads.

Due to COVID-19 restrictions [46] [47], dash camera footage could only be gathered in a circular area with a 15km radius, much of which was a body of water. Therefore, the range of areas available to gather dash camera footage for research question 3 was restricted to a handful of outer metropolitan suburbs. Footage from the Frankston, Langwarrin and Baxter area was used to train a the detection model to work with dash camera footage, and footage from Mount Eliza was used to test it.

Training was conducted as per the approach documented in section 3.3. Initially, the original model from section 4.2 that had been trained exclusively on Google Street View images was used to process the dash camera footage from the training area. The initial positive matches – whether true positives or false positives – were added to the “training” and “test” datasets according to an 80:20 split. The resulting “training” dataset consisted of 205 Google Street View images and 252 dash camera images. The “test” dataset consisted of 51 Google Street View images and 85 dash camera images.

Another “Faster R-CNN ResNet50 V1 640x640” model was then trained and evaluated using the combined Google Street View + dash camera footage datasets. To deal with remaining false positives, the dash camera images were labelled with additional classes to distinguish common misleading features from bicycle lane markings, and a detection mask was applied to remove reflections from the bonnet of the car, and distractions from the right hand side of the road. Please see section 3.3 for a full discussion of these enhancements. The results observed from the model after every 2,000 training steps are recorded in figure 4.7 and table 4.5.

Figures 4.7(a) and 4.7(b) show that there were diminishing returns from further training beyond 4,000 to 6,000 training steps, in terms of both the “Total Loss” reported in the training process, and the “mean Average Precision” of the bounding boxes reported from the evaluation process against the “test” dataset.

The model was applied to the “training” and “test” images via the tool documented in appendix A.1.15 to examine the bounding boxes it chose to draw, and whether it believed there was a detection or not. The results of this review are found in the “False +ve” and “False -ve” columns reported in table 4.5. After 6,000 training steps, the model was able correctly detect all bicycle lane images in the “training” data, with no false positives, false negatives, or misplaced bounding boxes. In the test data, there were a total of 3 false positives and 3 false negatives out of 136 test images. These false results came from a mix of Google Street View images and dash camera images. After 8,000 training steps, the performance improved further, in that there was 1 false positive and 2 false negatives when analysing the test data, and these were all for challenging Google Street View images. See figure 4.8.

In figure 4.8(a), the model has correctly identified two bicycle lane markings in the central business district of Melbourne, but it has also identified a white strip that was installed at a pedestrian crossing to assist people with impaired vision. A very high confidence of 95% was assigned to the false positive detection. To assist the model to deal with this situation better, more training images would be required, with these objects given a label to steer the model away from thinking of them as a bicycle lane marking.

In figure 4.8(b), the model has missed a very faint bicycle lane marking that was alongside the camera vehicle, in the shadow of the vehicle. This was a very challenging image due the the faintness of the marking, and it was common for the models tested in this research project to miss it.

In figure 4.8(c), the model has missed a bicycle lane marking on the opposite side of the road, at a challenging angle and distance. In general, most models struggled with this test image as well.

Training was terminated after 8,000 steps, and the model was then used to conduct a survey from dash camera footage in the “test” area of Mount Eliza. At 8,000 training steps, the only incorrect predictions reported were for Google Street View images. For this section, the model was being applied to images from dash camera footage, where there were many more frames available to compensate for any missed detections in individual frames.

Figure 4.9 shows a comparison between the bicycle lanes in Mount Eliza that were detected based on dash camera images, and the routes that are recorded in OpenStreetMap as having bicycle lanes, for roads that were surveyed in the footage only. The green routes show where there is agreement between the detection model and OpenStreetMap. The blue routes show bicycle lanes that were detected but not recorded in OpenStreetMap. The red routes show bicycle lanes that are recorded in OpenStreetMap but not detected.

The model correctly detected three sections that are not recorded in OpenStreetMap. There is a cycleway on the Nepean Highway that has not been tagged in OpenStreetMap in the northern section depicted on the map. There is a narrow cycleway on Baden Powell Drive to the north-east of Humphries road. And there is a cycleway on Two Bays Road to the south-east of the survey area.

There is a short segment in the northern section of Mount Eliza Way where a bicycle lane is tagged in OpenStreetMap but was not detected by the model. The detection model appears to be returning a reasonable result in this instance, the bicycle lane does not extend beyond the roundabout at Kenaud Avenue.

There is a segment on Wooralla Drive where a bicycle lane is tagged in OpenStreetMap. This is an error, there is no bicycle lane on that road.

There is a longer segment on the Nepean Highway, leading up to the intersection with Tower Road, where the detection model failed to draw a bicycle route. This is the only part of the map where the detection model appears to have made an error. During the survey, the camera vehicle turned into Tower Road and missed that segment. If the vehicle had continued along Nepean Highway, it might have picked up the next bicycle lane marking and drawn a continuous route.

A comparison between the improved detection model results and OpenStreetMap in terms of route lengths can be found in table 4.6.

Overall, the detection model performed well within the survey area. It is possible that multiple surveys spread over multiple weeks would have filled in the gap at the edge of the survey area, on the Nepean Highway. The model detected three route segments that are not recorded on OpenStreetMap, and correctly identified two route segments that appear to be recorded in OpenStreetMap in error. It missed one route segment that was recorded in OpenStreetMap, at the edge of the survey area, but perhaps repeat surveys or a survey route that extended further down the Nepean Highway may have caught this. Both OpenStreetMap and the detection model included two routes that are not listed in the “Principal Bicycle Network” dataset as existing routes.

The model proposed in section 3.4 was first applied to the training area footage from Frankston, Langwarrin, and Baxter, and then validated against the same test area footage from Mount Eliza as used for research question 3.

It was found that, when processing an individual frame, the proposed method was vulnerable to “noise” from objects just outside the paved area, and this led to the use of thresholds described in section 3.4.3. After analysing the observed data and ground truth for the “training” area, the following thresholds were set:

• •

  A minimum of 80% of frames in a road segment must have a paved shoulder detected

• •

  The mean width of paved shoulders detected in a road segment, measured at the upper top horizontal line drawn in the overlay, must be at least 75 pixels

• •

  The standard deviation of the X and Y co-ordinates at which the paved shoulder boundary lines would intersect must be less than or equal to 50 pixels

These thresholds were intended to filter out road segments with inconsistent detections, paved shoulders that are too narrow to be used by a cyclist, and false positive detections where the perceived paved shoulder boundaries jump around wildly from frame to frame.

The map of paved shoulders that was generated for the test area of Mount Eliza from the dash camera footage can be found in figure 4.10.

The map gives relatively robust results on Humphries Road and the Nepean Highway, where there is a bicycle lane, a special case of a paved shoulder. Part of the Nepean Highway was not detected, where the lane markings have faded and the “noise” from the foliage dragged the boundary of the lane detected for camera vehicle itself too far to the left.

The other lines drawn on the map represent true paved shoulders, but they were easily interrupted by “noise” from the side of the road, or parked cars. Many of the interruptions in the paved shoulder route on Old Mornington Road, to the west of the the Nepean Highway, were attributable to parked cars.

The model did not present any false positives, and it appeared to find most roads with paved shoulders, but the routes drawn were often broken up.

The use of Google Street View images to map bicycle lane routes in Mount Eliza was effective, and added value, finding routes that had not already been recorded in either the official “Principal Bicycle Network” dataset or the crowdsourced OpenStreetMap database. Repeating the exercise in the more heavily built-up suburb of Heidelberg was less effective: There were some false positives, and every real route that was identified was already included in the OpenStreetMap database.

The main limitations of the Google Street View approach are the cost of images and gaps in coverage they can provide.

The Google Street View API appears to be able to provide a distinct snapshot every 10 metres. A town or suburb can easily have many kilometres of roads, and if every snapshot in a town were sampled, or multiple towns were surveyed, the amount payable to Google for the use of their API could add up quickly. The proposed solution attempted to mitigate this by only sampling the area immediately around intersections, and making an assumption that a bicycle lane route continues interrupted if detections are missed at only one or two intersections in a row. If API costs need to be reduced further, then perhaps OpenStreetMap tags could be used to eliminate roads that are for “local traffic only”, where bicycle lanes might not be as necessary. Or, the survey could focus on roads that are adjacent to known bicycle lane routes, where continuations of the known routes are waiting to be discovered.

One significant challenge is the recency of the Google Street View images available. For example, the latest available Google Street View images for the Mount Eliza survey area are dated October 2019. If any new bicycle lanes had been constructed in the past two years, the model would have been unable to detect them. The Google Street View approach can only bring a dataset of bicycle lane routes up to date as far as Google’s last visit to the area.

The Google Street View camera vehicle typically only provides a view from one lane, on one side of the road. Bicycle lane markings on the other side of the road may be difficult to spot, especially for a divided road with foliage or other barriers dividing the traffic. This is especially important if the bicycle lane route map must show which sides of the road a bicycle lane is present. It may be impossible to detect a bicycle lane on a service road running in the opposite direction to the lane from which the Google Street View snapshot was taken. Hopefully these challenges are sometimes mitigated by taking samples near intersections from multiple approaches, where some of the bicycle lane markings might be clear.

The model was able to detect bicycle lane markings even when they were moderately far into the distance, despite the 640x640 resolution limit. Sometimes, the bicycle lane marking was “smeared” by artefacts in the Google Street View image, but with enough training examples, the model was able to detect these correctly.

The approach of using a 360 degree view at each intersection to infer routes was vulnerable to false positives where bicycle lane markings were visible from a separate road, nearby. Using front-facing footage from a dash camera and applying detection masks could resolve this issue.

The survey area of Mount Eliza is in an outer metropolitan area, and the COVID-19 lockdown restrictions have triggered political debate about whether it should even be considered “metropolitan” at all. Most residential roads in the area do not have pedestrian footpaths. When the model is applied to higher-density suburbs, such as Heidelberg, it is likely to need more images in the “training” and “test” datasets to cover the additional distractions that might appear on the side of the road. With more traffic on the roads, it might be much more difficult to detect a bicycle lane marking on the opposite side of a road or intersection. There might therefore be a much higher reliance on detecting markings immediately in front of and to the left of the camera vehicle, where there is less chance of another vehicle obscuring a bicycle lane marking.

Difficulties in heavily built-up areas might matter less if the bicycle lane routes in those areas have already been well mapped due to their priority and a large number of interested cyclists who could contribute to OpenStreetMap.

The use of dash cameras to gather images would overcome many of the limitations of the Google Street View data. If the latest available images are too old, then there is freedom to go and collect more at any time. Images can be gathered from both sides of the road, from any lane or service road required. The driver can drive conservatively, to limit the degree to which the view is obstructed by a vehicle in front. Many more snapshots are available: If the camera vehicle is driving at 50kmph, that equates to just under 14 metres per second. A camera recording at 60 frames per second would record a frame every 23cm, compared to the 10 metre intervals provided between snapshots by Google Street View. The limiting factor becomes how much data is meaningful to store and review, and missed detections may be compensated for by analysing nearby frames, or through repeat surveys of the area over time.

The results obtained from the dash camera survey were at least the equal of the Google Street View survey of a smaller sample area. The dash camera approach has the potential to be more robust, and better able to deal with the technical challenges that limit the Google Street View approach.

The main challenges with the dash camera approach involve logistics. With Google Street View, you can survey as wide an area as you can afford to download. With the dash camera approach, someone must install and maintain a dash camera, drive the required routes, and send the data off for processing. This could be done by interested volunteers for individual areas, or by individual local governments who are aware that they have significant deficiencies in their data. Perhaps the most promising prospect would be to “incidentally” gather footage from local government vehicles that regularly drive most routes for another purpose. For example, garbage trucks would cover all residential streets on a regular basis. The business case for collecting, managing, and processing the footage could be justified not only by the value of mapping bicycle lane routes, but by the value of any other information that could be gathered visually and correlated to a location. For example, a local council might be interested in knowing whether the garbage truck really missed someone’s bin, or whether the truck is being sent back because the resident actually forgot to put their bin out. Or they may be interested in monitoring local roads for defects that need to be repaired.

To scale the dash camera approach across multiple cameras, or to reproduce the research, it should be noted that the optical characteristics, resolution, and frame rate of each camera may differ, and placement of the camera in different positions, angles, and vehicles may materially affect the perspective of the images. These factors might result in the need to gather additional training and test images to ensure that the solution generalises well across cameras. If a model is being used to correct for lens distortion, each camera must go through its own calibration process, as per section 3.4.1.

The method that was used to quickly “bootstrap” a set of dash camera images for training and validation based on the Google Street View-trained model worked well in this instance. But the process of randomly splitting those images into “training” and “test” datasets would not have ensured the absolute independence of the “test” images. None of the “test” images were included in the “training” dataset, however some of the “training” images may have been very similar, due to them being taken from a nearby location. If the model had failed to generalise well despite good performance in its evaluation, this may have needed more attention.

It is likely that more “training” and “test” images would be required to get good results in inner metropolitan areas, but it was not possible to test this hypothesis.

The model that was applied to the problem of detecting paved shoulders had significant limitations. It was intended as a “proof of concept”, to show that the general technique used to map bicycle lanes from the bicycle lane markings could be applied to other problems where information can be collected visually.

The model relied on explicit programming of an algorithm, and manually selected thresholds applied to summary statistics based on observations in a training area. It did not attempt to apply any of the more recent advances in machine learning or deep learning, where the model “learns” to solve the problem for itself from the data. It was able to create an approximate map, without false positives, but the detected routes were often interrupted by parked cars or roadside “noise”.

These results were achieved by using training data from a suburb that was adjacent to the survey area. They most likely shared many characteristics. It might be difficult or impossible to tune the model to work well across a wider variety of settings.

The application of the Canny-Hough approach to lane detection across a variety of environments is challenging, and could be considered a separate research project in itself. To get a more consistent detection of the road boundary in the face of road-side “noise”, it would be worth exploring deep learning image segmentation techniques to train a model that can detect the road surface, similar to the approach taken by Mamidala et al., 2019 [38]. A deep learning model could also be trained to detect parked cars, and take them into account: A parked car should not be allowed to interrupt a detected paved shoulder route, but if there are many parked cars on the shoulder, is it worth recording the route at all?

COVID-19 lockdown restrictions imposed significant constraints on the location and variety of survey areas. Access to dash camera footage was especially limited. For the Google Street View approach, the restrictions constrained the areas for which an absolute “ground truth” could be established via an in-person survey. Further work would be required, once lockdown restrictions are eased, to show how well the dash camera solutions work in other environments.

During this research, surveys were conducted at a scale of one suburb at a time. OpenStreetMap data for an area was parsed and then cached in memory for quick access. It seemed reasonable to partition the work by suburb or “Local Government Area”, and the methods used worked well at that scale. Some operations might not scale well if they were applied to a whole state or country at a time. For example, for every point sampled in dash camera footage, we find the closest “way” and then the closest “node” in OpenStreetMap, to align the data to the OpenStreetMap version of the shape of each road. This allows us to identify and measure any differences in routes. If the process were applied at a state level instead of a suburb level, these searches might take significantly longer.