Using Computer Simulations to Investigate the Potential
Performance of ‘A to B’ Routing Systems for People with Mobility Impairments

The ‘A to B’ routing problem (sometimes called the sidewalk accessibility problem) involves finding the best route in the built environment to undertake a journey from one location (‘A’) to another (‘B’). It is distinct from the ‘within A/B’ problem which addresses the scenario as to whether a given location itself is accessible (as opposed to an accessible route to getting there in the first place). For many people with disabilities, pre-existing navigation tools are often inadequate, because they do not reliably accurately provide routes that are accessible to the end user (Gupta et al., 2020; Tannert et al., 2019). In the real world, accessibility barriers often permeate the built environment, with a recent study reporting averages of over 1 accessibility problem for every 100m travelled in some cities (Froehlich et al., 2020), whilst another investigation showed one US city had 80% of curbs that were not ADA compliant (Bagenstos, 2020). Providing effective and accurate navigation tools for people with mobility impairments is thus an important yet difficult challenge (Froehlich et al., 2019), given the large number of pre-existing barriers.

For people with mobility impairments, taking any journey can be burdensome, especially if attempting to do so in a reasonable time, with strategies aimed at somewhat addressing this including ‘reccies’ to avoid barriers (Hara et al., 2016), or avoiding travelling as a pedestrian where possible (Bromley et al., 2007). The overall effect is to “depriv[e] disabled individuals of social opportunities (because they cannot visit friends, family, and places of entertainment or recreation), and [often cause] serious physical injuries when wheelchair users tip over obstacles or are hit by cars when forced to travel in the streets” (Bagenstos, 2020). Other examples include longer term health damage due to taking suboptimal routes, e.g. to manual wheelchair users by way of cause damaging whole-body vibration (Bowtell, 2015; Rice et al., 2018), or in using inappropriately designed ramps that require undue amounts of force to access (Velho et al., 2016). At the same time the range of barriers (and thus implications) is diverse, with many different types of concerns being overlooked or underemphasized in existing studies (Gupta et al., 2020). This leads to two types of concerns in respect of accessible navigation: (i) barrier avoidance (aimed at ‘barrier-free’ routes, e.g. as in (Harriehausen-Mühlbauer and Roth, 2016; Kamaldin et al., 2019; Wu et al., 2020)) and (ii) a ‘quality aware’ approach (Siriaraya et al., 2020), which simply seeks to minimize the number of barriers encountered.

An effective navigation tool needs to be informed by accurate information about the accessibility of the built environment. This is easier said than done: indeed, even defining accessibility can be challenging (Church and Marston, 2003; Saha et al., 2021) and there is a wide range of different types of physical barriers (whose effects in turn depend on an individual’s specific disability). Nevertheless, there are various types of accessibility documentation systems that have been designed in an effort to provide this type of information, typically by identifying specific types of physical features. Some of these approaches are human-driven and rely on individuals to collect the relevant information. One of these approaches is to use experts, however, whilst sufficiently accurate, this approach is expensive and results in very limited coverage due to these exercises being “laborious and time consuming” (Froehlich et al., 2019), meaning that most cities do not actually have anything approaching an accurate map of accessibility barriers (Bagenstos, 2020; Bromley et al., 2007). Another strategy that has been tried is ‘geocrowdsourcing’ (Rice et al., 2018) yet this has encountered significant problems in respect of their ability to accurately document barriers (Rice et al., 2018) and has also lacked effective engagement and thus coverage (Froehlich et al., 2019; Mascetti et al., 2020).

Given the limitations of human-driven approaches, a growing body of accessibility documentation research has focused on using automated or semi-automated approaches (e.g. (Hara et al., 2014; Kirkham et al., 2017; Saha et al., 2019; Weld et al., 2019)), with (Lange et al., 2021) providing a body of principles for designing appropriate automated accessibility documentation systems. To date, the most successful automated example is based on Project Sidewalk data (Saha et al., 2019), which used computer vision (and RESNET) to achieve a f-score of 0.85 in a best case scenario (Weld et al., 2019) (albeit with some variation across classes, with weaker performance on ‘surface problems’). Taking into account the difficulty of documentation this is a strong recognition performance, especially given the variety of forms that physical barriers can take, and the disagreement amongst people with mobility impairments as to the real-world impact of given barriers on accessibility (Kirkham et al., 2017).

We also note that there are numerous papers that claim to offer a viable system for using inertial sensors mounted on mobility aids (e.g. wheelchairs) to automatically measure the accessibility of the built environment (Iwasawa et al., 2015b; Iwasawa and Yairi, 2012; Iwasawa et al., 2016; Kurauchi et al., 2019; Yairi et al., 2019). Unfortunately, the evaluations of these initial sensor based systems are typically problematic in two different ways: they are not naturalistic (or ecologically valid) which means that the study’s findings do not translate to realistic activity recognition problems (Poppe et al., 2007) and they fail to use the appropriate leave-one-out metric of evaluation (Hammerla and Plötz, 2015) (in some cases, participants were left out, but they all followed the same route, which is a different variant of the same mistake). The recent work of (Mascetti et al., 2020) showed that there is a stark difference in reported performance between a leave one out approach and a k-fold evaluation in the context of obstacle detection (with a f-score 0.616 for leave one out and 0.84 for the inappropriate k-fold evaluation). Accordingly, whilst there are systems that purport to have an f-score above 0.9, these systems have not been shown to have a realistic chance of working effectively in the real world.

This paper is also a different take on the meaningful performance metrics for activity recognition problem, as existing documentation tools are often activity recognition systems (where the ‘activity’ is some sort of interaction with a potential barrier). At present, we are unaware of any cases where the performance of these automatic documentation systems has been directly connected to real navigation tasks. There is an important gap, in that there are no current performance metrics that convert the documentation performance into navigation performance. More generally, the determination of appropriate performance metrics is a matter of active investigation in respect of activity recognition (Hammerla and Plötz, 2015; Ward et al., 2011). Our work accordingly examines how we can translate documentation performance into navigation performance, using computer simulations to do so.

At the same time, the meaningful metrics problem is also of concern in respect of ‘fair AI’ more generally. Whilst Fair AI has primarily focused on the protected attributes of gender and ethnicity, the need to ensure that people with disabilities has been mostly overlooked, being “largely omitted from the AI-bias conversation” (Whittaker et al., 2019). In particular, there is a real need to ensure that the diverse range of considerations in respect of disability are considered: the variety of disabilities makes training a AI system based on machine learning more challenging (Morris, 2020), whilst meaning that inherent biases are less likely to be detected. In the context of accessibility documentation tools, this might arise by inadvertently taking a narrow view of disability, and thus excluding subtle instances of accessibility barriers. If an accessibility documentation system is reliant on using the motion of a disabled person, then a particular impairment (e.g. an unusual gait) may also reduce recognition performance (Morris, 2020), and thus the reliability of documentation. Understanding the real world implications of a given systems recognition performance is thus an important consideration for ensuring people with disabilities are not disadvantaged by the use of AI systems, including to document inaccessibility in the built environment: indeed it is very similar to design engineering, which has also been recently advanced for more effective technology design for people with disabilities by conducting simulations (Kristensson et al., 2020).

An end-to-end documentation-routing system involves three different interrelated stages:

1. (1)

   The documentation of the built environment and whether a given path segment is accessible.

2. (2)

   The design of a routing algorithm that proposed the most appropriate routes (and indicates its confidence in its own predictions)

3. (3)

   An interface that sits on top of the routing algorithm so the user can apply the results to navigate from ‘A to B’

The difficulty is that these steps have been siloed from one another, when in reality they are closely interconnected. A routing algorithm can only operate on the accessibility information it is given. And the perceived performance of a navigation ‘interface’ in turn depends on how good the results provided by its underlying algorithm (as well as how that information is presented). At the same time, better documentation can also come from the reporting of system errors and mistakes by end users of navigation systems: but only if they are satisfied enough with a navigation system to engage with it. The result (Figure 1) is that these navigation and documentation challenges are heavily interconnected: nevertheless, in practice, they are studied in silos that need to be brought together. Notably, user studies deal with only the interface itself, which is several layers removed from the underlying accessibility documentation system.

Adding to that complexity is the inherently person-specific nature of the accessibility documentation problem: different people have different disabilities and personal circumstances, and thus different needs and expectations. Yet as explained in (Gupta et al., 2020), there is a tendency for navigation support approaches to be focused on one disability group, even though there are a wide range of impairments that a fully inclusive end-to-end navigation system needs to support. Although existing navigation systems often assume otherwise, different disabled people can have different objectives: one user might be interested in purely avoiding barriers (and thus is concerned with ‘barrier avoidance’ approach), whereas another might be satisfied with simply encountering them appreciably less often (i.e. the ‘quality aware’ routing approach), as these barriers are discomforts, rather than hard-edged dangers or obstacles. As such, any framework needs to be suitably abstract and flexible to deal with a broad range of navigation concerns and objectives: there is no fixed optimal solution given the diversity amongst disabled people. At the same time, the output representation also has to be optimized to suit the preferences and needs of end users, but determining how to best do this is tricky if studies are confounded by errors in the underlying accessibility documentation tool.

A real world understanding of what an accessibility documentation tool can accomplish is needed to inform the most optimum approach for designing navigation tools. In effect, this is a ‘design engineering’ [24] problem, wherein simulations can help determine the most appropriate form of system. The distinction is that we are using these simulations to create real-world performance metrics, that provide a clear illustration of the navigation performance of a system based on a documentation tool with a given recognition performance. These metrics in turn describe the bounds on what can be accomplished with a navigation system in the real world, thus helping to determine the best strategy for using the information available to it based on an imperfect documentation of the built environment. In turn, these answers inform expectations for any feedback loop that might improve performance.

To connect the performance of an accessibility documentation system to a navigation system that relies upon the information provided by it, we conducted extensive Monte-Carlo simulations, with over 460 million runs (or simulated journeys) in total. Our simulations were constructed to focus on the cases where a person is seeking to engage in barrier avoidance, as this is the stated goal of a substantial proportion of work that focusses on accessible navigation, with a view towards investigating the viability of this approach. A map contains a set of edges (or path segments) connected at nodes (or junctions): a proportion of these edges/path segments will have accessibility barriers (Figure 2). The Inaccessibility Rate is the proportion of edges that are inaccessible in our simulation for our hypothetical person with a mobility impairment. Our starting point is that an accessibility documentation system has both a True Positive Rate (TPR) which reflects the proportion of edges containing accessibility barriers (for that person) that it accurately detects, and a True Negative Rate (TNR), which indicates the proportion of accessible edges it correctly labels as being such. The Inaccessible Rate, the TNR and the TPR are all weighted (linearly) by distance, so that longer segments are more likely to have an accessibility barrier. On a map, this translates into marking a specific list of edges as being inaccessible from the perspective of the documentation tool – unless the tool has a perfect TPR and TNR, then this perspective will sometimes be inaccurate, potentially leading to the person in question being routed inappropriately (see Figure 3 for examples of this and the different types of difficulties it can cause). A navigation tool will then automatically choose the shortest route based on the barriers identified by the documentation tool.

To enable us to simulate such a large number of journeys, we structured our simulations with a bespoke design that enabled computational efficiency (Figure 4), developing our own novel approach for doing this based on ablation and pre-computation. After extracting regions from OpenStreetMap, we then generated a stratified sample (by distance) of different location pairs (‘A’ and ‘B’) up to a ‘crow flies’ distance of 1.2km (very short routes less than 300m were excluded): we selected 60 journeys in total. For each journey, we then pre-computed a list of all possible routes (and the distance) between ‘A’ to ‘B’ up to a limit of 1.5km total edge length using the CPU’s on an HPC cluster. Each simulated journey involved generating a list of inaccessible segments on the map. We then generated a list of segments actually determined to be inaccessible by the documentation tool (based on the information available to it). The route to be followed was then computed using an ablation approach on a GPU (with a speed increase of around 1000 compared to a single CPU core). This ablation approach involves identifying all the edges perceived as being inaccessible and deleting them from the network, which computationally means deleting the precomputed routes that contain one or more such edges and selecting the shortest remaining route, if it exists¹¹1If no route exists, then the system would report it is not possible to complete the journey. (the deletion is a matrix operation that can be done rapidly on the GPU). Then the relevant performance metrics, including whether an accessibility barrier was counted and if the system sent a person on an unduly long route were computed based on the selected route, thus enabling a report on the different types of errors and in turn, a cost benefit analysis to be conducted.

We first present our simulation results concerning an ideal documentation tool, which has perfect knowledge of each and every accessibility barrier, as these places a limit of how much benefit can be gained from any documentation tool. Figure 5(i) illustrates the influence of the inaccessibility rate, whilst Figure 5(ii) illustrates the increase in distance required for the remaining routes. As can be seen, a city that has an increase by even a moderate volume of accessibility barriers offers a severely impaired navigation experience. There is also a (relatively small) difference between different cities in our ‘bag of routes’, with our simulation of Prato being the most challenging, and Canberra being somewhat less affected by an increased inaccessibility rate (this difference is limited is despite the fact that these cities have markedly different structures, especially in terms of numbers of intersections per node⁵⁵5Prato likely performs worse overall due to the relatively reduced number nodes per intersection and its lower street density. Further investigation (beyond the scope of this paper) is needed to indicate the full implications of geometry on the relative (in)accessibility of a city.). These results clearly justify our focus on routes with a relatively low proportion of accessibility barriers, as with an inaccessibility rate above 0.15, it is unlikely that even a perfect tool will be able to offer a sufficiently useful navigation experience, given the increase in distance and (more markedly) the proportion of routes that are unavailable.

We consider that a route is impassible if there is no possible way to travel from A to B without interacting with at least one accessibility barrier, otherwise we call it navigable. A perfect tool (where $TPR=TNR=1$) will always accurately report on impassible routes. However, a tool that is inaccurate can wrongly report routes to be navigable, when they are not, or vice versa. Figure 6(i) illustrates the effect of the TNR upon navigable routes that are falsely reported to be impassible in the case where the $TPR=1$. As can be seen, a high TNR is necessary if a large proportion of accessible routes are not to be wrong reported as being impassible. In a realistic scenario of an inaccessibility rate of 0.2, a TNR of 0.95 would still wrongly report nearly 30% of all accessible routes as being impassible, whilst a TNR of 0.9 would lead to over 50% of routes being falsely reported as such. Even in environments where there are no accessibility barriers at all (i.e. an inaccessibility rate of zero), a substantial proportion of routes can falsely be reported as being impassible, even with respect to relatively high TNR’s (e.g. at a TNR=0.9, nearly 15% of routes are falsely reported as being impassible).

A somewhat lesser problem is a risk of false positives, namely routes are declared to be navigable when in fact they are not. As can be seen in Figure 6(ii), a lower TPR can lead to this issue occurring quite often: for instance, at a TPR of 0.85 and an inaccessibility rate of 0.2, this happens nearly 20% of the time. It is also worth noting that the inaccessibility rate is also a particularly influential factor, with the increase being non-linear, and this occurrence being rare at lower inaccessibility rates, even with a relatively low TPR – for example with an inaccessibility rate of 0.08, and a TPR of 0.7, this still happens less than 10% of the time.

We now present the performance of a navigation tool for the cases where it reports a navigable route and the reported route is indeed navigable. We do this by applying the distance measure set out in Section 3.3.5, looking at the relative increase in distance relative to a perfect navigation tool and taking into account the accessibility barriers encountered. Noting that an ideal tool does not encounter accessibility barriers, this is as follows:

In that formula, $dist_{tool}$ is the distance of the route selected by the routing tool, $dist_{perfect}$ is that of the journey selected by an ideal tool (i.e. where $TNR=TPR=1$) and $nbarriers_{tool}$ is the number of edges with accessibility barriers in the journey proposed by the navigation tool. All distances are in meters. The resulting performance is illustrated by surface plots in Figure 7.⁶⁶6Note that the plots in Figures 7-9 are fitted surfaces generated using MATLAB ^TM’s Curve Fitting Toolbox (and fourth degree polynomials). It is clear that the higher the inaccessibility rate, the worse the given score as a proportion of distance. Both TNR and TPR have an impact on performance, however lower TNR’s have less of an impact (on routes already reported as being passible), with this being particularly clear when one compares charts with the lower inaccessibility rates. At higher inaccessibility rates, the impact of lower TPR’s (and TNR’s) is particularly pronounced, with many journeys being substantially increased in relative length. It can be seen from Figure 8(i) that much of this arises from barriers being interacted with, as opposed someone being simply sent on a longer route.

The previous three subsections considered the performance of a navigation tool versus an ideal routing tool. We now consider the performance of a routing tool with different TNR and TPR relative to an oblivious navigation tool (i.e. one with $TNR=1$, but $TPR=0$), for the cases where a navigation tool reported an accessible route (notably an oblivious tool will always report there being an accessible route). As can be seen from Figure 8(ii), even with relatively moderate inaccessibility rates, e.g. 0.1, the increase in accessibility barriers encountered on an average route is over 1 (i.e. on each journey, one would expect to encounter an accessibility barrier over and above that captured by the navigation tool): by contrast, even with a TPR of 0.7 and an inaccessibility rate of 0.2, one would not necessarily encounter a barrier any more than 50% of the time (per Figure 8(i)). Thus, even relatively poorly performing tools can have a substantial impact on barrier avoidance and provide a markedly better experience. A strongly performing tool would avoid barriers most of the time. It is also important to consider performance by way of distance. We compute a relative score as follows:

All the above have the same meaning as with the previous score, whilst $dist_{oblivious}$ is the distance of the route proposed by the oblivious tool (i.e. $TPR=0$,$TNR=1$), and $nbarriers_{oblivious}$ is the number of barriers encountered. The formula is weighted in proportion to the best possible performance. The result of the exercise is provided in Figure 9 (with negative distance increases being better than an oblivious system). Asides cases where there is a relatively poor TPR and TNR, the oblivious tool is always outperformed, even at low inaccessibility rates. At an inaccessibility rate of just over 0.08 or more, even with $TPR=TNR=0.7$, the performance is still better than the oblivious tool. In most cases with inaccessibility rates above 0.1, even todays systems (with a $TNR=TPR=0.8-0.85$) still greatly outperform the oblivious tool, thereby offering a markedly improved experience (albeit an imperfect one).

.

Our simulations give a clear indication of what can be achieved with a navigation-documentation tool, both with today’s performance, as well as likely future improvements in practice. The overall real-world effect of our simulations is as follows:

Barrier Avoidance is not realistic in today’s built environment: Our results show that barrier avoidance is unlikely to be a feasible approach towards providing a navigation tool, at least in today’s built environment. As such, the stated goal of barrier avoidance of many works in accessible navigation is practically impossible in today’s circumstances, even with a perfect documentation tool (and thus a fully informed navigation system). This is because even a moderately inaccessible city will lead to a significantly degraded navigation experience – a substantial proportion of journey’s will be impossible, and those journeys which remain possible will be a lot longer. For example, at what we expect is a relatively optimistic 20% (or 0.2) inaccessibility rate⁷⁷7It is highly likely for some groups, such as electric wheelchair users, that the inaccessibility rate is over 0.3 in most cities. See (Froehlich et al., 2020), which provides the number of each type of barrier per 100m: in their best case city (Seattle), there was 0.3 obstacles per 100m, and 0.6 surface problems per 100m. Given the average segment length in Seattle was 71.61m, even half these values would produce an inaccessibility rate well beyond 0.3 (i.e. $(0.3+0.6)\times 71.61/100=0.64)$., this leads to 42.6% of all journeys being inaccessible on average (i.e., there is no accessible route that can be taken at all), with an average journey distance increase of 15% on those remaining routes which are navigable. Even at an inaccessibility rate of 10% (an unlikely scenario), 13.25% of journeys are impossible on average, and the average journey distance increase for the remaining journeys is 7.5%, thus being far from being an ideal experience.

Barrier Minimization and ‘quality aware’ navigation is feasible: Against this context, we argue that the present goal of an accessibility navigation tool should be the more modest task of barrier minimization, rather than avoidance, which might well change if documentation becomes far more accurate (i.e., substantially above 0.95 for the TPR and TNR). Indeed, it is more realistic under normal circumstances – as can be seen, even a relatively poorly performing accessibility documentation tool can greatly improve upon an oblivious one, as long as the limitation is clear. An important implication for design flows from this: accessibility navigation tools should be expressly clear in that they are likely to be inaccurate, and promoted as reducing the chance accessibility barriers will be encountered. Once we design systems based on realistic expectations, we will have made tangible progress towards them being deployed in the day-to-day life of people with mobility impairments.

Better navigation would potentially be possible with a strategic effort to improve the built environment: We assume that accessibility barriers are randomly distributed. However, there are a few cities who do make a concerted and strategic effort to provide accessible spaces in the right places, which may produce better navigation performance for a given inaccessibility rate. This strategy would align with work already done by city planners in respect of public transport services (Ceder, 2016; Victor and Ponnuswamy, 2012) (and separately, the provision of cycling routes (Winters et al., 2013; Larsen et al., 2013)), who carefully configure the spatial layout of routes to optimise performance, suggesting a related approach could potentially work with accessibility as well. For instance, it might be that if some accessible ‘trunk routes’ were to be deliberately built and identified in advance (perhaps in a grid formation as suggested for public transport in (Ceder, 2016; Pemberton, 2020)), it would presumably be possible to improve substantially upon this performance, but in that case, there would be a lot of (expensive) building work required by a city or local council. Notably, the results of different approaches towards embedding accessibility in the built-environment can be simulated using our framework (by making the relevant edges always accessible and automatically detected by the documentation tool as being such) with a small modification, meaning these experiments can be done in the future.

The likely performance of a navigation tool across our range of scenarios set out in our results also has a range of important consequences on a human level, thereby leading to wider implications for the design of navigation systems for people with mobility impairments.

Navigation is not a panacea: Our results show that even providing a perfect solution to the documentation of the built environment does not solve the navigation problem (although it would help in identifying barriers to be addressed). Instead, the built environment needs to be made a lot more accessible, in order to provide a reasonable experience for people with many mobility impairments. There is a real risk that the capability of tools to provide barrier minimization and thus a substantially better experience for many people with mobility impairments, leads to fewer physical improvements being made. For example, barrier minimization would likely lead to a lot better experience for manual wheelchair users (perhaps even a nearly equal one to people without mobility impairments) who would then be less concerned about physical improvements, but it would not address the problem for most electric wheelchair users (for whom physical improvements are necessary to give them a reasonable experience). Given the complicated politics around disability and the built environment (see e.g. (Kirkham et al., 2017; Saha et al., 2021)), the risk is that the most disabled people are disadvantaged by navigation systems, by less resources being spent on fixing the built environment, thus marginalizing them even further.

Transparent and explainable navigation tools: By acting as implicit performance metrics, our simulations can be used to manage expectations to provide better navigation systems: this works by being clear as to the likely navigation performance of a given system, which can be done provided there are estimates available of key variables (i.e. the inaccessibility rate, as well as the TPR / TNR of a documentation tool). Specifically, someone can be informed of the chance or probability of encountering one or more accessibility barriers – for example, it would be possible to say ‘On average you might encounter 0.3 accessibility barriers on your journey taking this route’, or ‘Your journey will normally take between 15 and 25 minutes, considering the likelihood of having to re-route around accessibility barriers’. Being transparent in this manner may provide a better experience for end users of these systems (e.g. journey planning can better consider the risk of being misdirected), as well as ensuring that these systems are used with realistic expectations in mind, because the expected performance under given circumstances can be directly quantified, especially in cities when the rough inaccessibility rate for different types of mobility impairments is known.

Whilst the need for this has already been identified (see e.g. (Tannert et al., 2019)), we are unaware of any widely used accessible navigation system that presents options and routes in terms of risk. The simulations we have presented allow for this risk to be estimated and presented to end users in the context of a navigation system, thus overcoming this hurdle if the accuracy of the documentation tool can be estimated, likewise with the distribution of accessibility barriers in a city. By being upfront with people with mobility impairments, we can expect that in return they would participate in the reporting of barriers, thus enabling a tool to become gradually more accurate and informative as it is used, thereby closing this important feedback loop.

Studying Navigation Tools: Another implication of our results is that relatively small changes in inaccessibility rate can lead to markedly different user experiences (as the relationship is not linear). This has substantial implications for user experience studies of these tools. In particular, studies based on past experience (e.g. as with (Gupta et al., 2020; Hara et al., 2016)) of using existing tools present this difficulty, because it would be difficult to separate out the performance of the tool from the level of accessibility in the built environment. In the immediate term, this may mean that these investigations should be done in respect of controlled simulations (rather than real world studies), so the performance of the tool and the variance across locations can be disaggregated from the interface design problem. These studies are particularly important going forward, to help determine which types of error should be avoided and for which groups of people. For instance, some people might be less concerned about a tool falsely reporting a journey is impossible, than by being sent on a journey that contains accessibility barriers. The right balance to be struck will likely vary substantially across different populations of people with mobility impairments, as well as being governed somewhat by individual preferences.

The simulation approach that we have developed is potentially applicable in a wide range of circumstances, with some adaptions, making this work be of general import to the field of Mobile HCI beyond the accessible navigation. There are three main domains in which this work can be expanded.

Other accessibility navigation problems: The modelling approach that we have developed can also be applied to other accessibility related problems, with some modifications or different parameters. For example, a city planner could use this same approach to determine the impact of addressing specific accessibility barriers on the accessible navigation of their city (e.g. by setting purpose-built accessible edges/street segments to be always accessible and marked as such to a documentation tool, which would in turn be 100% accurate in identifying it), thus optimizing the use of scarce public resources to make accessibility improvements. Similarly, our approach could also be applied to optimize the performance of a navigation tool (especially in terms of trade-offs between TNR and TPR) to provide the best experience for individual users, depending on their preferences and needs.

With further expansion, it would also be possible to modify our approach to include user weights for different barriers: our simulations took the view that accessibility barriers should be always avoided (with the result of demonstrating the infeasibility of a barrier avoidance approach) and thus gave an infinite weight to an accessibility barrier, but this value can be changed. For instance, a manual wheelchair user might simply wish to somewhat reduce the amount of times they have to ‘curb hop’ due to a missing dropped curb, and would benefit from a system that gave a potential dropped curb a penalty (but a smaller one than automatically avoiding a given route) This would involve adding a distance weight (based on user preferences) for a perceived barrier of this nature and then picking the shortest route with this weight added. This approach can be expanded to include different weights for different types of barriers, depending on the person in question.

Navigation beyond accessibility: Navigation in the real world has been a long-standing topic of interest in the wider MobileHCI community, including the design of routing interfaces. The minimization of the risk of error is one such concern that arises in a range of circumstances, be it as-the-crow-flies navigation (Savino et al., 2020), missing GPS information (where there is a degraded signal) (Ranasinghe et al., 2019; Burigat and Chittaro, 2011), or the provision of ‘scenic’ or more comfortable routes (Johnson et al., 2017). These types of problems could also benefit from the identification of street elements that likely contain problems, be it a lack of a scenic feature, an area that is likely to have degraded mobile or GPS signals, or have other features which make navigation more challenging. Just as with accessibility barriers, there is likely to be imperfect information about these features, and thus a risk of encountering a suboptimal route. This is important for at least two reasons: first, a user can be given a choice of avoiding this risk, and secondly the risk of encountering a particular barrier can be quantified (which in turn could feed into interface design of a ‘quality aware’ navigation system (Siriaraya et al., 2020)). At the same time, the concept of an accessibility related feature can be a wide one if accessibility is considered in a holistic manner: for instance, some people seek to avoid areas of crime or poor weather (Johnson et al., 2017), which may be sometimes be for reasons arising from a mental health condition, or another impairment, but is not generally considered as the primary concern.

Monte-Carlo Simulations as Practical and Implicit Performance Metrics: Performance metrics are descriptive statistics used to summarise the result of a machine learning system, including a plethora of mobile sensing systems (especially activity trackers). Yet providing meaningful metrics is an important challenge, both for evaluating algorithms (by researchers) but also representing results to the public. Our simulation approach is important because it is distinct from the static and hard-edged performance metrics typically used for measuring classifier performance, and instead provides a practical measure of real-world performance focused on the relevant problem at hand. Given that accessibility documentation is similar to activity recognition problems (and indeed systems using inertial sensors and the motion of users to detect barriers are activity recognition systems) this is thus a new approach towards addressing the ‘performance metrics’ for activity recognition problem (see e.g.(Hammerla and Plötz, 2015; Ward et al., 2011) for a discussion of this problem).

As such, this work demonstrates how to use simulations to construct implicit performance metrics, that meaningfully translate static performance metrics into problem-focused ones. Whilst existing performance metrics are helpful when comparing the relative performance of algorithms, these simulations answer a different question: what performance is ‘good enough’ (in respect of a given real-world problem)? These implicit performance metrics are therefore an important complement to existing metrics traditionally used to measure classifier performance, and provide a platform for further exploration of real-world performance. This approach could in principle be adapted more widely across the domain of Mobile HCI, with Monte-Carlo simulations being used to provide information that is more relevant to end-users in respect of any system that depends on mobile-sensing.

Whilst our work clearly shows the benefits of using simulations to provide an indication of real-world performance, it should be observed that the simulations we have conducted are somewhat exploratory in nature. Our goal was to demonstrate some of the opportunities of following such an approach for the accessibility navigation/documentation problem, as well as illustrating how this can be practically implemented: this has been successful in raising a number of important concerns for the design of these systems and the advancement of this important research agenda. Nevertheless, given the complexity of the issue (and the simplifying assumptions we necessarily made), there are various parameter changes that potentially could lead to different results, including our choice of the bag of routes and city structures, our choice of navigation strategy, and our decision to constrain the length of routes, as well as smaller decisions in the design and implementation of our experiments. Our simulation assumed that the main objective was avoidance, with the effect that encounters with barriers were heavily penalized: some people with disabilities might be less demanding in their expectations (especially if their goal is to minimize discomfort). This means that there is perhaps an element of pessimism in our arguments, as a weaker tool might be useful for a minority of disabled people (even though this would be far away from the ambitions of the accessibility documentation agenda). Given the wide lacuna that this work fits into, we hope to see future investigations and simulations that explore alternative navigation methodologies and objectives.