Technology Developments in Touch-Based Accessible Graphics: A Systematic Review of Research 2010-2020

Accessibility guidelines recommend the use of tactile graphics for presenting graphics, such as maps or charts, in which spatial relationships are important (of North America, 2010; Center, [n.d.]). Tactile graphics, however, require specialist equipment to produce and, as specialist transcribers are typically required, are expensive and time consuming to produce. Furthermore, the reader needs sufficient tactile reading skills, including the ability to read braille, to best understand the graphic content.

The widespread use of touch screen technologies, the arrival of low-cost maker technologies, and developments in computer vision and machine learning that can facilitate automated transcription, have opened up opportunities to take the provision of touch-based graphics into a new era. As such, new techniques for the creation of accessible graphics is a significant area of research. Key areas of research include new approaches to 3D-printed tactile models  (Carfagni et al., 2012; Grice et al., 2015; Holloway et al., 2018; Stangl et al., 2015; Teshima et al., 2010), haptic feedback (Darrah, 2013; Gay et al., 2018; Palani and Giudice, 2017; Palani et al., 2020; Tennison and Gorlewicz, [n.d.]), robotic systems  (Guinness et al., 2019), touch-triggered auditory feedback (Coughlan et al., 2020; Reichinger et al., [n.d.]; Shimada et al., 2010), and automated tactile generation (Wang et al., 2009; Chen et al., 2014; León and Martini, 2015; Taguchi et al., 2012), such as utilising online mapping systems (Götzelmann and Pavkovic, 2014; Taylor et al., 2015; Zeng et al., 2015).

Given the rapid rate at which new research is being conducted in the use of these new technologies for touch-based graphics, it is timely to review the field in order to better understand the nature of this research, and to identify best practice to better inform research in this context.

Systematic Literature Reviews (SLRs) are conducted across many different research fields. They are undertaken by researchers in order to examine a research field more holistically by understanding its recent areas of focus, the gaps, and looking to the future of the field. When conducted in a rigorous manner they provide a vehicle by which a research field can reflect upon and improve its best practice.

While most research is underpinned by a review of related work, it is often very targeted and typically self-serving to the research being presented. SLRs seek to be broader in scope, synthesising “existing work in a manner that is fair and seen to be fair” (Kitchenham and Charters, 2007), as well as remove bias (Mulrow, 1994). Not only can they provide a more complete picture of the type of work being undertaken, they can also examine a breadth of settings, method and contexts (Kitchenham and Charters, 2007; Brereton et al., 2007). Thus they allow us to understand the current state of the field in a more holistic manner.

Further to this, SLRs capture a point in time and reveal what has led the research field to this point. This is important in technology-related fields, such as assistive technologies, where advances in technology and techniques have a large influence on the direction that a research field takes.

Finally, SLRs allow us to identify the key challenges moving forward, regarding both topic and method. They not only identify the current state of the art, but also the gaps that are yet to be addressed and provide data for rational decision making (Mulrow, 1994). They can be influential in guiding the research agenda for the years to follow. As such, “The aim of an SLR is not just to aggregate all existing evidence on a research question; it is also intended to support the development of evidence-based guidelines for practitioners.” (Kitchenham et al., 2009). For example, the work of Barkhuus and Rode (Barkhuus and Rode, 2007) presents a systematically derived set of accepted evaluation methods in the CHI community and demonstrates how they are important to specific influential research communities.

There is evidence of the benefit of systematic reviews within the domain of assistive technologies. Kelly and Smith (Kelly and Smith, 2011), in their multi-decade review found that, while of benefit to the professional community, most work lacked methodological rigour. Wabiński and Moscicka (Wabiński and Mościcka, 2019) analysed works relating to automatic tactile map production and found that very few of the working solutions reported ever made it to widespread adoption. More recently, Brulé et. al. (Brulé et al., 2020) sought to better understand the nature of empirical evaluation of assistive technologies for BLV users, and found that meaningful participation by BLV end users in evaluation was lacking, as well as a disproportionately low involvement of low vision participants relative to their proportion in the BLV community. Other reviews focus on a subset of work, such as the development of interactive maps (Zeng et al., 2015; Ducasse et al., 2018), and highlight research questions specific to that context. These works demonstrate that systematic reviews can shine a light on key issues and provoke the assistive technologies research community to reflect and improve on their practice.

This work focused on a number of questions regarding research publications in touch-based graphics over the past decade. These are consistent with other SLRs, whereby it is the intention to capture a snapshot of current practice, identify gaps, and foreshadow the future of research in the area. Key questions are:

• •

  What are the main types of graphics and application domains studied in research exploring touch-based graphics?

• •

  What are the primary presentation technologies and how have they changed over the past decade?

• •

  What methodological approaches are used and how inclusive are they of the end users?

• •

  What are key gaps or challenges that researchers should consider moving forward?

This SLR aims to cover work relating to technological developments in touch-based graphics for people who are blind or have low vision (BLV). On this basis, the research team documented a number of key inclusion/exclusion examples that informed the selection of works. Notable inclusion criteria required that:

• •

  The work must relate primarily to novel touch-based graphics representations. In this context novelty refers to focus on a new production or presentation approach. A broad interpretation of ‘new’ was adopted when considering work for inclusion;

• •

  Auditory work could be considered if a touch interaction strategy was the basis;

• •

  Traditional touch-based graphics, such as tactile graphics, are only included if part of a comparative study with new technologies or if exploring new production techniques. Papers solely describing the use of tactile graphics were not regarded as ‘novel’.

Key exclusion examples were:

• •

  Blind or low vision users are not an identified focus of the work;

• •

  Primarily focusing on braille text;

• •

  Exclusive focus on user interfaces;

• •

  Audio interfaces with no touch interaction.

A ten-year period was chosen for the review, rather than a longer period. This was for two primary reasons. Firstly, a large volume of relevant papers was identified in this period (over 350). It was also felt the last decade covered the emergence of topical new technologies such as 3D printing.

In order to identify relevant work, multiple search approaches were used to reduce publication bias (Kitchenham and Charters, 2007). A systematic review of ‘gold standard’ venues was combined with a keyword search. Gold standard venues were those that fell into one of the following categories:

• •

  Research publication venues with known interest in assistive technology, i.e.: ACM SIGACCESS Conference on Computers and Accessibility (ASSETS); ACM CHI Conference on Human Factors in Computing Systems (CHI); International Conference on Computers Helping People with Special Needs (ICCHP); PErvasive Technologies Related to Assistive Environments (PETRA); ACM Transactions on Accessible Computing (TACCESS); International Conference on Tangible, Embedded and Embodied Interaction (TEI); ACM Transactions on Computer-Human Interaction (TOCHI); and ACM Symposium on User Interface Software and Technology (UIST). These include those recognised as top-tier venues by (Brulé et al., 2020).

• •

  Industry journals: Journal of Visual Impairment and Blindness (JVIB); Journal of Blindness Innovation and Research (JBIR); and British Journal of Visual Impairment (BJVI).

All works in these venues were reviewed, using titles and abstracts as the primary guide. As per (Kitchenham and Charters, 2007), a liberal and inclusive approach was taken, knowing that works could be removed at a later stage of the process. All works in the venues were considered, including full papers, short papers, late-breaking works, etc.

In order to identify work published in other venues, a keyword search was also undertaken. This could capture work presented outside of venues specifically associated with assistive technologies. A Google Scholar advanced search was conducted to identify articles between 2010 and 2020, using the following keywords:

• •

  At least one of: disability; accessibility; blind; low vision;

• •

  Plus at least one of: tactile; touch; haptic;

• •

  Plus at least one of: graphic; diagram; map.

Rather than search specific databases e.g. ACM/IEEE, Google Scholar was used as it includes these databases. We did not use exact phrases in the search, thus the search results included synonyms. We reviewed the first 100 search results, whereby no new relevant papers were found on the next results page. After conducting the search using these two methods, 352 papers were included in the initial corpus.

Based on the identified aims of the SLR, the research team created a set of analysis criteria by which each paper would be reviewed. Each criteria was discussed by the entire team to ensure agreement of its meaning was reached. Criteria were presented in a Google Form, with one form submission made for each paper.

A small cross-section of papers were reviewed by the entire team to assess the appropriateness of the analysis criteria. The research team then met to discuss the outcomes of the preliminary review, during which a revised and expanded set of analysis criteria was developed. Using a revised Google Form to capture data, all papers were reviewed against the new criteria. Results were captured for analysis in an accompanying Google Sheet. The following criteria were analysed: contributions; graphics and their application domain; presentation technology; production methods; research methodology; evaluation methods and participation; and distribution of technology outcomes.

All 352 papers were fully reviewed. During this process each paper was also re-assessed for relevance (Kitchenham and Charters, 2007) and could be excluded from the corpus if it was deemed to be outside the scope of the literature review, as based on the initial inclusion criteria. If a reviewer believed a paper should be excluded, another reviewer would also analyse the paper. If both believed the paper should be excluded then it was formally removed from the data set. If there was disagreement, a third reviewer would make the final decision. This ensured reliability of the inclusions and exclusions (Kitchenham and Charters, 2007). Examples of papers that were excluded at this stage include those where only widely adopted production and presentation methods with tactile diagrams were used, work where the context of application was not identified as being for BLV end users, and papers where the work only related to accessibility of user interfaces and not graphical information.

To support the authors as well as the wider touch-based graphics research community, the corpus of identified documents has been catalogued in a publicly accessible web format, based on the interactive sentiment visualisation tool SentimentViz (Kucher et al., 2018). Like SentimentViz, our tool - AGRep (Accessible Graphics Repository) - is designed to help users to “discover interesting patterns and facilitate data exploration” (Kucher et al., 2018).

Using AGRep (Figure 1) it is possible to explore and perform custom searches on the corpus of resources based on our categorisation markup, access persistent identifiers or URLs for each resource, and view tabulated summary data. Additionally, as AGRep is envisioned as a living resource capturing what has been done in the field of accessible graphics technology, users can update the corpus with newly published resources in the future.

On completion of the full review, the final corpus contained 292 papers from 97 unique venues. Following are the basic characteristics of the corpus. Note that the largest body of work came from ICCHP. As this conference runs every second year, data is aggregated in two-year blocks, so as not to misrepresent alternate years when ICCHP was not held.

Table 1 shows the distribution of papers over the decade. The number of papers can be seen to have significantly risen in 2014. A possible reason for this will be discussed when we examine paper content later in this review.

The corpus was drawn from 97 unique venues. As can be seen in Table 2, the vast majority of venues from which papers were chosen (70.41%) were represented by only one paper in the corpus. Only 4 venues had on average one or more paper per year. These were: ICCHP (61 papers); ASSETS (44 papers); CHI (35 papers); and TACCESS (10 papers).

The majority of papers collected were classified as ‘Full Papers’ by their venue (Table 3). There were also a number that were considered as ‘Short Papers’ or were an accompanying paper to a poster or demonstration. Short papers typically do not have the same expectation of rigour regarding evaluation, and include such things as preliminary studies or late-breaking work. The majority of short, poster and demo papers were in the top three venues (ICCHP, ASSETS and CHI) with a little under half being considered ‘Full Papers’ in those specific venues.

Across the 292 papers, there were 641 unique authors. All authors were counted in the metrics regardless of the author order and weightings were not applied to authorship. The large number of authors is consistent with the finding that there are a very high number of venues with only one paper (Table 2). Table 4 shows that 488 authors (76.13%) contributed to only one paper in the corpus, i.e. have only one work relating to touch-based graphics in the last 10 years. This suggests that a significant amount of research is a ‘one-off’ by authors: they explore a single idea regarding an innovation with touch-based graphics and do not further develop or evaluate it. In accord with this, very few authors have multiple publications. Only 32 authors (5%) have 5 or more papers published relating to touch-based graphics over the past decade.

The long tail in both venues and authors shows little evidence of ‘deep dives’ into a particular research direction by researchers in the field. While this is somewhat speculative, it raises the concern that most authors do not have a deep knowledge of what others have done in the field and are possibly ‘reinventing the wheel’. It also raises the question of whether one-off research will lead to meaningful change in BLV community practice. On the other hand, it may be that to effect change in this area, many different approaches need to be tried to ‘see what sticks’. Indeed, this was illustrated clearly in the Transforming Braille Project, initiated in 2011 to develop a low-cost refreshable braille display and bring it to market, whereby over 60 projects were examined before leading to a commercialised product (Skutchan, 2016). This does not, however, stop us considering the question of how work in this domain can be better shared and built upon to effect meaningful change more quickly.

Graphics and their application domain provide an insight into the primary use cases of the research undertaken within the corpus.

Three categories dominated the types of graphics represented in the corpus: STEM related (115); Maps and Plans (110); and General or Unspecified (104). STEM related graphics encapsulated a number of other sub-types, including those relating to math, science and technology. “General or Unspecified” was a catch-all for graphics that were not explicitly defined or could be applied across domains. These included those represented by work exploring general presentation systems. Graphics captured by “Other” include: Puzzles and Games (4); Screen Layout (2); and Patterns (1) and Handwriting (1).

The focus on STEM related graphics and maps and plans is to be expected as it reflects that the main application areas for tactile graphics are in education and Orientation and Mobility (O&M) training. Maps and plans also lend themselves to technology driven innovations. For example, they are a strong use case for 3D printing, audio labelling, and in particular automated creation, where open data is easily available.

“Arts and Culture” is the other main category of graphic found in the corpus. This is somewhat surprising as they are not a typical use case for the provision of tactile graphics. It is exciting to see that researchers are exploring innovation outside of the more ‘obvious’ application areas of tactile graphics.

We also examined if the type of graphic had varied over the past decade. We found there was no significant change in focus. The only exception being a small increase in graphics relating to Arts and Culture in the latter years.

The intended application domain of the research publications (Figure 6) is consistent with the types of graphics they consider. Education and O&M dominate, which is reflective of the the primary application areas for tactile diagrams. The category of “Other” only captured one other use case, being for research.

Next we considered the area in which a paper made its primary contribution (Table 7). A paper could make more than one contribution.

The majority of works collected were those with a new application of existing technologies or techniques. That is not unexpected, and reflects that many works came from publication venues outside those related specifically to assistive technologies. Those works often explored the provision of touch-based graphics in a new context, for example the arts, cartography, etc.

Behind this, three contributions shared the majority of focus of other papers. These were new presentation methods, new production methods, and new interaction methods. It is significant and positive to see that researchers are not simply focusing on new presentation methods for touch-based graphics but are also devoting equal attention to their creation and the ways in which the reader can interact with them.

Within new production techniques, the dominant topic is the automation of accessible graphic creation, especially within the context of maps and plans. This is very important, as the timely and cost effective production of accessible graphics is a significant barrier to their more widespread use, especially in contexts outside of education and O&M. As such it is heartening to see the strong interest by researchers but it remains an area that would benefit from increased attention.

Other contributions that were captured by papers in the corpus included: reviews of technology (6); development or evaluation of guidelines or standards (5); exploration of design spaces and approaches (5); feasibility or usability studies (3); and understanding of tactile perception (2).

We also evaluated whether a paper was an improvement of an existing production technique or presentation technology, i.e. if it was heavily based on a preceding piece of research. While this is a more subjective measure than our other measures we believe it is still suggestive. Within the corpus, 38 studies were considered to be focused on improving existing work. This relatively low number is consistent with the ‘long tails’ of both authors and venues as presented in section 4.1, and again should be an area of reflection for researchers.

Table 9 identifies the presentation technologies used in the corpus. These technologies can be combined. The ‘base’ presentation technologies which provide information about the spatial layout of the graphic are, in descending order of occurrence: tactile graphics; 3D models; touch screens; refreshable tactile displays; and force feedback devices. These base technologies were frequently combined with the other presentation technologies. In the case of 3D models, audio labels were common while touch screens also provided audio labels and sometimes tactile overlays and/or vibratory feedback.

It may be surprising that tactile graphics, a well established technology, features so prominently. This reflects that they are currently the ‘gold standard’ for accessible graphics provision and so are a frequent benchmark for comparing new presentation technologies or are the focus of new production methods.

We were particularly interested to understand the level of involvement of end users in the research. While BLV people are the primary end users, end users also include sector professionals such as transcribers or O&M trainers.

Table 10 shows the involvement of participants in each research study, across the design, prototype and evaluation stages of each study. The participants are divided into three categories: BLV, Professional and Sighted. Note that participants might take part across multiple stages within a single study. Similarly, members from multiple participant groups may be used within a single stage within a study (e.g. both blind and low vision participants within an evaluation).

Importantly, BLV participants are by far the most represented group, with the strongest representation by blind participants in the evaluation stage. One thing that is noticeable is that BLV participants, however, are much less likely to be involved in the design or prototyping stages than the final evaluation. This is potentially a serious problem as it means end user feedback is only considered late in the development. The following section explores BLV representation further.

As previously stated, commercial work was not considered as part of the literature review. However, in order to gauge the potential ongoing impact of the research in the corpus, the availability of its outcomes was also evaluated during analysis. This was based upon statements in the paper and so probably understates the actual availability of the research.

Table 13 shows the availability of the outcomes of each research paper in the corpus. Of the work where the nature of availability was clear, almost half the outcomes of the research were considered to be unavailable to the broader research community or end user, due to the research being undertaken with a custom prototype. Close to 40% of the outcomes are available in some form, whether it be through the use of commodity technologies or with the outcomes being explicitly made available through open source or commercialisation.

It is encouraging that 81 papers did use commodity technologies. However, it is also acknowledged that there is significant prototyping being undertaken utilising low cost technologies (such as 3D printing, simple electronics, or simple vision systems). This may signal a shift toward research that will not require significant cost to become more widely adopted. As such, continued work should strive to balance innovation with widely adopted commodity technologies as well as with low cost development technologies in order to lead to impact in the BLV community.