Towards a Structural Framework
for Explicit Domain Knowledge in Visual Analytics

Since “Illuminating the Path” [40, p. 35], incorporating prior domain knowledge and “build[ing] knowledge structures” has been on VA’s agenda. This is underscored by the pivotal position of knowledge in the VA process model by Keim et al. [20, 19] and further process models such as the knowledge generation model by Sacha et al. [34] and the visualization model by van Wijk [42]. However, these process models do not differentiate between knowledge in the human space and in the machine space. Based on Wang et al. [47], Federico, Wagner et al. [15] delineate tacit knowledge that is exclusively available to human reasoning, from explicit knowledge that can be leveraged by the VA environment. How explicit knowledge is integrated into the VA process is formalized in several recent models by Wang et al. [47], Ribarsky et al. [32], and Federico, Wagner et al. [15].

Beyond the role of knowledge in the VA process, only few works discuss the content and structure of explicit knowledge on a general level. Andrienko et al. [4] conceptualize domain knowledge as a model of a part of reality and provide definitions for different types of models but they do not specify the form and medium how the model is represented. Schulz et al. [36] formalize data descriptors that include domain knowledge about data. Tominski [41] captures domain knowledge as event types that are specified using predicate logic. Lammarsch at al. [25] propose a data structure for knowledge about temporal patterns leveraging the structure of time. The generation of adapted visualizations which are based on ontological datasets and the specification of ontological mappings are treated by Falconer et al. [12]. Therefore, they use the ‘COGZ’ tool, converting ontological mappings in software transformation rules so that it describes a model which fits the visualization. A similar approach for adapted visualizations is also followed by Gilson et al. [17], describing a general system pipeline which combines ontology mapping and probabilistic reasoning techniques. Thereby, they describe the automated generation of visualizations of domain-specific data from the web. However, none of these approaches aim for a general framework.

The application of visualization techniques to healthcare has sparked a lot of interest to integrate knowledge. Already the early LifeLines [30] approach envisioned how domain knowledge is used to highlight relationships between events, which was then realized by a simple full-text search. A number of approaches such as Midgaard [5, 1] and QualizonGraph [13] enrich the display of time series of medical parameters with qualitative levels. Thus, a period of critical conditions can be detected and visually highlighted, for example by color. The ViTA-Lab environment [24] and its preceding work [23, 38] leverage complex temporal data abstractions for pattern discovery and provide a case study of longitudinal analysis of 22,000 diabetes patient records. The Five W’s [52] environment arranges events of the health record hierarchically based on the knowledge about diseases formalized in the ICD9 taxonomy. Gnaeus [14] is a guideline-based knowledge-assisted visualization of electronic health records for cohorts. Evidence-based clinical practice guidelines are sets of statements and recommendations used to improve health care by providing a trustworthy comparison of treatment options in terms of risks and benefits according to patient’s status. The KAVAGait [45] tool is a knowledge-assisted VA environment for clinical gait analysis that supports analysts during diagnosis and clinical decision making. Users can load, visualize and compare patient gait data containing ground reaction forces (GRF) measurements gaining new knowledge, identify unseen pattern and recognize connections. KAMAS [44] is a similar knowledge-assisted VA environment for analyzing event sequences and categorizing suspicious sequences according to a taxonomy of behaviors. Even though KAMAS was originally designed for IT-security analysts, comparable data structures and analysis problems are also relevant for healthcare (e.g., [46, 51]).

Summarizing these findings, it can be seen, that most of the discussed approaches cover how explicit domain knowledge can be exploited to enhance visual representation and data analysis; some approaches provide methods to generate explicit knowledge. Additionally, most of the currently implemented knowledge-assisted VA environments are focused on the integration of specific domain knowledge, which could only be used for precisely defined analysis tasks. In general, explicit knowledge is now a first-class artifact in the VA process but its form and structure are left unspecified. None of the presented approaches provides a structural framework for describing and storing explicit knowledge in VA environments. Thus, a structural framework is needed and combined with the theoretical process model by Federico, Wagner et al. [15] it would provide valuable generative guidelines for the development of novel knowledge-assisted VA environments.

The following desiderata for a structural model of explicit domain knowledge were established in the reflective discussions of two recent design study projects, in which our collaboration with domain experts resulted in a VA environment [2, 44, 45] and preceding design study work that only sketched the integration of explicit knowledge [33, 1, 13]. Additional inputs result from analysis of the scientific work cited above. In particular, we build upon the characterization of knowledge by Federico, Wagner et al. [15] with its three axes space, type, and origin. Their knowledge characterization, however, is only descriptive about what is possible and is used to categorize knowledge-assisted VA environments for their survey.

Overall, we envision explicit domain knowledge as a first-class artifact in VA process that is both an input and an output of VA activities [15, 25]. As analytics in the “wild” are seldom constrained to a single isolated VA environment or a single data backend [16], we regard it imperative to design a framework for explicit domain knowledge in a way to allow manifestation of knowledge in various data structures and backends as well as utilization in different VA environments.

The nine desiderata are:

1. D1:

   Explicit knowledge, by definition, resides in the machine space and is machine interpretable [15, 47]. Structured forms of knowledge representation allow the VA environment to reason about relationships within the knowledge (e.g., controlled vocabularies with same/different, taxonomies with hierarchical, or ontologies with custom relationships). While free text annotations or hand drawn polygons can capture knowledge in the machine space, they are, in their raw form, opaque to machine reasoning.

2. D2:

   The structural model should focus on domain knowledge, i.e. interpreting the data. In contrast, operational knowledge, i.e. effectively using the VA environment, is out of its scope. While the latter can be tackled by usable user interfaces, user onboarding, and automated visualization recommendation, the former is essential for the success analysis, either as explicit knowledge or as tacit knowledge of the domain experts as user [15].

3. D3:

   It should be possible to reuse pre-existing taxonomies or other knowledge artifacts that have been established within a community of practice.

4. D4:

   The structural framework should facilitate the exchange of explicit knowledge between different VA environments.

5. D5:

   The structural framework should be compatible with heterogeneous data structures and storage technologies.

6. D6:

   Runtime editing of explicit knowledge should be possible from within the VA environment. Thus, all the origins of knowledge sketched by Federico, Wagner et al. [15] should be supported: (i) knowledge artifacts can predate the VA environment (cp. D3); (ii) explicit knowledge can be prepared during the VA environment’s design process; (iii) a single user can interactively externalize domain knowledge; (iv) multiple users can externalize and share knowledge; or (v) knowledge can be automatically derived from data [15].

7. D7:

   In order to support provenance and accountability of knowledge generated, especially in the post-design phase, the framework should provide a standardized form to include authorship details and further provenance information.

8. D8:

   There should be good software library support to integrate explicit knowledge into the source code of a VA environment. Widely used development languages like JavaScript, Python, and Java should be supported.

9. D9:

   The notation of explicit knowledge should be easily readable and editable by knowledge engineers and visualization designers for development and debugging.

The last two desiderata (D8 and D9) might appear to be overly specific. Of course, it is possible to rely on custom software solutions and knowledge editors. However, our practical experience in designing knowledge-assisted VA environments underscored the importance of suitable software development support and easily debuggable notation to allow for the rapid feedback cycles typical in visualization design studies[27].

The desiderata collected above characterize a generalized structural framework that enables the communication and reuse of explicit domain knowledge across different KAVA environments (D4), different datasets (D5), and different users (D6). In order to separate these aspects clearly, we suggest a structural framework consisting of three components (Fig. 1).

1. 1.

   Concepts from the application domain and their relationships.

   Example (diabetes): the diagnosis ‘hyperglycemia’, i.e. high blood sugar.

   Example (physiotherapy): a gait abnormality of the knee during mid-stance phase.

2. 2.

   A mapping between concepts and the dataset(s) to manifest knowledge in the relevant data items.

   Example: a blood sugar level higher than 200 mg/dl is diagnosed as hyperglycemia.

3. 3.

   A mapping between concepts and the KAVA environment(s) to utilize the knowledge.

   Example (diabetes): visually represent period of hyperglycemia as a horizontal line.

   Example (physiotherapy): provide list of known gait patterns and highlight patient marks on selection.

The vocabulary of concepts will remain comparatively stable for most domain problems (Table 1). Therefore, knowledge about concepts can act as an anchor when linking additional datasets to the analysis. If these datasets are heterogeneously structured (D5), adaptations to the manifestation component will be needed. Likewise, if additional VA environments are used (D4), the knowledge utilization mapping may be adapted. All three knowledge components can be manipulated by analysts via interaction with the VA environment (D6), whereby we assume that most changes will affect the manifestation component.

Next, we demonstrate how the explicit knowledge of an exemplary VA environment can be structured according to our framework. This scenario is based on the KAVAGait [45] design study with physiotherapists and illustrated using the processes of the knowledge-assisted VA model [15, 43].

KAVAGait [45] is a ‘Knowledge-assisted VA System for Clinical Gait Analysis’, whereby the analysts (clinicians) are supported during analysis and clinical decision making (see Fig. 4). The analysts have the ability to load patient gait data containing time series of ground reaction force measurements for each foot. The time series are visualized as line plots in the center of the user interface, describing the force over time. Additionally, 16 spatio-temporal parameters (e.g., step time, stance time, cadence) related to the loaded patient’s gait are calculated, visualized, and used for automated patient comparison and categorization based on the introduced interactive twin box plots. One primary goal during clinical gait analysis is to assess whether a recorded gait measurement displays ‘normal gait’ behavior or if not, which specific ‘gait abnormality’ is present. Thus, the environment’s explicit knowledge store contains several categories of ‘gait abnormalities’ (relating to e.g., knee, hip, ankle) as well as a category including ‘healthy gait pattern’ data, used for analysis and comparison by default. Each category is described by the data of patients that were previously assigned to this category. In particular, the $[min,max]$ ranges of the patients’ 16 spatio-temporal parameters are calculated. Based on these category descriptions, automated data analysis of newly loaded patient data is provided (e.g., automatically calculated category matching). This automated data analysis supports the analysts in their interactive data exploration. To achieve a second goal, clinicians can generate new explicit knowledge by adding the analysis result to the explicit knowledge store. KAVAGait also provides the ability to interactively explore and adjust the internally stored explicit knowledge.

Assessment of Patient Data: If the analyst loads a gait analysis file $D$ of a patient into KAVAGait (see Fig. 5 for a graphical overview of the knowledge processes), first, the contained time series are visualized $V$ based on the systems specification $S$. Second, the stored explicit knowledge $K^{\epsilon}$ and the automated data analysis methods $A$ are strongly intertwined with all components of the VA environment. Thus, this pipeline immediately calculates the 16 spatio-temporal parameters based on the loaded time series and the matching to the different knowledge categories, affecting the specification. The combination of both former described procedures can be expressed as the initial analysis and visualization pipeline. However, if the specification is not influenced by the analyst (e.g., zooming, filtering, sorting), all stored explicit knowledge is used for analysis and comparison. Based on the generated visualization, the image $I$ is perceived by the analyst, gaining tacit knowledge $K^{\tau}$, influencing the analysts perception $P$. Depending on the tacit knowledge, the analyst has now the ability to interactively explore $E$ the visualized time series and spatio-temporal parameters by using the environment’s provided methods (e.g., zooming, filtering, sorting). During this iterative process, the analyst gains further tacit knowledge based on the adjusted visualization.

Explicit Knowledge Generation and Adjustment: To generate explicit knowledge $K^{\epsilon}$ (see Fig. 5 for a graphical overview), the analyst has the ability to include the spatio-temporal parameters of analyzed patients based on his/her clinical decisions to the explicit knowledge store, which can be described as the extraction $X$ of tacit knowledge. This explicit knowledge can be visualized in a separated view, whereby the explicit knowledge is automatically transformed into a dataset. Different views are providing the adjustment of the stored explicit knowledge by the analyst’s tacit knowledge.

Structuring the Explicit Knowledge: KAVAGait operates with explicit domain knowledge about different gait patterns. Thus, it needs a concept for each pattern. While a multi-level taxonomy of gait patterns would be possible (cp. Listing 1), for the KAVAGait prototype a flat list of 7 concepts is sufficient (‘affected knee’, …, ‘norm data’). The explicit knowledge is manifested in $[min,max]$ ranges for the spatio-temporal parameters. Therefore, KAVAGait needs a two-part approach: (1) Typically, the clinician categorizes the gait pattern of a patients and includes them in the explicit knowledge store under a category. In the explicit knowledge framework, this is expressed as a direct mapping linking the concept to their prototypical patients (cp. Listing 3). The $[min,max]$ ranges need to be calculated dynamically from the prototypical patients’ spatio-temporal parameters, because the patient population can be filtered e.g., by age. (2) If the clinician manually adjusts a $[min,max]$ range, this can be expressed as an indirect mapping with the given value range (cp. Listing 4). This explicit knowledge is utilized in multiple ways: (i) the concepts are represented as rows in the ‘Knowledge Table’ (Fig. 4.1.a). The values of the prototypical patients’ spatio-temporal parameters are shown as hatches in the ‘Parameter Explorer’ (Fig. 4.3) and to calculate the matchings and category differences. The $[min,max]$ ranges, which are either calculated from prototypical patients or manually adjusted, are visualized by the range sliders in the ‘Parameter Explorer’ and the boxes of the ‘Knowledge Table’.

Summary: This design study demonstrates that explicit knowledge extracted from the clinicians tacit knowledge opens the possibility to support clinicians during clinical decision making. Additionally, KAVAGait could also be used to share the knowledge of domain experts as well as to use it for educational support. In contrast to other analysis systems (e.g., based on MatLab), KAVAGait uses analytical and visual representation methods to provide a scalable and problem-tailored visualization solution following the visual analytics agenda [40, 19]. For keeping up with the large number of patients stored as explicit knowledge, clinical gait analysts need to continuously adapt the systems settings during the clinical decision making process. Supporting such interactive workflows is a key strength of visualization systems. Clinical gait analysis in particular profits from extensive interaction and annotation because it is a very knowledge-intensive job. By providing knowledge-oriented interactions, externalized knowledge can subsequently be used in the analysis process to support the clinicians. The newly developed visual metaphors provide an easy way to inspect variability of the data (e.g., standard deviation), allow to identify outliers in the data, and provide an easy to understand overview of the data and automated matching results. Additionally, based on the interactive twin box plots, it is possible to perform intercategory and patient comparisons by details on demand to find similarities in the data.

Addressing the need for deeper integration of domain knowledge, existing VA environments have found a multitude of mechanisms to structure and manage explicit domain knowledge. This paper set out to reflect and generalize the results from existing design study projects and work towards a structural framework for explicit domain knowledge in VA. This reflection has identified nine desiderata for such a structural framework. A preliminary structural framework is proposed that separates the concerns of explicit knowledge into three components: concepts, manifestation, and utilization. For the components concepts and manifestation, we apply linked data using SKOS and a custom RDF vocabulary for direct and indirect mapping to data items. The utilization component can be addressed based on the JSON-based Vega-Lite language. Being limited to the retrospective reflection of several knowledge-assisted VA design studies, this preliminary structural framework lacks any claims of completeness. Notwithstanding its limitations, this work certainly adds to our understanding of explicit knowledge in VA environments and provides a frame of reference for future design studies.

A natural progression of this work is to analyze a larger sample of VA environments in more detail by reverse engineering their explicit knowledge using this theoretical framework. Such a structured literature review can both survey the landscape of knowledge-assisted VA environments and identify missing vocabulary for knowledge manifestation in heterogeneous datasets. In particular, further modelling needs to examine more closely how concrete variables in datasets can be group to abstract concepts (e.g., a concept for blood sugar). In addition, the scope of knowledge utilization approaches can be more comprehensively surveyed. As a next step, the emerging structural knowledge framework needs to be tested in practice, either in new design study project or through evolution of an existing VA environment. Given the general claim, this implementation can also result in a reusable software library. Finally, the aspired real-world utility can only be assessed through empirical studies with users of VA environments.