The Influence of Visual Provenance Representations on Strategies in a Collaborative Hand-off Data Analysis Scenario

From the early work of decision theory [63, 10], the goal was to understand how users arrived at a clear conclusion when working with ill-structured data. Researchers continued this work with the study of the sensemaking process, which covers the human tendency to oscillate between foraging for new information and schematizing how it fits with what one already knows [48]. While multiple definitions have evolved from this preliminary work, the general understanding is that sensemaking “involves the ongoing retrospective development of plausible images that rationalize what people are doing” [69]. Of interest to the work discussed in this paper is the Data-Frame model proposed by Klein et al. [34]. In their work, they explain how something is introduced can have a direct impact on how users frame and continue their analysis. Intelligence analysis [18, 9], medical diagnosis [23, 39], and even humble internet research [42, 41, 49] all share tasks where complex data requires thoughtful consideration and artifact synthesis to arrive at and present a formal conclusion [58]. So, while common analytic settings clearly employ parts of the sensemaking process, they also define operations involving collaborative teaming or hierarchical units [37]. Collaborators can distribute workload, but they need to establish a common ground to understand the investigation status and contribute toward the goal [53].

Even without the added complexity of coordination and knowledge sharing among collaborators, sensemaking tasks are often nonlinear by nature. Insights and connections may be discovered independent of a strict method or procedural scaffolding, making it more challenging to systematically describe how concepts build on each other or explain the overall relationships. Add to the combination that collaborating analysts are dealing with multiple layers of uncertainty and trust [55], and the situation becomes even more complicated.

Among the many purposes targeted for provenance support in visual analysis applications [50], aid for collaborative work is bolstered by helping people recall what they know, clearly communicate the steps taken, and making it possible to reproduce past work [58]. As demonstrated by Mathisen et al. [38] in their description of the InsideInsights tool, capturing a user’s interactions as they worked and providing an interface for the addition of annotations, improved the collaboration and established common ground. This technique of enhancing interaction data with analyst-generated annotations is a common way to help maintain context for an analyst or different audience members [46]. Still, often the process of annotating (e.g., writing notes, tagging information) distracts from the gathering of information because it requires users to synthesize their fuzzy concepts into specific terms that may or may not communicate their exact meaning upon later review [58]. At the same time, the process of exploratory data analysis is inherently dynamic, leading to the requirement for plans to constantly change with the situation [61]. Without the transcription of accurate mental schemes, details can be forgotten, leading to false conclusions and inaccuracies [52, 51]. By partnering with computers, human analysts can focus less on annotation tasks, like recording how they arrived at different concepts, and shift their attention toward directing the analysis and hypothesizing relationships between discovered ideas [11, 19, 49]. In this study, we examine how the representation of a user’s process impacts the sensemaking processes of new analysts.

Analysis tools that capture analytic provenance information aim to improve the understandability [4], reproducibility [47], and transparency [28, 14] of insight generation over time by providing the “story” of data exploration and interpretation[46]. Yet, there are a variety of provenance types that serve different purposes depending on the context [50, 29]. Provenance can aid in the recall of past work, the verification of others’ work, the recovery of past actions, the review and optimization for future analysis, the presentation of new information, and ultimately the communication of insights between collaborators [50].

Visual provenance summarizations describe events that occur over time, and myriad approaches have been demonstrated for both algorithmic summarization and visual representation to ease human interpretation of the workflow, especially in collaborative settings. There is a need for accurate techniques that compress temporal events that occur while matching an appropriate level of temporal granularity and summarization to best serve different audiences. Some techniques focus on preserving the timing and order of events to allow for the review of specific analysis turning points (i.e., History representations) [17, 42, 75] while others provide a high-level summary of topics reviewed and remove elements of timing completely to make it easy to see what has been explored and what needs further analysis (i.e., Coverage representations) [68, 57, 20]. Provenance helps collaborators to maintain common ground as they work synchronously [14], or asynchronously [75, 72]. To assist in collaborator communication, many provenance visualizations focus on providing a reference to a user’s interaction history or data actions using a timeline [75] or branching trees [41, 17, 28]. Others take a drastically different approach by removing the temporal aspect entirely and instead aim to describe what data was explored rather than when it was explored. By ignoring time in the representation, viewers can focus on the context of data coverage and patterns (e.g., [57, 68, 72, 74]). These techniques trade a higher level of summarization for less emphasis on analysis step replication and the specific actions taken to arrive at an analysis state.

The potential value of provenance summarization features in analysis tools is well justified by prior literature [28, 45, 50, 71, 72, 74]. However, it is less clear how differences in the way the provenance information is summarized and visualized can affect an analyst’s process and decision-making. Numerous studies from the visualization community have demonstrated that differences in visual representation or the addition of new information can influence user bias. For example, Dimara et al. [16] have shown how allowing users to intentionally remove salient data (e.g., outliers) can lead to less bias and more rational decisions. Also, Wall et al. [68] have shown how providing users a summarization of what they have reviewed (i.e., an overview of their data coverage) can increase some user’s awareness of unconscious biases, while potentially encouraging others to amplify their biases by intentionally focusing their analysis on specific areas or hypothesis too.

Considering the common aim of using provenance visualization to support collaboration among multiple analysts [14, 56], and the impact that visualization can have on user interpretations[13], there is a need to better understand the ways the availability and representation of provenance information from one analyst may influence another analyst’s behaviors, biases, or conclusions. In addressing this gap, our work draws from lessons learned from existing empirical studies on visual design, provenance, and users’ analysis choices to further understand how to best provide provenance information to users.

Within the visualization community, there is a long history of evaluating the effects of using visual tools on user performance which is consistent with provenance representations. Of specific interest are the behavioral effects when making history information available. For example, Zhao et al. [75] focused on the various strategies participants used when examining the work of collaborators, suggesting that different strategies lead to different degrees of investigation completeness. Similar work has evaluated the types of strategies users employ when completing sensemaking tasks [32] and there are concerns that the strategies are influenced by the interface.

This becomes quite obvious when considering the types of interactions afforded to users in an interface. For example, tools like Hindsight clearly describe recall provenance through scented widgets to make it clear what parts of the data individuals have already explored in more detail [20]. By lowering the opacity for parts of the data recently examined, unexplored areas were, therefore, more prominent, and users were influenced to extend their examination. Xu et al. [72], in a more collaborative setting, similarly invited participants to review what data had been previously inspected and how it was visualized by prior analysts. By spatially distributing others’ attempts in a “constellation,” finding commonly used data and alternative areas of interest become more available to the user. This is a direct result of the interface and its influence on user interactions. Consistent with this line of work, multiple studies have shown how cognitive factors (like the anchoring effect [12]) can significantly impact users’ insights and exploration in decision-making tasks [15].

It is clear that representations influence analyst behaviors [55]. One of the goals of analytic provenance research is to prevent the effects of belief perseverance [12], base rate biases [40], misuse of representative heuristics [1], and resolving conflicting insights [35]. As an example, visualization techniques have tried to prevent selection bias by showing an overview of how the data has been filtered, and data type comparisons to help users recognize when they may be examining a subset of data too closely or with too much emphasis [8]. Similarly, by showing the work already complete and suggesting ways for the analysis to continue, SOMflow also helped analysts complete a more thorough investigation  [54]. These techniques rely on provenance information to help users recall what has been explored and potentially rectify their cognitive biases. Applicable to this study, Sarvghad and Tory [56], compared how coverage and timeline representations improved an analyst’s accuracy, and the amount of data explored when analyzing structured numerical data. To extend their findings, we compare effects in textual data analysis and also inspect the strategies employed. Although there is evidence that provenance representation influences user strategies and performance, there are outstanding questions about how different provenance representations compare, especially in textual data analysis.

Designs for summarizing analytic provenance can take a wide variety of forms. For the focus of our study, we consider two common classes of provenance summaries as generalizations of designs found in the research literature. Specifically, we distinguish provenance representations into either interaction history or data coverage summaries. Both designs bring uniquely different benefits to analysts in practice.

Provenance summaries using an interaction history approach tend to provide a timeline of events or describe which data are processed over time. This type of provenance helps identify critical moments of failure, reproduce results, or provide additional data transparency, but it takes more time to review because there is often more data to make sense of. Unfortunately, while computing systems are capable of capturing interaction events, this process can quickly bloom into large sequences that are challenging to summarize. Many types of provenance visualization tools do not distill meaning from raw interaction logs, choosing instead to visually represent all interactions or analysis stages in interactive tools to uncover patterns or flows (e.g., [14, 28]).

In contrast to provenance designs emphasizing the temporal flow of the analysis, we describe a separate generalized class of provenance summary as data coverage designs, which typically represent an overview of what data was explored instead of when it was explored. By compressing time, users can see what has been explored and what remains [56, 57, 68, 72, 74]. These techniques provide a higher level of summarization, focusing on providing a sense of context but lack enough detail to clarify or recreate past work [7].

While modern techniques still implement examples from both data coverage and interaction timelines [75, 60, 66], prior work has seen greater emphasis on using machine learning to extract patterns and assist in the summarization of time [27, 24, 59, 74] and less emphasis on comparative studies investigating the implications of different provenance summaries with people. Questions remain about how best to summarize interaction histories in digestible ways that help analysts by balancing content and cognitive load. Many works have shown that provenance representations influence the analytical behaviors of users [4, 13, 20, 25], yet they do not directly compare the effects of the prototypical provenance representations we discussed. A more direct comparison between interaction history and data coverage would be beneficial since both techniques summarize and present the past in a digestible way and future automation techniques would benefit from guidance on selecting the appropriate level of detail for a user’s task.

Therefore, we designed a between-subject experiment to study how individuals work with different forms of provenance information while completing a textual data investigation. We address the following research questions to help direct our analysis:

1. 1.

   RQ1: How does the inclusion of analysis history or data coverage influence the conclusions reached by a secondary analyst?

2. 2.

   RQ2: How do secondary analyst behaviors differ when provided the analysis history or data coverage from a prior analyst?

3. 3.

   RQ3: How does the inclusion of analysis history or data coverage influence the types of strategies a secondary analyst uses to solve the problem?

As a basis for the study, we used a browser-based, direct manipulation interface to display documents and record participant interactions. Since provenance representations are commonly used in collaborative data analysis scenarios, we simulated a hand-off scenario where users pick up and finish the analysis started by someone else. In an online study, participants were asked to review a set of documents and describe any associations they were able to make. Provenance representations informed by the same prior analyst allowed for comparing analyst conclusions and strategies. Based on prior work with provenance, we hypothesized two main effects on behavior in collaborative hand-off cases. We expected history summaries to encourage the continuing analyst to engage in more verification of the prior analyst’s progress due to the inclusion of a more complete record of the prior analyst’s work (Hypothesis H1). And since data coverage summaries provide context about what has been explored at a glance, we expected users to quickly understand the prior analysis and explore other topics (Hypothesis H2).

With a combination of a think-aloud protocol [43], screen recordings, interaction logs, and semi-structured interviews, we sought to identify differences in participant conclusions reached and strategies used to better characterize the impact of provenance on collaborative analysis.

In collaborative data analysis tasks, users work together to uncover relationships and share results. Frequently, these analysis tasks require users to communicate ill-structured and potentially relevant information for future analysts to recognize and use. For our study of how a second analyst picks up after a prior analyst makes partial progress, we used a synthetic intelligence analysis scenario based on an existing publicly available dataset from the first micro challenge of the 2010 VAST Challenge data series [26]. This dataset, and others from the VAST Challenge, are commonly used as realistic proxies to simulate analysis tasks for visualization research (e.g. [75, 56]). The data consists of fictional phone transcripts, email correspondence, forum posts, newspaper articles, and other intelligence reports about fictional illegal arms traders. Of the 103 documents in the dataset, only 16 contain information relevant to the solution we asked participants to complete.

Within the tool, users could flexibly move and collapse documents as they explored, similar to other analysis workspace tools [33, 2, 52]. Specifically, users were tasked with determining if illegal arms traders were responsible for the spread of a mysterious pandemic. Participants could right-click to access a context menu. From this menu, they could trigger a search event for the term under their cursor or type out their own query in a text field. Searches highlighted the set of document title bars that contained exact string matches to the queried text. Critical to this experiment was identifying when and what kinds of information was revealed to users. The tool would actively log user events (e.g., document opens, mouse enters and exits, and searches conducted) as they explored the dataset. Each logged interaction was recorded with the time since the session began, the event type, the element’s identifier, and other relevant information (position on screen, content of search, etc.). All participants were given the same set of documents, instructions, a “Summary for Supervisor” field, and a note from the prior analyst (see figure  1). The instructions further reiterated the scenario introduced to the participants and the “Summary for Supervisor” was a blank note where participants would type out their conclusions at the end of the session. We describe the provenance representations and their interactions in the next section.

As described in Section 1, we define two general categories for provenance representations based on how they communicate time. To further explore how data coverage and interaction histories impact investigation conclusions, behaviors, and strategies, we examined prototypical representations of each. Below, we describe the different provenance representations used in the experiment:

To serve as a Control, all participants were given a textual description designed to mimic the series of conclusions or cursory annotations a prior analyst may string together as they completed the same analysis scenario (i.e., the “Notes from Analyst A”). This note served as an easy way to provide context and offer potential entry points for a participant’s data exploration. Inspired by similar methodologies [56, 75], the note was based on the interaction log of a researcher’s pantomimed analysis that followed specific details through documents to arrive at a partially correct conclusion. Several statements were intentionally hedged to provide ample openness for where participants could begin. In the pantomimed interactions, 15 documents were opened, along with 31 searches and 53 highlights. Generally, the referenced documents and approach would uncover the majority of details required to solve the whole solution (i.e., 6/16 documents). These participants were not given additional panels to filter the dataset and had to rely on the search tool described earlier to find information and solve the scenario. We used this same interaction history to construct the other provenance representations discussed below.

Some participants were additionally given a Coverage representation (Figure 2 A) of the data explored by the prior analyst. This view showed the countries that received the most attention from the prior analyst as a list with miniature bars. Each country’s bar displayed the ratio of documents explored by the prior analyst and the bars were arranged in descending order by the country’s mention frequency. The original dataset [26] did not have labels for the countries mentioned in a document, so these had to be hand-labeled by the researchers. Intending to simulate how a hypothetical tool would work, the researchers attributed the first city mentioned in a document to its corresponding country. These hand-labeled countries were also added to the preamble of each document, to help balance the conditions. This way the other conditions could also filter specific countries with the search tool. The Coverage panel made these country-focused searches conveniently accessible by displaying them as a list. By clicking a country from the list, a “tool-use” event would be logged, and the affiliated documents would be revealed—distinguishing those explored by the prior analyst and those not reviewed—by coloring the documents’ title bar. Clicking a selected country again removed document coloring. Participants could select a country to filter the documents and help direct their investigation.

Other participants were provided a History representation (Figure 2 B) that summarized the steps the prior analyst took to complete their task. This view gathered and displayed the searches and highlights of analyst A as different segments of the analysis. The interactions from the baseline log were augmented with a handful of additional highlight terms or relevant search terms to provide a bit more content for the users of the History panel. Participants could scan through the History panel to help get a feel for the general terms (searches) and evidence (highlights) the prior analyst cared about over time. While manually segmented, the segmentation was done systematically. Segments were delineated based on a search event and contained the corresponding highlight events that took place between searches. In total, there were 11 segments, each labeled with the number of documents reviewed as well as a small timeline in the right corner of each segment to visualize its duration and placement in the prior analysis. Much like the Coverage representation, when a participant clicks a segment, the corresponding documents from that segment would be revealed by changing the color of affiliated document titles. Clicking a search or highlight term would run the interface’s search command and simulate the results the prior analyst would have seen. Clicking a segment again removed the applied colors. Participants could select segments to filter the documents and help review aspects of the prior analysis.

This research was approved by the organization’s institutional review board (IRB). Participants joined a virtual meeting room and were asked to complete a demographic questionnaire capturing age, gender, academic program, and self-report measures for the ability to complete analysis tasks and their communication abilities. The experimenter then explained the web application interface and its functions to participants using a set of slides via screen-share. Within the tutorial slides, participants were introduced to the think-aloud protocol and then asked to demonstrate the technique with a short, irrelevant document. The researcher offered feedback and ways to help improve their think-aloud (e.g., they were asked to read aloud and verbalize their plans for what they would do next). Also, it is known that first impressions can have a large degree of influence over what people focus on [62], so all participants were read the same starting scenario (see section 2.2) to control for variations in wording. Participants were invited to ask questions about interactions throughout the tutorial, and a researcher was present with them in the interface to answer interface function questions as they worked. Participants were not told to explicitly “strategize,” but rather to “choose what to read with intention” because “they would not have enough time to read everything.”

Starting a screen recording, participants were given 30 minutes to complete the task and asked to think aloud [21] as they read and contemplated what associations they saw with the expectation that reading aloud would encourage deeper reflection [6]. At 10, 20, 25, and at the end of 30 minutes, participants were warned about the time elapsed and reminded of their task: “Try to identify what associations may exist and prepare your summary for your supervisor.” After the analysis concluded, a post-task interview (included in the session recording) helped capture their mental model and opinions of the tools they used. With their final analysis workspace still visible, Participants were asked a series of semi-structured interview questions to further specify their understanding. These questions asked participants about the relationships they were aware of, identify retroactive strategies they used to arrive at their conclusions, their thoughts about the prior analyst’s work, and the provenance views as applicable. Critically, they were asked to make a judgment on the relationship of arms dealing with the Nigerian disease (referred to as their conclusion).

We recruited 41 undergraduate university students from an upper-level computer science course as participants in the study. Participants were compensated with course credits. Data from five participants were excluded from analysis due to technical problems, communication uncertainty because of language barriers, or misunderstanding of task instructions. Of the remaining 36 participants, they were initially randomly distributed, before researchers assigned later participants to the smallest groups to finish with experimental groups of equal size (12 participants per condition). Fourteen participants (38%) identified as female, and 1 (3%) identified as non-binary/third-gender. All but 6 students (83%) were completing a degree in computer science, computer engineering, or software engineering.

The majority of participants were between 18 and 24 years of age (77%), 5 participants (14%) were within the 25–34 age category and 3 participants (8%) were older than 35. We asked participants to self-report their ability to complete data analysis on a scale from 1–10; an average reported score of 3.14 signified general inexperience in completing analysis tasks, indicating the participant sample might be considered analogous to novice analysts with no or limited experience.

We studied whether the availability of the provenance views affected the participants’ findings and final conclusions (RQ1) to look for implications of early bias influencing analysts’ ability to correct their preconceived expectations. A single author scored each participant’s written conclusions using a four-level rubric, according to four factors, including the accuracy of their reported findings; the recognition of errors made in the prior analysis; the number of findings and amount of detail provided; and the depth of relationships or connections among entities and events in the data.

These qualities were a set of features we expected differences in based on provenance representations, yet the scores varied greatly. Due especially to the open-ended nature of user-directed analysis, the diversity of user aptitude with analysis and the various ways participants could write their findings, our analysis did not find systematic differences in participants’ written conclusions caused by the provenance conditions. For example, when summarizing key findings, some participants formatted a formal report in prose, while others opted for a series of evidentiary bullet points. Often the written conclusion left out parts of the analysis and therefore was not a fair representation of the area’s a participant explored. For a similar reason, we do not report on participant encounters with the 16 solution-relevant documents, because we wanted to understand what information “stuck” and was reported in their concluding thoughts. Due to the range of individual differences among participants and personal styles for reporting, the quality and completeness of written conclusions varied greatly. We did not find meaningful differences in analyst written conclusions. Therefore, we prioritized the analysis of data from the personalized post-study interview questions as a basis for identifying differences in participant conclusions and analysis behavior.

As part of this analysis, we assessed participants’ confidence in the post-task interview when describing their final answer. High confidence implies that the user has convinced themselves of a specific relationship, and we want to see if that behavior varied systematically with the experimental conditions. One author coded participants’ interview responses to the question regarding their final answer about the possible existence of a relationship in the data (i.e., the main investigation goal in the analysis scenario). With the support from a second author to review the handful of edge cases, we separated participant responses into two categories (high and low confidence) based on the number of hedge statements in their verbal conclusion. To do this, in the interview, we asked participants if they identified a relationship between the concepts they were investigating. Participants who stated clearly that there was or was not a relationship were designated as high confidence (55%). Alternatively, those who suggested only a potential relationship, a need for additional time to review the data, or uttered more than two hedged phrases (e.g., “there might be…,” “I guess…,” “I think…,”, etc.), were placed in the low confidence category (45%).

Upon first inspection, Figure 3 shows a strong difference in participant confidence. We see a main effect with Fisher’s exact test ($p<0.05$) by condition. Yet, post hoc comparisons with Bonferroni correction fail to identify the specific differences between conditions. While we cannot state one condition varied significantly from the others, those with a Coverage representation tend to exhibit conclusions more similar to the Control than those with the History representation. This is to say that the majority of conclusion qualities are not as dramatically impacted by the representation of provenance information. We do see some impact on participants’ confidence when they verbally explain their conclusion, but without significant interaction effects, we instead turn our attention to the behaviors exhibited by participants to better explain how their analysis differed.

The way individuals interact with an interface indicates how they make sense of the data. With the variety of user activities, we choose to use a quantitative indicator to more clearly describe behavior patterns in participant approaches (RQ2). As captured in their interaction logs, we turn to the various analysis events and actions participants ran in the interface as a quantitative proxy for their understanding and to tease out the influence of provenance representation on user analysis behaviors. We look at three key representations of interest: the degree of similarity, and difference to the prior analyst’s review, and the rate of filtering for specific information.

Of concern to the inclusion of provenance information is the influence on how much repetition is in subsequent analyses, or the amount of similarity to the prior analyst. While verification can be beneficial when auditing the veracity of a result, in most cases repeated work is not encouraged. We wanted to compare the analysis of each participant to the referenced prior analysis to understand if the provided provenance representation influenced how they addressed the problem of “continuing the analysis.” If participants chose to look into the same concepts as the prior analyst, how similar was their review (overlap), and if they looked at different things, how unique was their investigation (independence)? To quantify the amount of overlap and independence in participants’ behaviors we looked at which documents in a participant’s interaction history were opened. Since critical information in the underlying dataset is separated into individual documents, we can determine how similar an investigation was to another by examining the set of documents opened. Both provenance representations were based on the same set of 15 documents, and we calculated two separate, but similar ratios (overlap and independence) from the sets of documents participants reviewed. We calculated a participant’s overlap ratio by considering the intersection between the set of documents a participant reviewed that was also reviewed by the prior analyst and divided by the number of documents the prior analyst reviewed ($\frac{\includegraphics[scale={2}]{figures/overlap-mini-BLANK.pdf}}{\includegraphics[scale={2}]{figures/overlap-mini-analyst-BLANK.pdf}}=\frac{\{User\}\cap\{Analyst\}}{\{Analyst\}}$). An overview ratio of 1.0 implies that a participant saw all of the documents that analyst A reviewed.

The independence ratio captures the proportion of documents a participant reviewed that were different from the set reviewed by the prior analyst ($\frac{\includegraphics[scale={2}]{figures/ind-mini-BLANK.pdf}}{\includegraphics[scale={2}]{figures/ind-mini-user-BLANK.pdf}}=\frac{\{User\}-\{Analyst\}}{\{User\}}$). An independence ratio of 1.0 implies that a participant only reviewed documents that the prior analyst did not. Although these ratios examine similar participant behavior properties, they are not inverse because they compare different document sets.

When we compare the degree of overlap among participants (Figure 4a), we do not see a strong difference between the conditions, implying that the amount of overlap a participant may have with a prior analysis has more to do with individual differences in approach and investigation intentions. But, some expected trends are visible. For example, those in the Control condition straddle the center ($\sim$50%) as though they were unaware of which documents were reviewed by the prior analyst. We also see more spread in the Coverage condition, likely because they could filter documents and choose to follow or avoid the prior analysis. Finally, the History condition has the highest median overlap ratio likely since their representation emphasized how the prior analyst worked through documents.

We see a much different behavior among participants’ ratios when we examine how independent their analysis was from the prior analyst (Figure 4b). A Kruskal-Wallis test revealed a significant difference in independence ratios $H(2)=7.01,p<0.05$. For those in the Control condition, more than half of the documents they reviewed had not been explored by the prior analyst. The result can be explained since they had no idea which documents were specifically reviewed by the prior analyst, and opened many more documents on average. With more documents opened, the likelihood of a document belonging to the set from the prior analyst goes down, leading to a higher independence ratio because there were many more documents that were not reviewed by the analyst. We also see a long tail for those in the History condition. The pairwise post-hoc Dunn test with Bonferroni adjustments showed to be only significant for the History and the Control ($p<0.05$) suggesting that they generally spent their session reviewing mostly the same documents as the prior analyst. They also looked at fewer documents overall. With fewer documents reviewed, and an emphasis on documents opened by the prior analyst, these participants were more likely to maintain lower Independence ratios. Overall, we see that while the degree of overlap is not significant, there appear to be some differences in the diversity of documents participants are exposed to when given various provenance representations. This is to say that the affordances provided by different tools can influence which information participants review.

Another aspect of our analysis relates to how participants direct their investigation. We looked at the total number of interactions and divided it by the length of a participant’s session to determine an average rate of interaction. The interaction rate (see Figure 4c) can serve as a proxy for the level of control a participant has over the interface and their investigation. Participants with exceptionally low interaction rates may be taking a long time to review documents or working very methodologically, whereas exceptionally high interaction rates may imply they have opened numerous documents at once or are not spending enough time understanding each document. For example, the Control condition opened the most documents (41.5 documents) on median, while maintaining the slowest median interaction rate (0.95 interactions/sec). This contradictory phenomenon may suggest that those in the Control condition opened many more documents on median in an attempt to understand the data, but worked through the documents and the task slowly. Yet, uncovering more descriptive interpretations would require the review of additional metrics.

For example, we can turn to the frequency of filtering events to understand how participants search and reduce the data space (See Figure 4d). While the differences are not significant, there are more median filtering events completed (18.0) by those in the History condition while they also look at the fewest documents (28.5) on average. This is to suggest that those in the History condition were more likely to be methodological and consistent since they were not exposed to as many documents and spent more of their session filtering and refining their investigation criteria. On the other hand, those in the Coverage condition appear to have overall investigation behaviors most similar to the Control, but also have the fastest interaction rate. Those with the Coverage representation may have allowed participants to work faster and independently.

To further describe the ways participants were exposed to information, we examined the timing of users’ searches and provenance representation usage. In this case, we define the term filtering to describe the sum of searches and provenance panel usage per participant. Although those in the Control condition did not have access to a provenance representation, they relied more heavily on search to find information. This is to say that search and panel usage both helped to reduce the search space in the dataset and their combination provides a more universal comparison. We examine when these behaviors occur to understand when users are selecting and directing their investigations. Since each participant filtered the data their own number of times, we calculate a filter percentage over time. All but 2 participants eventually reached 100% of their filtering events within their 30-minute session, but ultimately, we draw attention to the rate of change. In Figure 5, there is lots of overlap in when participants are completing their filtering behaviors. To clarify the trends, we apply a 4-factor polynomial since it gave the highest $r^{2}$ coefficient (0.745) and helps characterize the behaviors observed. As evidenced in the modeled regression, participants from the History condition complete about 50% of their filtering events by about 12 minutes, while it takes 17 or almost 19 minutes for the Coverage and Control conditions to conduct half of their filtering events respectfully. We counted the number of participants who had completed at least half of their filtering events by the halfway point in the investigation (15 minutes) as a proxy for how active participants were selecting data to review. Because the provenance representations provided a more convenient (i.e., a single click) way to filter the data, we see that both the History and Coverage condition completed the majority of their filter interactions before those in the Control condition. A Fisher’s exact test confirmed that there were differences in the rate of filtering among the different conditions ($p<0.001$). Further post hoc pairwise Fisher’s exact tests with Bonferroni corrections confirm that the amount of filtering completed by those in the History condition differed from the Control ($p=0.001$) and Coverage conditions ($p<0.001$), but no difference was detected between the Coverage and Control conditions at 15 minutes. More specifically, those in the History condition complete the majority of their filtering behaviors before the other conditions (and in the first half of the session).

We believe this difference is due to the interaction pattern required for the use of the History representation. Since information about which documents were reviewed is only accessible after clicking a segment, participants from the History condition would commonly click through multiple segments at once in pursuit of a subset of documents to review instead of intentionally selecting a segment to review or independently typing out their own search query (discussed more in Section 6). This pattern of clicking through each segment in time, or scanning the resulting documents to find something previously reviewed likely led to an increase in filtering events overall (median 18.0 events) and also a higher frequency of events earlier in a session while participants were still gathering information. On the other hand, it appears that the addition of Coverage information for participants did not significantly change when participants would be selecting/filtering data (as compared to the Control). Ultimately, we see some quantitative differences in participant interactions, including how little content those in the History condition were able to review, and how frequently they were filtering the dataset. We now take these quantitative differences and construct some qualitative definitions for user strategies.

The strategies employed by participants during an open-ended evaluation likely depends on the kinds of information they are presented with (RQ3). Comparing and analyzing the strategies users take when approaching their exploration can shed light on how provenance information is used when investigating the relationships of various data.

In alignment with the work on situated planning [61], of the participants who verbalized a preparatory plan, most were vague and often only consisted of 2–3 steps. These early plans were interesting to us as we wanted to see if the availability of provenance information would influence how plans were made and adapted (see RQ3). With analysis plans being constantly adjusted and renegotiated as new information is acquired, we ultimately simplified our analysis by focusing on initial plans and the actions observed in the first 10 minutes of activity.

From participants’ think-aloud and retrospective interviews, we tried to reconstruct participants’ intentions as they began their analysis. To do this, one author analyzed the data by conducting two rounds of open coding to establish a set of emergent features from user actions and analysis foci. In a similar data analysis task, Zhao et al. [75] defined a set of analysis strategies derived from their own qualitative coding. Their work compared the strategies participants used when constructing knowledge graphs with or without an interactive state timeline. Our work differs in that we compare different provenance representations, while they only evaluate differences with or without provenance features. While their study offered participants different data analysis affordances, we borrow similar qualitative analysis steps in our work as well. From their identified strategy definitions, we cross-referenced our most common tags and refined a set of similar analysis strategies. In contrast, ours are more focused on the amount of similarity to the original analyst’s investigation as well as our users’ commitment to their investigation plan. We clarify and define our categories below.

• •

  Keyword Browsing - These participants set a plan for how they wanted to understand the data before they began or shortly after reviewing the note provided by the analysts. They made an intentional plan, maintained attention to the planned areas of interest, and frequently hypothesized different relationships. They had a medium amount of overlap with the prior analyst and tended to have a higher independence score since they were trying to extend the analysis and were more likely to have a more active role in the investigation.

• •

  Random Access - These participants had less structure to their investigations—often working without stating how they wanted to systematically approach the problem—and spending the majority of their time gathering information instead of synthesizing hypotheses. These participants bounced around the dataset with about equal amounts of overlap and independence from the prior analyst.

• •

  Reviewing Origin - These participants expressed a plan to verify the work from the prior analyst. Often this was motivated by a lack of trust in the conclusions made in the analyst’s summary or started verifying the prior analyst’s work and ran out of time for their own investigation. Therefore these participants have noticeably lower independence scores and higher amounts of overlap.

• •

  Starting Over - These participants explicitly stated that they wanted to work independent of any influence from the prior analyst. They were worried explicitly about bias or interested in comparing their own understanding with the work by the prior analyst later. These participants intentionally closed the guidance from the prior analyst and waited till at least 15 minutes into the task before they reviewed the analyst’s summary or the provenance panels.

As reported in Figure 7, there are some interesting differences among users and their strategies. With only a handful of participants in the starting over group (n=4), we rely on descriptive statistics and visual analysis to describe the groups and their differences instead of standard statistical methods. To begin, all groups appear to open the same number of documents, except for those in the starting over group (45.5 document open events). Those in the starting over group also had the highest median interaction rate (1.20 interactions per second). Much of this is likely due to the small number of participants in the group, but the way most of these participants worked was often without a stated plan and typically focused on gathering information generally. Similar behaviors were seen in the random access group. Without a plan, these participants also had high median interaction rates (1.07 interactions per second) and the highest interquartile range (0.90–1.74 interactions per second). This can be explained by the lack of intention and purposeful activity these participants appeared to execute during the analysis and reduced specificity of what they were looking for. Frequently, these participants had bursts of activity where they would be gathering information slowly before they would find a recognizable piece of information and open multiple documents they already reviewed to find interrelationships and similarities. Without a strong plan set at the beginning and generally less filtering to help find the information of interest, these participants were most likely to require more review of the data.

The other two categories had more explicit intentions for what they wanted to review. Those in the reviewing origin group completed the most filtering events on median (23.0 events). Likely due to the high ratio of participants from the History condition, this characteristically high number of filtering events is a direct result of using the History panel to review various segments of time; users had to trigger a filtering event each time they wanted to examine the documents reviewed in each segment of the prior analysis. These participants also maintain a high degree of overlap with the prior analyst (80%) and the lowest median independence (45%). On the other hand, those in the keyword browsing group also had a stated plan and had a much different degree of overlap with the prior analyst (only 40%). Most of these participants intended to explore key aspects of the data described by the prior analyst and tended to open more documents the prior analyst had not touched. We noticed that those who used a keyword browsing strategy generally uncovered a more complete and accurate picture of the dataset. We believe this is because they independently verified aspects of interest and followed keywords instead of the work by the prior analyst.

Peculiarly, while there is not a significant difference, the distribution of strategies almost appear to follow with the three conditions(see Figure 6), where keyword browsing was mostly associated with those in the Coverage condition, random access was mostly members from the Control condition and those in the reviewing origin group were from the History condition. Yet, without strong evidence that provenance representations influence the type of strategy participants employ when planning for their analysis, we cannot conclude that these strategies are determined by the affordances in the interface.

Our study of how users pick up an analysis with provenance information from prior collaborators was based on a single analysis scenario and targeted participants with limited data analysis experience. To build further knowledge on the topic, following research is needed with additional data sets and with participants with varying backgrounds from different collaborative data analysis communities like intelligence operators, medical teams, and academic researchers. Studies could also consider how differences in participants’ abilities, problem-solving aptitude, or preferences for particular analytic strategies might influence different approaches or interaction patterns.

A challenge with working with open-ended sensemaking tasks is that a user’s conclusions may not fit within certain bounds of available conclusions and captured data metrics. We see this in capturing participant conclusions. We set out to capture participant accuracy, error correction, findings made, amount of detail, and depth of relationships but did not have the fidelity in the task complexity nor measures to capture meaningful differences. In future work, more thoughtful care and constraints should be taken to more formally compare these factors and the influence of provenance.

In this work, the researchers describe the tool and scenario to participants. Due to the natural variation in speech, there are potential confounds introduced based on different inflections being interpreted by different users. While all participants were read the same statements, future work could better control for these effects by using a prerecorded procedure or expanding to additional datasets with different contexts to help generalize findings.

While our work introduced the comparison of two prototypical techniques used in the literature, there are also more ways of representing past work and their influence on user performance ought to be explored. for example, some work focuses on generating textual summaries [70, 31], while others show branching timelines [8, 67, 17, 65] that communicate how analysis adapts, transforms, and evolves. Still, others use graphs and networks as a way of providing concept maps [75, 73, 36], while others design comics as a technique for summarizing segments of time [3, 30]. Questions remain not only in the representation provenance should take, but also at what level of detail the provenance should be maintained. The comparison among how these techniques influence user performance as well as variations in the level of detail ought to be explored in the future.