Talk2Data: A Natural Language Interface for
Exploratory Visual Analysis via Question Decomposition

Natural Language Interfaces (NLI) provide an accessible approach for data analysis, greatly lowering the requirements of user knowledge. With the goal of improving the usability of visualization NLIs, various systems have been explored both within the research community (Sun et al., 2010; Gao et al., 2015; Setlur et al., 2016; Aurisano et al., 2016; Narechania et al., 2020; Hoque et al., 2017; Cox et al., 2001; Wen et al., 2005; Srinivasan et al., 2020; Liu et al., 2021; Luo et al., 2021) and industry (pow, [n. d.]; tab, [n. d.]). A common challenge for NLIs is how to precisely understand users’ intentions that are presented in a nature language (NL). In order to enhance the capability of NL interpretation, existing NLIs are designed either to guide users to provide a more concrete nature language query (Cox et al., 2001; Setlur et al., 2016; Yu and Silva, 2019) or untangle ambiguities in the input query (Gao et al., 2015; Wen et al., 2005; Sun et al., 2010; tab, [n. d.]) to better capture user’ requirements and more precisely drive the underlying data analysis for question answering.

The initial prototype of visualization NLI (Cox et al., 2001) does not relay on any intelligent approach to interpret user questions but create a set of supporting commands to guide users’ inputs. Flowsense (Yu and Silva, 2019) and Eviza (Setlur et al., 2016) depends on the pre-defined grammar to capture query patterns. When the user types a partial query and pauses, the system triggers the feature of query auto-completion to guide users’ queries. However, the grammar-based methods limits the range of questions as it is impossible to cover all of the possible tasks.

To improve the flexibility in posing data-related questions while managing ambiguities in NL queries, many NLIs leverage the sophisticated NLP parsing techniques (e.g., dependencies) to understand the intuitions of queries and detect ambiguities present in the queries. In Articulate (Sun et al., 2010), the translation of user’s imprecise specification is based on a NL parser imbued with machine learning algorithms that are able to make reasoned decisions automatically. DataTone (Gao et al., 2015) adopts a mixed-initiative approach to manage ambiguities in NLIs. Specifically, the system displays ambiguity widgets along with the main visualization, therefore users are allowed to switch the content in widgets to get desired alternative views. The idea in DataTone are extended to NL4DV toolkit (Narechania et al., 2020), a python-based library released to translate user queries into a high-level visualization grammar. For the developers without experience with NLP, NL4DV can save their efforts on learning NLP knowledge when building the visualization NLIs.

The aforementioned NLIs are only able to answer the precise simple questions. The grammatical-based methods do not support complex questions as interpreting multiple tasks in one query poses parsing difficulty (Yu and Silva, 2019), on the other hand, the NLP parsing techniques leverage rule-based parsers to comprehend user instructions and questions. Thus, when a question does not specifically contain keywords for analytic tasks, these interfaces cannot precisely understand user intentions. However, for the users with few knowledge about the data, it is almost impossible for them to ask precise and concrete question. In practically, the usability of these systems is under user expectations. In order to overcome these limitations, We built a novel deep-learning based model that not only can resolve a complex question into a series of relevant specific simple questions, but also improve the robustness of NL interpretation. In Talk2Data, users can ask both simple and complex questions about a table, and get well-designed diagrams that represent the facts extracted for answering the input question.

Question Answering (QA) is a well-researched area about building systems that can answer NL questions. Advances in NLP facilitate the development of various QA systems, such as Text-Based QA (Iyyer et al., 2014; Tran and Niederée, 2018; Feldman and El-Yaniv, 2019; Min et al., 2019; Ding et al., 2019; Perez et al., 2020), Knowledge-Based QA  (Bao et al., 2016, 2014; Berant et al., 2013; Saxena et al., 2020), and Table-Based QA (Pasupat and Liang, 2015; Yin et al., 2016; Jauhar et al., 2016; Yin et al., 2020).

In this paper, our work concerns about Table-Based QA, where we are tasked to answer both complex and simple queries given a table. The existing Table-Based QA system, such as  (Pasupat and Liang, 2015; Yin et al., 2016; Neelakantan et al., 2016), are designed to answer the simple questions. Given a NL query and a table, Neural Enquirer  (Yin et al., 2016) first encodes the query and table into distributed representations, and then use a multi-layer executor to derive the answer. It can be trained using Query-Answer pairs, where the distributed representations of queries and the table are optimized together with the query execution logic in an end-to-end fashion. However, Cho et al. (Cho et al., 2018) stated that only using the answer annotated dataset for training and evaluation may count “spurious” programs, the system will accidentally lead to correct answers by using the wrong cells or operations. Therefore, the authors propose a multi-layer sequential network with attention supervision to answer questions; it uses multiple Selective Recurrent Units to improve the interpretability of the model. Moreover, translating NL to SQL queries is a commonly used approach to answer the questions related to spreadsheets or database  (Zhong et al., 2017; Xu et al., 2017; Yu et al., 2018). Seq2SQL (Zhong et al., 2017) leverage the policy-based reinforcement learning to translate the NL queries to corresponding SQL queries. SQLNET (Xu et al., 2017) proposes a sequence-to-set model and a column attention mechanism to synthesize the SQL queries from NL as a slot filling problem. In the past few years, the large-scale pre-trained language models have rapidly improved the ability to understand and answer the free-form questions. The most recent work, TaBERT (Yin et al., 2020), is a pre-trained language model that is built on top of BERT (Devlin et al., 2019) to jointly learn contextual representations for queries and tables, which can be used as encoder of query and table in QA systems.

Although efficient, the above existing Table-Based QA systems are designed to conduct information retrieval tasks, which cannot answer the questions requiring to analyzing the data. Moreover, the input questions for existing systems have to be precise and concrete. When the input question contains multiple tasks or drops the anchor words, the accuracy of system will be strongly impacted (Pasupat and Liang, 2015). Existing works, proposed to resolving complex questions, are concentrated on text-based QA, such as  (Perez et al., 2020; Min et al., 2019). To our best knowledge, there is no prior study working on resolving the complex questions about a table. To fill this gap, we introduce Talk2Data, a data-oriented question and answering system that supports both simple and complex questions. In order to answer the complex questions, in Talk2Data, we adopt a novel decomposition model to resolve the complex questions into a series of simples that can be answered by data facts. The decomposition model extends the classic sequence to sequences architecture (Sutskever et al., 2014) and integrate attention and copy mechanism to guide the generation of each simple questions.

Our goal is to design and develop a data-oriented question and answering system that is able to automatically extract data facts from an input spreadsheet to answer users’ complex questions about the data. To achieve the goal, a number of requirements should be fulfilled:

1. R1

   Elaborate complex questions in context. The system should be able to resolve complex questions and elaborate the problem gradually from different aspects to give a comprehensive answer in context of the input data.

2. R2

   Rank the answers. The system should be able to rank the potential answers, i.e., data facts, in order according to their relevance to the question.

3. R3

   Clear answers narration. The answers to a complex question, should be visualized with narrative information such as captions and annotations and arranged in a logic order so that the users can easily read and understand them in a short time.

4. R4

   Real-time communication and responding. To improve the user experience, the system should be able to support real-time query and should search for the results and respond to users immediately without latency.

To fulfill the above requirements, as shown in Fig. 2, we design Talk2Data system with four major modules: (a) the preprocessing module, (b) the decomposition module, (c) the answer seeking module, and (d) the answer representation module. In particular, the preprocessing module parses the input tabular data $X$ and the corresponding question $Q$ and combines the parsing results together as word sequence $Q_{x}$ in the following form to facilitate computation:

where $w_{i}$ is a word / phrase in $Q$ and $c_{i}$ is a column in $X$ and $\langle N|T|C\rangle$ shows its corresponding data type, i.e., numerical ($N$), temporal ($T$) and categorical ($C$) respectively.

The decomposition module introduces a deep learning model by extending the classic sequence-to-sequence model to resolve a complex data-oriented question (represented by $Q_{x}$) into a series of relevant low-level questions that cover different aspects of the question to guide the answer seeking process (R1):

where $s_{i}$ is a hidden vector corresponding to a low-level question that can be answered by specific data facts. The details about the decomposition algorithm is discussed in Section 4.

In the answer extraction module, the system searches the data space $X$ to extract data facts $f_{i}$ that are relevant to each of the low-level questions $q_{i}$ and rank them to find out the answers (R2). The whole process is based on a parallel beam-search algorithm that guarantees the performance requirement as described in (R4). This step can be formally presented as:

where each data fact $f_{j}$ is a potential answer to the question $q_{i}$. The facts are ordered based on their relevance to the question. We define the data fact $f_{i}$ as a 5-tuple following the definition introduced in  (Shi et al., 2020), which is briefly described as follows:

where type (denoted as $t_{i}$) indicates the type of analysis task of the fact, whose value is one of the following cases: showing value, difference, proportion, trend, categorization, distribution, rank, association, extreme, and outlier; subspace (denoted as $s_{i}$) is the data scope given by a set of filters; breakdown (denote as $b_{i}$) is given by temporal or categorical data fields based on which the data items in the subspace can be divided in groups; measure (denote as $m_{i}$) is a numerical data field based on which the program can retrieve a data value or compute a derived aggregated value in the subspace or each data group; focus (denote as $x_{i}$) indicates a set of specific data items in the subspace that require extra attention.

Finally, the representation module organizes the data facts in order and visualize them via a set of captioned and annotated charts (R3) that are specifically designed to help with the narration of data semantics. In the following sections, we will describe the technique details of the decomposition, answer extraction, and representation modules.

Our system is designed to resolve two types of complex questions corresponding to two ways of defining compound tasks (bottom-up and top-down) (Schulz et al., 2013):

• •

  Type-I are questions directly mentioning multiple data facts in the sentence, which can be resolved by the bottom-up approach via enumerating and separating the simple individual questions. For example, the question “In the year with most reviews, what is the distribution of price over different genres?” can be resolved into two simple questions “Which year has the highest reviews?” and “What is the overall distribution of price over genre?”.

• •

  Type-II are questions involving multiple potential data facts due to the omission of some details, which can be resolved by top-down approach via exploring possible individual simple questions. For instance, the question “Which genre of book is an outlier?” can be resolved as simple questions such as “Which genre of books has anomaly user ratings?” and “Which genre of books has anomaly reviews?”.

Fig. 3 illustrates the running pipeline of the system’s question decomposition algorithm. Given an input question $Q$ the algorithm first check if the question is a complex or simple question based on a pre-trained classifier. The simple questions will be directly send to the system’s answering module, but the complex ones will be decomposed iteratively into a series of relevant simple questions via a deep decomposition model. The model resolves an formalized input complex question $Q_{x}$ into a set of sub-questions $(q_{1},q_{2},...,q_{n})$. If $q_{i}$ is a simple question, it will be directly answered, otherwise it will be decomposed again. This process runs iteratively until all the complex questions are resolved.

The decomposition model is designed by extending the classic sequence-to-sequence model (Sutskever et al., 2014) from four aspects: (1) we pre-train a BERT-classifier (Devlin et al., 2019) to compute a conditional vector for each input question $Q_{x}$ to indicates the question type (i.e., Type-1 or Type-II) so a proper decomposition method could be chosen by the model; (2) we add a decomposition layer in the model to transform the encoding vector $h$ of $Q_{x}$ into two hidden vectors ($h_{q1},h_{q2}$) corresponding to two output questions; (3) we employ the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015) in the model to ensure and enhance the relevance between the input complex question and the output questions. (4) we integrate a copying mechanism (Gu et al., 2016) to generalize the model so that it could make a correct response when users ask questions about a new dataset beyond the training corpus. To make it simple, our model decomposes a complex question into exactly two sub-questions that could either be a simple question or another complex question.

Specifically, given the formulated question $Q_{x}$, an embedding layer $W_{emb}$ projects each of the input words into a latent vector $v$, which is further encoded by a bidirectional GRU encoder:

where $h$ is the vector representation of $Q_{x}$ that captures the semantics of the input question and the corresponding tabular data.

At the same time, as shown in Fig 5-a, a condition vector $c$ that indicates the question type is calculated by classifying $Q_{x}$ based on BERT (Devlin et al., 2019), a large scale pre-trained nature language model:

where $u$ is the intermediate vector encoded by BERT and $W_{c}$ is a parameter matrix to be trained. The output $c$ is a two-dimensional one-hot conditional vector that indicates the question type with $[0,1]$ indicating Type-I questions and $[1,0]$ indicating Type-II questions. With the vector, the model is able to select a proper approach to decompose the input question.

In the next, a decomposition layer is introduced in the model to transform $h^{*}=[h,c]$ into two hidden vectors $h_{q_{1}}$ and $h_{q_{2}}$ (Fig 5-b). It is implemented by a fully connected feed-forward neural network. Formally, the decomposition process is defined as follows:

where ${W_{q_{i}}}$, and ${b_{q_{i}}}$ are the weight matrix and bias vector to be trained.

Finally, a GRU is used to decode $h_{q_{1}}$ and $h_{q_{1}}$ to generate two sub-questions $q_{1}$ and $q_{2}$ word by word as the final output of the model:

In the above process, an attention mechanism (Fig 5-c) (Bahdanau et al., 2015; Luong et al., 2015) is incorporated to allow the decoder referencing to the relevant words in the input question when generating each word in the decomposed sub-question. It further enhances the semantic relevance between the input and output questions. At the time step $t$, the attention layer first calculates the attention weights by considering current hidden state $h_{t}$ in the decoder and all the hidden states of the encoder $h_{encoder}=[h_{1},h_{2},...,h_{n}]$:

The contextual vector for the input question $v_{ctx}$ is then computed as the weighted average over all the encoder states. After that, the attention layer concatenate ${v}_{ctx}$ and $h_{t}$ to produce a new hidden state $h^{{}^{\prime}}_{t}$ for further predicting next word in the decoder:

where $W_{attn}$ is the weight matrix to be trained.

To design a robust model, we have to consider the situation when decomposing a question about a dataset outside the scope of the training corpus. In this case, an attention mechanism is not enough as it is difficulty to predict an unseen word, e.g., the column name in the new dataset, when generating a sub-question. To address this issue, we integrated the copying mechanism (Fig 5-d) (Gu et al., 2016) in the model, which generalized the model by selectively copying some unseen words directly from the input (either question or data columns) when generating a sub-question. Intuitively, it estimates the probability $p_{c}\in[0,1]$ of using a word copied from the question/data column instead of using a word generated by decoder when produces a new sub-question. Formally, $p_{c}$ is computed by combining the current hidden state $h_{t}$ in the RNN model, the contextual vector $v_{ctx}$ from the attention mechanism, and the last generated word $w_{t-1}$ together:

where $v_{c_{i}}$ is the trainable weighting vector that transform the above three vectors into a single value to predict the probability.

To encourage the output of the decomposition model as identical as possible with the target sentences in our training corpus, the model is trained by minimizing the word-level negative log likelihood loss (Edunov et al., 2018):

where $t_{i}$ is the current reference word in the target sentence. Given the previous words $t_{1},\ldots,t_{i-1}$ and the input question $Q_{x}$, this loss function tends to maximize the probability of the reference word $t_{i}$ as the prediction for current word.

To train our model, we prepared a new table-based question decomposition corpus (Table 1) with the help of 700 English native speakers from the crowdsourcing platform¹¹1https://www.prolific.co/. The corpus consists of 7,096 complex questions including 3,492 Type-I questions and 3,604 Type-II questions. Two simple questions were prepared as the decomposition results for each of these complex questions, i.e., 14,192 simple questions were prepared. In our corpus, we guarantee each simple question corresponds to an simple analysis task to ensure the question can be answered by at least one data fact. In general, we prepared the corpus via four steps: (1) selecting a set of meaningful data tables in various domains based on which complex questions will be prepared; (2) generating complex questions with all type of structures by a computer program; (3) manually polishing the machine-generated questions to reach the standard of natural language via the crowdsourcing platform; (4) eliminating the low-quality questions.

Table Selection. We collected 70 tabular datasets in different domains from Kaggle and Google dataset search. These datasets were further filtered based on three criteria: (1) containing meaningful data column headers; (2) having sufficient data columns and diverse column types to support all types of questions; and (3) containing informative data insights. As a result, 26 data tables were selected. Each of them has a meaningful column header and contains at least one numerical, one temporal, and one categorical field. All of them were tested by the online auto-insights platform (Shi et al., 2020) to make sure meaningful data insights could be discovered from the data. We also make the size of the data diverse, the number of rows ranges from 26 to 86,454 (mean 7,067); the number of columns ranges from 4 to 16 (mean 9).

Question Generation. To generate a complex question, we created a set of random facts and select the insightful ones based on the methods introduced in (Shi et al., 2020). After that, we enumerated the facts to generate meaningful fact combinations that are potential answers to a complex question based on the methods introduced in  (Schulz et al., 2013; Min et al., 2019). Finally, the fact combinations are translated into a complex question based on over 200 manually prepared question templates. We traversed all 26 selected data tables and generated 5,500 Type-I questions and 7,500 Type-II questions, respectively.

Specifically, to create the Type-I questions, we selected data facts via three question reasoning methods, i.e., comparison, intersection, and bridging, as introduced in (Min et al., 2019). In particular, the “comparison” type of questions compare facts within different data scopes based on the same measurement. The “intersection” type of questions seek for data elements that satisfy a number of conditions specified by different data facts. Finally, the “bridging” type of questions asks for a data fact that satisfies a prior condition specified by another.

To generate Type-II questions, we consider three forms of complex questions: (1) the questions without mentioning an analysis task (i.e., no fact type) ; (2) the questions without mentioning the aspect to be estimated (i.e., no measure) ; (3) the questions without mentioning data divisions (i.e., breakdown methods). Obviously these questions do not have a unique answer. Therefore, we choose all the facts that potentially answers such a complex question to help generate questions in these three forms.

Question Rephrasing. To produce high-quality natural language questions, we employed a group of native English speakers to rephrase and polish the generated questions manually. An online system was developed to split the job by randomly allocating 50 machine-generated questions to each participant. They were asked to fix the grammar errors and polish the questions into a natural language representation without changing their original meanings. Finally, 700 native English speakers were involved in our job and 35,000 rephrased questions were collected with an average cost of 0.12 USD per question.

Question Validation. To ensure a high-quality corpus, we validated the rephrased questions through a strict process. First, all the empty and short (less than 3 words) submissions are eliminated from the corpus. After that, from each of the 50 questions processed by a participant, we manually review a 10% question sample. Any problem found in the sample will result in an immediate rejection of all the questions rephrased by the same participant.

Finally, we checked both the semantic $S_{s}(\cdot)$ and text $S_{t}(\cdot)$ similarities between the machine-generated $q_{m}$ and the rephrased $q_{r}$ questions to eliminate the questions that simply copy the original sentence or greatly alter the original meaning based on the following metric:

where we employ sentence-BERT (Reimers and Gurevych, 2019) to project a machine-generated question $q_{m}$ and rephrased questions $q_{r}$ into the same vector space and estimate their $S_{s}(\cdot)$ based on the cosine-similarity between the corresponding vectors. We computed the text similarity based on the Levenshtein distance $D(q_{m},q_{r})$, which directly estimates the word differences between two sentences that is formally defined as:

Intuitively, a positive $S(s_{m},s_{r})$ score indicates $s_{r}$ and $s_{m}$ share the similar semantics but are different in the text representation, i.e., $s_{r}$ is a high-quality rephrasing of $s_{m}$. In opposite, a negative $S(\cdot)$ score indicates the two questions share a similar textual representation but have different meanings, which should be eliminated.

The first thing to retrieve an answer fact $f_{i}$ to a simple question $q_{i}$ is to figure out the analysis tasks that are mentioned or implied in a question, so that the fact type could be determined and the rest fields in the fact could be explored guiding by the fact type. To this end, we pre-trained a BERT-classifer to indicate the type of fact $f_{i}$ given a simple question $q_{i}$. We fine-tuned the BERT model by combining an additional classification layer and trained it on our corpus to classify the input simple question:

where $u$ is the intermediate vector encoded by the BERT model and $W_{t}$ is the trainable matrix in the classification layer. The output $t$ is a ten-dimensional one-hot vector that represents the type of answer fact $f_{i}$ to the input simple question $q_{i}$.

Given a preferred fact type ($t_{i}$), we employ the beam search algorithm (Ow and Morton, 1988) to determine the rest of fact fields, i.e., subspace ($s_{i}$), breakdown ($b_{i}$) , measure ($m_{i}$) and focus ($x_{i}$) by searching through the entire data space $X$. In particular, we use semantic similarities between the resulting facts $f_{i}$ and the input question $q_{i}$ as the heuristic function to guide the searching process with the goal of retrieving facts that are most relevant to the question. As shown in Fig 5 and summarized in Algorithm 1, the algorithm explores the data space $X$ to examine a number of candidate choices for each fact field step by step via a search tree $T$. In each searching step, the algorithm chooses a candidate value (e.g, a data column) for the corresponding fact field that makes the fact having the highest reward. Finally, the fact is determined by a path from the root to a leaf in the search tree.

The search tree $T$ is gradually generated through a searching process as described in Algorithm 1. In particular, the algorithm takes a simple question $q_{i}$, a fact type $t$, a tabular data $X$, and a beam width $k$ (the number of selected facts at each round) as the input and automatically generates a set of data facts $\mathcal{F}$ that are most relevant to the question. At the beginning, the type $t$ initializes a data fact $f_{initial}$ to confirm the rest fields in the fact, and use $f_{initial}$ as the root of the $T$ (line 1, Fig 5-b). In next, the algorithm generates data facts by iteratively searching the rest fields in facts via three major steps: selection, expansion, ranking. The first step selects $T_{top}$ that contains the highest $k$ facts ranked in $T$, from which the next expansion step will be performed (line 3, Fig 5-b1). The second step expands the $T$ by creating a set of data facts. (line 4 - 8, Fig 5-b2). The new data facts are generated by filling the field $p_{i}$ in $f_{i}$ with candidate values from $X$. The third step ranks the new facts in $T$ based on the semantic similarities between $f_{i}$ and question $q_{i}$ (line 10). The top $k$ facts ranked in $T$ are identified as the most relevant facts for the question $q_{i}$ (line 12).

To retrieve a set of facts $\mathcal{F}$ that are most relevant to the input simple question $q_{i}$, in each round of expansion, the facts $f_{i}$ in $T$ are ranked by their semantic similarity between $f_{i}$ and $q_{i}$. As facts $f_{i}$ are in the form of 5-tuple, we first use hand-written templates to transform facts into machine-generated questions. If the facts $f_{i}$ are not complete, questions will be generated based on existing fields. Then we employ the Sentence-BERT (Reimers and Gurevych, 2019) to project machine-generated questions and the input question into the same vector space and use the cosine-similarity between corresponding vectors to estimate their semantic similarity.

The user interface of the Talk2Data system consists of two views: (1) the data view (Fig. 6(a)) and (2) the answer view (Fig. 6(b)). In particular, the data view is designed to illustrate the raw data to users so that they could initiate a question. In particular, the data is shown in a data table (Fig. 6(a1)) whose columns are colored by the corresponding data types. A question panel (Fig. 6(a2)) is also provided in the view to display potential questions that could be asked about the data. When a data column is selected, the question list will be updated accordingly to show questions that are only relevant to the selected column. The answer view (Fig. 6(b)) represents data facts that answer the question via a library of annotated charts, the charts are arranged in order to facilitate the interpretation and narration of the answers. In particular, in this view, user’s question is represented as the title of view (Fig. 6(b1)), and decomposed questions are shown as sub-titles in each section that are answered by data facts (Fig. 6(b2)). Each data fact is visualized by an annotated chart with a narrative caption.

A floating tool bar (Fig. 6(a3, b3)) is designed and placed at the bottom of both views, through which users can upload the data, ask a question via both voice and text input, enter the edit mode for editing the answers, and switch between the data and answer views. We use the Web Speech API ²²2Web Speech API: https://developer.mozilla.org/Web/API/Web_Speech_API to convert audio to text in the web browser. Users can also enter the full-screen mode or covert the results into a PDF report by clicking the buttons in the up-right corner (Fig. 6(b4)).

In data storytelling, annotations in charts will help emphasize the information and avoid ambiguity (Lee et al., 2015). Therefore, we design a library of annotated statistical charts ³³3The chart library will be released after the review to display the data facts that answer the question with the goal of helping with the answer narration and interpretation. Our chart library consists of 5 types of basic statistic diagrams (bar chart, line chart, pie chart, area chart, and scatter plots) that are frequently used in data stories as summarized in  (Shi et al., 2021). To design the annotations, we further investigated a large number of relevant designs by exploring the design of the charts frequently used in over 200 data videos and over 1500 info-graphics. As a result, annotations involving text, colors, shapes, pointers, lines are designed for showing values, illustrating the trends and relations, highlighting anomalies and extremes, emphasizing differences and ranks, and differentiate categories and proportions. Applying annotations in the aforementioned 5 types of charts to represent different narrative semantics gives us 15 different annotated charts(Fig. 7). For example, we use dash-lines in a bar chart to emphasize difference but use trending lines in bar chart to illustrate trend, which result in two different annotated charts.

Despite the above annotations, caption is another crucial component in each of the annotated charts. It usually describes the important data patterns in a nature language to help users quickly capture the information shown in the chart. To generate the caption, we adopt the sentence template for each type of fact introduced in (Shi et al., 2020). Considering these templates may generate problematic descriptions that have grammar errors, our system enables a free editing function, through which users can easily edit the captions to fix the errors when necessary.

We display the answers to the input question in the form of a dashboard that could be easily displayed on a big screen to facilitate online discussions about the data in real-time. The annotated charts are arranged in order to facilitate reading and answer narration. In particular, we divide the screen into several regions and allocate to decomposed questions. Within each region, we arrange the charts in order according to their relevance to the corresponding questions, and place them one by one from left to right and top to bottom to facilitate reading. The size of each chart is determined by their relevance score to the question.

We performed a case study by inviting an expert from a business school to explore and analyzing a marking dataset about car sales records. The dataset consists of four dimensions: year, sales value, model, and brand (Fig. 1). The user started with a complex fuzzy question (i.e., Type-II) “How is the sales?”. The system automatically decomposed the question into three relevant simple questions: “which category has the highest sales?” (Q1.1), “what is the sales over years?” (Q1.2), and “what is the overall value of the sales?” (Q1.3), which are answered by three annotated charts showing the best selling model, the overall sales trend, and the total sales value as shown in Fig. 1.

Having the above overview of the data, the user would like to dig deeper into the data. He asked “does any brand sell a lot and have an increasing trend?” (i.e., Type-I). The system resolves the question into three relevant simple questions: (1) “what is the trend of sales?” (Q2.1), (2)“which brand has the highest/lowest sales?” (Q2.2), and (3) “which brand has an increasing trend of sales?” (Q2.3). The answers to these questions are shown in a group of charts as illustrated in Fig. 8, which respectively illustrates the sales trend, showing the highest sales record among different brands and the sales trend of each brand.

We estimate the quality of the question decomposition results based on the testing corpus via two frequently used metrics in nature language processing, i.e., BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005). These metrics are originally designed to estimate the quality of sentence translation. In particular, BLEU estimates the word-level translation precision based on the number of exactly matched words between the translated sentences and the ground-truth. METEOR computes a weighted F1-score to estimate the translation quality by comparing the translated sentences and ground-truth based on WordNet (Miller, 1995). These metrics are verified to be able to provide estimations that are consistent with humans’ judgments (Nema and Khapra, 2018). In our experiment, we use these matrices to estimate the decomposition quality by comparing the decomposed sub-questions to the corresponding targets in the testing corpus.

We estimate the performance of the proposed decomposition model by comparing it to three simplified versions that respectively have (1) no copying mechanism, (2) no copying and attention mechanisms, and (3) no copying, attention, and question type classification components. All these models were trained based on the question decomposition corpus under the same parameters settings. In particular, the training set, validation set, and evaluation set respectively takes 80%, 10%, and 10% of the corpus. We also involve a high-quality sentence rewriting technique as the baseline for comparison (Xiao et al., 2020).

The evaluation results are summarized in Table 2, which shows that (1) the sentence quality of our technique is equivalent to that of the high-quality sentence rewriting technique, and (2) our designs of the key components (question type classifier, attention, and copying mechanism) indeed improve the performance of the question decomposition model.

To estimate the usability of the system, we conducted a controlled within-subject study with 20 participants to make a comparison between Talk2Data and a baseline system developed based on NL4DV and vega-lite charts. The participants (13 female, 7 male, between 21 and 28 years old (M = 24.8, SD = 1.94)) are university students major in design and literature. They have limited knowledge about data analysis.

Two real datasets were used for the study. The first one describes 549 Amazon bestselling books (rows) from six dimensions (columns) including book title, rating, number of reviews, published year, price, and genre. The second dataset describes 275 car sales records from four dimensions including the sales value, brand, model, and the year. These two datasets were used in both the Talk2Data and the baseline system during the study in a counterbalanced order.

During the study, we first introduced the systems and let the participants to try it by their own. After the users were getting familiar with systems, they were asked to finish six tasks by asking relevant questions by their own. Three of these tasks were simple ones such as “find the distribution of ratings over books” but the other three were complex ones such as “find the most popular author”. During the experiment, we recorded the number of questions asked by a user to finish each task in each system. This number were together with the accuracy to estimate their performance. Finally, the participants were also asked to finish a post-study questionnaire. The study results are reported as follows:

Accuracy. As shown in Fig. 9(a), Talk2Data (M=95%, SD=0.16) and the baseline (M=95%, SD=0.12) had a similar accuracy when finding answers to low level questions. However, our system (M=87.6%, SD=0.17) significantly outperformed the baseline system (M=67.5%, SD=0.28) in case of resolving complex questions based on the paired-t test (t(19) = 4.08, p <0.01).

Efficiency. We compared the averaged number of questions that users need to finish each of the tasks to demonstrate a system’s efficiency. As shown in Fig. 9(b), when solving simple tasks, users explored a similar number of questions when using Talk2Data (M = 1.17,SD = 0.49) and the baseline (M = 1.32,SD = 0.76) system. However, for complex tasks, users obviously tend to ask more questions when using the baseline system (M = 2.63,SD = 1.53) comparing to that of Talk2Data (M = 1.62,SD = 1.11). The difference is significant regarding to the paired-t test (t(59) = 4.4, p <0.01).

Feedback. The results (Fig. 9(c)) of our post-study questionnaire showed that all the users preferred our system. Most of them mentioned “Talk2Data is a useful tool”, “it can greatly save one’s efforts when exploring the data”, and “the system is easy to use”. Many users also mentioned “dividing a complex question into simple ones and answer them one by one is an intuitive and effective way to solve the problem”.