Ad Hoc Table Retrieval using Semantic Similarity

In the simplest case, all text associated with a given table is used as the table’s representation. This representation is then scored using existing retrieval methods, such as BM25 or language models.

Rather than collapsing all textual content into a single-field document, it may be organized into multiple fields, such as table caption, table headers, table body, etc. (cf. Sect. 2.2). For multi-field ranking, Pimplikar and Sarawagi (2012) employ a late fusion strategy (Zhang and Balog, 2017a). That is, each field is scored independently against the query, then a weighted sum of the field-level similarity scores is taken:

where $f_{i}$ denotes the $i$th (document) field for table $T$ and $w_{i}$ is the corresponding field weight (such that $\sum_{i}w_{i}=1$). $\mathit{score}(q,f_{i})$ may be computed using any standard retrieval method. We use language models in our experiments.

Query features have been shown to improve retrieval performance for document ranking (Macdonald et al., 2012). We adopt two query features from document retrieval, namely, the number of terms in the query (Tyree et al., 2011), and query IDF (Qin et al., 2010) according to: $\mathit{IDF}_{f}(q)=\sum_{t\in q}\mathit{IDF}_{f}(t)$, where $\mathit{IDF}_{f}(t)$ is the IDF score of term $t$ in field $f$. This feature is computed for the following fields: page title, section title, table caption, table heading, table body, and “catch-all” (the concatenation of all textual content in the table).

Table features depend only on the table itself and aim to reflect the quality of the given table (irrespective of the query). Some features are simple characteristics, like the number of rows, columns, and empty cells (Cafarella et al., 2008b; Bhagavatula et al., 2013). A table’s PMI is computed by calculating the PMI values between all pairs of column headings of that table, and then taking their average. Following (Cafarella et al., 2008b), we compute PMI by obtaining frequency statistics from the Attribute Correlation Statistics Database (ACSDb) (Cafarella et al., 2008a), which contains table heading information derived from millions of tables extracted from a large web crawl.

Another group of features has to do with the page that embeds the table, by considering its connectivity (inLinks and outLinks), popularity (pageViews), and the table’s importance within the page (tableImportance and tablePageFraction).

Features in the last group express the degree of matching between the query and a given table. This matching may be based on occurrences of query terms in the page title (qInPgTitle) or in the table caption (qInTableTitle). Alternatively, it may be based on specific parts of the table, such as the leftmost column (#hitsLC), second-to-left column (#hitsSLC), or table body (#hitsB). Tables are typically embedded in (web) pages. The rank at which a table’s parent page is retrieved by an external search engine is also used as a feature (yRank). (In our experiments, we use the Wikipedia search API to obtain this ranking.) Furthermore, we take the Mixture of Language Models (MLM) similarity score (Ogilvie and Callan, 2003) as a feature, which is actually the best performing method among the four text-based baseline methods (cf. Sect. 5). Importantly, all these features are based on lexical matching. Our goal in this paper is to also enable semantic matching; this is what we shall discuss in the next section.

It is a natural choice to simply use word tokens to represent query/table content. That is, $\{q_{1},\dots,q_{n}\}$ is comprised of the unique words in the query. As for the table, we let $\{t_{1},\dots,t_{m}\}$ contain all unique words from the title, caption, and headings of the table. Mind that at this stage we are only considering the presence/absence of words. During the query-table similarity matching, the importance of the words will also be taken into account (Sect. 3.3.1).

Many tables are focused on specific entities (Zhang and Balog, 2017b). Therefore, considering the entities contained in a table amounts to a meaningful representation of its content. We use the DBpedia knowledge base as our entity repository. Since we work with tables extracted from Wikipedia, the entity annotations are readily available (otherwise, entity annotations could be obtained automatically, see, e.g.,  (Venetis et al., 2011)). Importantly, instead of blindly including all entities mentioned in the table, we wish to focus on salient entities. It has been observed in prior work (Venetis et al., 2011; Bhagavatula et al., 2015) that tables often have a core column, containing mostly entities, while the rest of the columns contain properties of these entities (many of which are entities themselves). We write $E_{cc}$ to denote the set of entities that are contained in the core column of the table, and describe our core column detection method in Sect. 3.1.3. In addition to the entities taken directly from the body part of the table, we also include entities that are related to the page title ($T_{pt}$) and to the table caption ($T_{tc}$). We obtain those by using the page title and the table caption, respectively, to retrieve relevant entities from the knowledge base. We write $R_{k}(s)$ to denote the set of top-$k$ entities retrieved for the query $s$. We detail the entity ranking method in Sect. 3.1.4. Finally, the table is represented as the union of three sets of entities, originating from the core column, page title, and table caption: $\{t_{1},\dots,t_{m}\}=E_{cc}\cup R_{k}(T_{pt})\cup R_{k}(T_{tc})$.

To get an entity-based representation for the query, we issue the query against a knowledge base to retrieve relevant entities, using the same retrieval method as above. I.e., $\{q_{1},\dots,q_{n}\}=R_{k}(q)$.

We introduce a simple and effective core column detection method. It is based on the notion of column entity rate, which is defined as the ratio of cells in a column that contain an entity. We write $\mathit{cer}(T_{c[j]})$ to denote the column entity rate of column $j$ in table $T$. Then, the index of the core column becomes: $\arg\max_{j=1..T_{|c|}}\mathit{cer}(T_{c[j]})$, where $T_{|c|}$ is the number of columns in $T$.

We employ a fielded entity representation with five fields (names, categories, attributes, similar entity names, and related entity names) and rank entities using the Mixture of Language Models approach (Ogilvie and Callan, 2003). The field weights are set uniformly. This corresponds to the MLM-all model in (Hasibi et al., 2017b) and is shown to be a solid baseline. We return the top-$k$ entities, where $k$ is set to 10.

One alternative for moving from the lexical to the semantic space is to represent tables/queries using specific concepts. In this work, we use entities and categories from a knowledge base. These two semantic spaces have been used in the past for various retrieval tasks, in duet with the traditional bag-of-words content representation. For example, entity-based representations have been used for document retrieval (Xiong et al., 2017; Raviv et al., 2016) and category-based representations have been used for entity retrieval (Balog et al., 2011). One important difference from previous work is that instead of representing the entire query/table using a single semantic vector, we map each individual query/table term to a separate semantic vector, thereby obtaining a richer representation.

We use the entity-based raw representation from the previous section, that is, $t_{i}$ and $q_{j}$ are specific entities. Below, we explain how table terms $t_{j}$ are projected to $\vec{t_{i}}$, which is a sparse discrete vector in the entity/category space; for query terms it follows analogously.

Bag-of-entities:

Each element in $\vec{t}_{i}$ corresponds to a unique entity. Thus, the dimensionality of $\vec{t}_{i}$ is the number of entities in the knowledge base (on the order of millions). $\vec{t}_{i}[j]$ has a value of $1$ if entities $i$ and $j$ are related (there exists a link between them in the knowledge base), and $0$ otherwise.

Bag-of-categories:

Each element in $\vec{t}_{i}$ corresponds to a Wikipedia category. Thus, the dimensionality of $\vec{t}_{i}$ amounts to the number of Wikipedia categories (on the order hundreds of thousands). The value of $\vec{t}_{i}[j]$ is $1$ if entity $i$ is assigned to Wikipedia category $j$, and $0$ otherwise.

Recently, unsupervised representation learning methods have been proposed for obtaining embeddings that predict a distributional context, i.e., word embeddings (Mikolov et al., 2013; Pennington et al., 2014) or graph embeddings (Perozzi et al., 2014; Tang et al., 2015; Ristoski and Paulheim, 2016). Such vector representations have been utilized successfully in a range of IR tasks, including ad hoc retrieval (Ganguly et al., 2015; Mitra et al., 2016), contextual suggestion (Manotumruksa et al., 2016), cross-lingual IR (Vulić and Moens, 2015), community question answering (Zhou et al., 2015), short text similarity (Kenter and de Rijke, 2015), and sponsored search (Grbovic et al., 2015). We consider both word-based and entity-based raw representations from the previous section and use the corresponding (pre-trained) embeddings as follows.

Word embeddings:

We map each query/table word to a word embedding. Specifically, we use word2vec (Mikolov et al., 2013) with 300 dimensions, trained on Google News data.

Graph embeddings:

We map each query/table entity to a graph embedding. In particular, we use RDF2vec (Ristoski and Paulheim, 2016) with 200 dimensions, trained on DBpedia 2015-10.

The first idea is to represent the query and the table each with a single vector. Their similarity can then simply be expressed as the similarity of the corresponding vectors. We let $\vec{C}_{q}$ be the centroid of the query term vectors ($\vec{C}_{q}=\sum_{i=1}^{n}\vec{q}_{i}/n$). Similarly, $\vec{C}_{T}$ denotes the centroid of the table term vectors. The query-table similarity is then computed by taking the cosine similarity of the centroid vectors. When query/table content is represented in terms of words, we additionally make use of word importance by employing standard TF-IDF term weighting. Note that this only applies to word embeddings (as the other three semantic representations are based on entities). In case of word embeddings, the centroid vectors are calculated as $\vec{C}_{T}=\sum_{i=1}^{m}\vec{t}_{i}\times TFIDF(t_{i})$. The computation of $\vec{C}_{q}$ follows analogously.

Instead of combining all semantic vectors $q_{i}$ and $t_{j}$ into a single one, late fusion computes the pairwise similarity between all query and table vectors first, and then aggregates those. We let $S$ be a set that holds all pairwise cosine similarity scores: $S=\{\cos(\vec{q}_{i},\vec{t}_{j}):i\in[1..n],j\in[1..m]\}$. The query-table similarity score is then computed as $\mathrm{aggr}(S)$, where $\mathrm{aggr}()$ is an aggregation function. Specifically, we use $\max()$, $\mathrm{sum}()$ and $\mathrm{avg}()$ as aggregators; see the last three rows in Table 3 for the equations.

Figure 4 shows the importance of individual features for the table retrieval task, measured in terms of Gini importance. The novel features are distinguished by color. We observe that 8 out of the top 10 features are semantic features introduced in this paper.

To analyze how the four semantic representations affect retrieval performance on the level of individual queries, we plot the difference between the LTR baseline and each semantic representation in Figure 5. The histograms show the distribution of queries according to NDCG@20 score difference ($\Delta$): the middle bar represents no change ($\Delta<$0.05), while the leftmost and rightmost bars represents the number of queries that were hurt and helped substantially, respectively ($\Delta>$0.25). We observe similar patterns for the bag-of-entities and word embeddings representations; the former has less queries that were significantly helped or hurt, while the overall improvement (over all topics) is larger. We further note the similarity of the shapes of the distributions for bag-of-categories and graph embeddings.

On Figure 6, we plot the results for the LTR baseline and for our STR method according to the two query subsets, QS-1 and QS-2, in terms of NDCG@20. Generally, both methods perform better on QS-1 than on QS-2. This is mainly because QS-2 queries are more focused (each targeting a specific type of instance, with a required property), and thus are considered more difficult. Importantly, STR achieves consistent improvements over LTR on both query subsets.

We plot the difference between the LTR baseline and STR for the two query subsets in Figure 7. Table 7 lists the queries that we discuss below. The leftmost bar in Figure 7(a) corresponds to the query “stocks.” For this broad query, there are two relevant and one highly relevant tables. LTR does not retrieve any highly relevant tables in the top 20, while STR manages to return one highly relevant table in the top 10. The rightmost bar in Figure 7(a) corresponds to the query “ibanez guitars.” For this query, there are two relevant and one highly relevant tables. LTR produces an almost perfect ranking for this query, by returning the highly relevant table at the top rank, and the two relevant tables at ranks 2 and 4. STR returns a non-relevant table at the top rank, thereby pushing the relevant results down in the ranking by a single position, resulting in a decrease of 0.29 in NDCG@20.

The leftmost bar in Figure 7(b) corresponds to the query “board games number of players.” For this query, there are only two relevant tables according to the ground truth. STR managed to place them in the 1st and 3rd rank positions, while LTR returned only one of them at position 13th. The rightmost bar in Figure 7(b) is the query “cereals nutritional value.” Here, there is only one highly relevant result. LTR managed to place it in rank one, while it is ranked eighth by STR. Another interesting query is “irish counties area” (third bar from the left in Figure 7(b)), with three highly relevant and three relevant results according to the ground truth. LTR returned two highly relevant and one relevant results at ranks 1, 2, and 4. STR, on the other hand, placed the three highly relevant results in the top 3 positions and also returned the three relevant tables at positions 4, 6, and 7.