EVE: Explainable Vector Based Embedding Technique Using Wikipedia

For a given “topical type”, we randomly choose four items belonging to one category and one intruder item from a different category of the same “topical type”. After repeating this process exhaustively for all combinations for all topical types, we generated 13,532,280 results for this task. Table 3 shows the breakdown of the total number of queries for each of the “topical types”.

For the “topical type” European cities, we randomly choose four related items from the “category” Great Britain such as London, Birmingham, Manchester, Liverpool, while we randomly choose an intruder item Berlin from the “category” Germany. Each of the models is presented with the five items, where the challenge is to identify Berlin as the intruder – the rest of the items are related to each other as they are cities in Great Britain, while Berlin is a city in Germany.

In order to discover the intruder item, we formulate the problem as a maximization of pairwise similarity across all items, the item receiving the least score is least similar to all other items, and thus identified as the intruder. Formally, for each model we compute

where the $similarity$ function is cosine similarity (Manning et al, 2008), $k$ and $i$ are the item positions in the list of items, and $item_{(k)}$ and $item_{(i)}$ are the vectors returned by the model under consideration.

To evaluate the effectiveness of the EVE model against the baselines for this task, we use $accuracy$ (Manning et al, 2008) as the measure for finding the intruder item. Accuracy is defined as the ratio of correct results (or correct number of intruder items) to the total number of results returned by the model:

Table 4 shows the experimental results for the six models in this task. From the table it is evident that the EVE model significantly outperforms rest of the models overall. However, in the case of two “topical types”, the FastText CBOW yields better results. To explain this, we next show explanations generated by the EVE model while making decisions for the intruder detection task.

Using the labeled dimensions in vectors produced by EVE, we define the process to generate effective explanations for the intruder detection task in Algorithm 1 as follows. The inputs to this algorithm are the vectors of items, and the intruder item identified by the EVE model. In step 1, we calculate the mean vector of all the vectors. In step 2 and 3, we subtract the influence of intruder and mean of vectors from each other to obtain dominant vector spaces to represent detected coherent items and intruder item respectively. In step 4 and 5, we order the labeled dimensions by their informativeness (i.e. the dimension with the highest score is the most informative dimension). Finally, we return a ranked list of informative vector dimensions for the both non-intruders and the intruder as an explanation for the output of the task.

Table 5 and 6 show sample explanations generated by the EVE model, where the model has detected a correct and incorrect intruder item respectively. In Table 5, the query has items selected from “topical type” animal class, where four of the items belong to the “category” birds, while the item ‘snake’ belongs to the “category” reptile. As can be seen from the table, the bold features in the non-intruder and intruder column obviously represent bird family and snake respectively, which is the correct inference. Furthermore, the non-bold features in the non-intruder and intruder columns represent deeper relevant relations which may require some domain expertise. For instance, falconiformes are a family of 60+ species in the order of birds and turonian is the evolutionary era of the specific genera.

In the example in Table 6, the query has items selected from the “topical type” movie genres, where four of the items belong to the “category” horror film, while the intruder item ‘Children of Men’ belongs to the “category” science fiction film. In this example, EVE identifies the wrong intruder item according to the ground truth, recommending instead the item ‘Final Destination (film)’. From the explanation in the table, it becomes clear why the model made this recommendation. We observe that the non-intruder items have a coherent relationship with ‘post-apocalyptic films’ and ‘films based on science fiction novels’ (both ‘I am Legend (film)’ and ‘Children of Men’ belong to these categories). Whereas ‘Final Destination (film)’ was recommended by the model based on features relating to filming location. A key advantage of having an explanation from the model is that it allows us to understand why a mistake occurs and how we might improve the model. In this case, one way to make improvement might be to add a rule filtering Wikipedia categories relating to locations when consider movie genres.

For all items in a specific “topical type”, we construct an embedding space without using information about the category to which the items belong. The purpose is then to measure the extent to which these items cluster together in the space relative to the ground truth categories. This is done by measuring distances in the space between items that should belong together (i.e. intra-cluster distances) and items that should be kept apart (i.e. inter-cluster distances), as determine by the categories. Since there are seven “topical types”, there are also even queries in this task.

For the “topical type” Cuisine, we are provided with a list of 100 items in total, where each of the five categories has 20 items. These correspond to cuisine items from five different countries. The idea is to measure the ability of each embedding model to cluster these 100 items back into five categories.

To formally measure the ability of a model to cluster items, we conduct a two-step strategy as follows:

1. 1.

   Calculate a pairwise similarity matrix between all items of a given “topical type”. The similarity function that we use for this task is the cosine similarity.

2. 2.

   Transform the similarity matrix to a distance matrix⁵⁵5by simply, 1 - normalized similarity score over each dimension which is used to measure inter and intra-cluster distances relative to the ground truth categories.

To evaluate the ability to cluster, there are typically two objectives: within-cluster cohesion and between-cluster separation. To this end, we use three well-known cluster validity measures in this task. Firstly, the within-cluster distance (Everitt et al, 2001) is the total of the squared distances between each item $x_{i}$ and the centroid vector $\mu_{c}$ of the cluster $C_{c}$ to which it has been assigned:

Typically this value is normalized with respect to the number of clusters $k$. The higher the score, the more coherent the clusters. Secondly, the between-cluster distance is the total of the squares of the distances between the each cluster centroid and the centroid of the entire dataset, denoted $\hat{\mu}$:

This value is also normalized with respect to the number of clusters $k$. The lower the score, the more well-separated the clusters. Finally, the two above objectives are combined via the CH-Index (Caliński and Harabasz, 1974), using the ratio:

The higher the value of this measure, the better the overall clustering.

From Table 7, we can see that EVE generally performs better than rest of the embedding methods for the within-cluster measure. In Table 8, for the between-cluster measure, EVE is outperformed by FastText CBOW, Word2Vec CBOW, and FastText SG mainly due to the “topical type” Cuisine and European cities where EVE does not perform well. Finally, in Table 9 where the combined aim of clustering is captured through the CH-Index, EVE outperforms the rest of the methods, except in the case of the “topical type” European cities.

Using labeled dimensions from the EVE model, we define a similar strategy for explanation as used in the previous task. However, now instead of discovering an intruder item, the goal is to define categories from items and to define the overall space. Algorithm 2 shows the strategy which requires three inputs: the $vector{space}$ representing the entire embedding; the list of categories $categories$; the $categories\_vector_{space}$ which is the vector space of items belonging to each category. In step 1, we calculate the mean vector representing for the entire space. In step 2, we order the labeled dimensions of the mean vector by the informativeness. In steps 3–6 we iterate over the list of categories (of a “topical type” such as Cuisine) and calculate mean vector for each category’s vector space, which is followed by the ordering of dimensions of the mean vector of category vector space by the informativeness. Finally, we return the most informative features of the entire space and of each category’s vector space.

Tables 10 and 11 show the explanations generated by the EVE model, in the cases where the model performed best and worse against baselines respectively. In Table 10, the query is the list of items from “topical type” cuisine. As can be seen from the bold entries in the table, the explanation conveys the main idea about both the overall space and the individual categories. For example, in the overall space, we can see the cuisines by different nationalities, and likewise we can see the name of nationality from which the cuisine is originated from (e.g. Italian cuisine for the “Italian category” and Pakistani breads for the “Pakistani category”). As for the non-bold entries, we can also observe relevant features but at a deeper semantic level. For example, cuisine of Lombardy in “Italian category” where Lombardy is a region in Italy, and likewise tortilla-based dishes in the Mexican category where tortilla is a primary ingredient in Mexican cuisine.

In Table 11, the query is the list of items from “topical type” European cities and this is the example where EVE model performs worse. However, the explanation allows us to understand why this is the case. As can been from the explanation table, the bold features show historic relationships across different countries, such as “capitals of former nations”, “fortified settlements”, and “Roman sites in Spain”. Similarly, it can also be observed in non-bold features such as “former capital of Italy”. Based on this explanation, we could potentially decide to apply a rule that would exclude any historical articles or categories when generating the embedding for this type of task in future.

Since scatter plots are often used to represent the output of a cluster analysis process, we generate a visualization of all embeddings using T-SNE (Maaten and Hinton, 2008), which is a tool to visually represent high-dimensional data by reducing it to 2–3 dimensions for presentation.⁶⁶6The full set of experimental visualizations is available at http://mlg.ucd.ie/eve/. For the interest of reader, Fig. 4 shows a visualization generated using EVE and GloVe when the list of items are selected from the “topical type” country to continent. As can be seen from the plot, the ground truth categories exhibit better clustering behavior when using the space from the EVE model, when compared to the Glove model. This is also reflected in the corresponding scores in Tables 7, 8, and 9.

The objective of this task is to rank a list of items based on their relevance to a given query item. According to the ground truth associated with our dataset, items which belong to the same ‘category’ of “topical type” as the query should be ranked above items which do not belong that ‘category’ (i.e. they are irrelevant to the query). In this task the total number of queries is equal to the total number of categories in the dataset – i.e. 36 (see table 1).

Unlike the previous tasks, here ‘category’ is used as a query in this task. For example, for the ‘category’ Nobel laureates in Physics, the task is to sort all items from “topical type” Nobel laureates such that the list of items from ‘category’ Nobel laureates in Physics are ranked ahead of the rest of the items. Thus, Niels Bohr, who is a laureate in Physics, should appear near the top of the ranking, unlike Elihu Root, who is a prize winner in Peace.

In order to sort items relevant to a category, we define a simple two-step strategy as follows:

1. 1.

   Calculate the cosine similarity between all items and a category belonging to “topical type” in the model space.

2. 2.

   Sort the list of items in descending order according to their similarity scores with the category.

Based on this strategy, a successful model should rank items with the same ‘category’ before irrelevant items.

We use precision-at-$k$ ($P@k$) and average precision ($AP$) (Manning et al, 2008) as the measures to evaluate the effectiveness of the sorting ability of each embedding model with respect to relevance of items to a category. $P@k$ captures how many relevant items are calculated at a certain rank (or in the $top-k$ results), while $AP$ captures how early a relevant item is retrieved on average. It may happen that two models have the same value of $P@k$, while one of the models retrieves relevant items in an earlier order of rank, thus achieving a higher $AP$ value. $P@k$ is defined as the ratio of relevant items retrieved in the $top-k$ retrieved items, whereas $AP$ is the average of $P@k$ values computed after each relevant item is retrieved. Equations 8 and 9 show the formal definitions of both measures.

Tables 12 and 13 show the experimental results of the sorting relevant items first task. We choose $P@20$ ($k=20$), since on average there are 20 items in each category in the dataset. As can be seen from tables, the EVE model generally outperforms the rest of models, except for the “topical type” European cities where it gets outperformed by a factor of 1.05 and 1.09 times in terms of $P@k$ and $AP$ respectively, while in all other cases EVE outperforms other algorithm by at least 1.51 and 1.37 times in terms of $P@k$ and $AP$ respectively. On average, the EVE model outperforms the second best algorithm by a factor of 1.8 and 1.67 times in terms of $P@k$ and $AP$ respectively. In the next section, we show the corresponding explanations generated by the EVE model for this task.

Using the labeled dimensions provided by the EVE model, we define a strategy for generating explanations for the sorting relevant items first task in Algorithm 3. The strategy requires three inputs. The first is the $vector_{space}$ which is composed of category vector and item vectors. The second input is the $Sim_{wrt\_category}$ which is a column matrix, composed of similarity score between the category vector with itself and item vectors. In this matrix the first entry is 1.0 because of the self similarity of the category vector. The final input is a list of items $items$. In the step 1 and 2, a weighted mean vector of space is calculated, where the weights are the similarity scores between the vectors in the space and the category vector. In steps 3–6, we iterate over the list of items and calculate the product between the weighted mean vector of the space and the item vector. After taking the product, we order the dimensions by the informativeness. Finally, we return the ranked list of informative features for each item.

Tables 14 and 15 show sample explanations generated by the EVE model. For illustration purposes we select the “topical types” Nobel laureates and Music genres for explanations, as these are the only remaining “topical types” which we have not looked at so far in the other tasks.

In Table 14, the query is ‘category’ Nobel laureates in Chemistry from the “topical type” nobel laureates. We show the informative features for two cases – the first correct result which appears at rank 1 in the sorted lists produced by EVE, and the first incorrect result which appears at rank 20. The bold features indicates that both individuals are Nobel laureates in Chemistry. However, Linus Pauling also appears to be associated with the Peace category. This reflects that fact that, in fact, Linus Pauling is a two time Nobel laureate in two different categories, Chemistry and Peace. While generating the dataset used in our evaluations, the annotators randomly selected items to belong to a category from the full set of available items, without taking into account occasional cases where an item may belong into two categories. This case highlights the fact that EVE explanations are meaningful and can inform the choices made by human annotators.

In Table 15, the query is ‘category’ Classical music from the “topical type” music genres. We see that the first correct result is observed at rank 1 and the first incorrect result is at rank 18. The bold features show that both individuals are associated with classical music. Looking at the biography of the musician Herbie Hancock more closely, we find that he received an education in classical music and he is also well known in the classical genre, although not as strongly as he is known for Jazz music. This again goes to show that explanations generated using the EVE model are insightful and can support the activity of manual annotators.