Relational Word Embeddings

Our goal here is to learn relation vectors for closely related words. For both the selection of the vocabulary and the method to learn relation vectors we mainly follow the initialization method of Camacho-Collados et al. (2019, Relative_init) except for an important difference explained below regarding the symmetry of the relations. Other relation embedding methods could be used as well, e.g., Jameel et al. (2018); Washio and Kato (2018b); Espinosa Anke and Schockaert (2018); Joshi et al. (2019), but this method has the advantage of being highly efficient. In the following we describe this procedure for learning relation vectors: we first explain how a set of potentially related word pairs is selected, and then focus on how relation vectors $\mathbf{r_{wv}}$ for these word pairs can be learned.

Specifically, to construct the relation vector $\mathbf{r_{wv}}$ capturing the relationship between the words $w$ and $v$ we proceed as follows. First, we compute a bag of words representation $\{(w_{1},f_{1}),...,(w_{n},f_{n})\}$, where $f_{i}$ is the number of times the word $w_{i}$ occurs in between the words $w$ and $v$ in any given sentence in the corpus. The relation vector $\mathbf{r_{wv}}$ is then essentially computed as a weighted average:

where we write $\mathbf{w_{i}}$ for the vector representation of $w_{i}$ in some given pre-trained word embedding, and $\textit{norm}(\mathbf{v})=\frac{\mathbf{v}}{\|\mathbf{v}\|}$.

In contrast to other approaches, we do not differentiate between sentences where $w$ occurs before $v$ and sentences where $v$ occurs before $w$. This means that our relation vectors are symmetric in the sense that $\mathbf{r_{wv}}=\mathbf{r_{vw}}$. This has the advantage of alleviating sparsity issues. While the directionality of many relations is important, the direction can often be recovered from other information we have about the words $w$ and $v$. For instance, knowing that $w$ and $v$ are in a capital-of relationship, it is trivial to derive that “$v$ is the capital of $w$”, rather than the other way around, if we also know that $w$ is a country.

The relation vectors $\mathbf{r_{wv}}$ capture relational information about the word pairs in $\mathcal{R}$. The relational word vectors will be induced from these relation vectors by encoding the requirement that $\mathbf{e_{w}}$ and $\mathbf{e_{v}}$ should be predictive of $\mathbf{r_{wv}}$, for each $(w,v)\in\mathcal{R}$. To this end, we use a simple neural network with one hidden layer,⁴⁴4More complex architectures could be used, e.g., Joshi et al. (2019), but in this case we decided to use a simple architecture as the main aim of this paper is to encode all relational information into the word vectors, not in the network itself. whose input is given by $(\mathbf{e_{w}}+\mathbf{e_{v}})\oplus(\mathbf{e_{w}}\odot\mathbf{e_{v}})$, where we write $\oplus$ for vector concatenation and $\odot$ for the component-wise multiplication. Note that the input needs to be symmetric, given that our relation vectors are symmetric, which makes the vector addition and component-wise multiplication two straightforward encoding choices. Figure 1 depicts an overview of the architecture of our model. The network is defined as follows:

for some activation function $f$. We train this network to predict the relation vector $\mathbf{r_{wv}}$, by minimizing the following loss:

The relational word vectors $\mathbf{e_{w}}$ can be initialized using standard word embeddings trained on the same corpus.

Given a pre-defined set of relation types and a pair of words, the relation classification task consists in selecting the relation type that best describes the relationship between the two words. As test sets we used DiffVec Vylomova et al. (2016) and BLESS⁸⁸8http://clic.cimec.unitn.it/distsem Baroni and Lenci (2011). The DiffVec dataset includes 12,458 word pairs, covering fifteen relation types including hypernymy, cause-purpose or verb-noun derivations. On the other hand, BLESS includes semantic relations such as hypernymy, meronymy, and co-hyponymy.⁹⁹9Note that both datasets exhibit overlap in a number of relations as some instances from DiffVec were taken from BLESS. BLESS includes a train-test partition, with 13,258 and 6,629 word pairs, respectively. This task is treated as a multi-class classification problem

As a baseline model (Diff), we consider the usual representation of word pairs in terms of their vector differences Fu et al. (2014); Roller et al. (2014); Weeds et al. (2014), using FastText word embeddings. Since our goal is to show the complementarity of relational word embeddings with standard word vectors, for our method we concatenate the difference $\mathbf{w_{j}}-\mathbf{w_{i}}$ with the vectors $\mathbf{e_{i}}+\mathbf{e_{j}}$ and $\mathbf{e_{i}}\cdot\mathbf{e_{j}}$ (referred to as the Mult+Avg setting; our method is referred to as rwe). We use a similar representation for the other methods, simply replacing the relational word vectors by the corresponding vectors (but keeping the FastText vector difference). We also consider a variant in which the FastText vector difference is concatenated with $\mathbf{w_{i}}+\mathbf{w_{j}}$ and $\mathbf{w_{i}}\cdot\mathbf{w_{j}}$, which offers a more direct comparison with the other methods. This goes in line with recent works that have shown how adding complementary features on top of the vector differences, e.g. multiplicative features Vu and Shwartz (2018), help improve the performance. Finally, for completeness, we also include variants where the average $\mathbf{e_{i}}+\mathbf{e_{j}}$ is replaced by the concatenation $\mathbf{e_{i}}\oplus\mathbf{e_{j}}$ (referred to as Mult+Conc), which is the encoding considered in Joshi et al. (2019).

For these experiments we train a linear SVM classifier directly on the word pair encoding, performing a 10-fold cross-validation in the case of DiffVec, and using the train-test splits of BLESS.

Standard word embedding models tend to capture semantic similarity rather well Baroni et al. (2014); Levy et al. (2015a). However, even though other kinds of lexical properties may also be encoded Gupta et al. (2015), they are not explicitly modeled. Based on the hypothesis that relational word embeddings should allow us to model such properties in a more consistent and transparent fashion, we select the well-known McRae Feature Norms benchmark McRae et al. (2005) as testbed. This dataset¹⁰¹⁰10Downloaded from https://sites.google.com/site/kenmcraelab/norms-data is composed of 541 words (or concepts), each of them associated with one or more features. For example, ‘a bear is an animal’, or ‘a bowl is round’. As for the specifics of our evaluation, given that some features are only associated with a few words, we follow the setting of Rubinstein et al. (2015) and consider the eight features with the largest number of associated words. We carry out this evaluation by treating the task as a multi-class classification problem, where the labels are the word features. As in the previous task, we use a linear SVM classifier and perform 3-fold cross-validation. For each input word, the word embedding of the corresponding feature is fed to the classifier concatenated with its baseline FastText embedding.

Given that the McRae Feature Norms benchmark is focused on nouns, we complement this experiment with a specific evaluation on verbs. To this end, we use the verb set of QVEC¹¹¹¹11https://github.com/ytsvetko/qvec Tsvetkov et al. (2015), a dataset specifically aimed at measuring the degree to which word vectors capture semantic properties which has shown to strongly correlate with performance in downstream tasks such as text categorization and sentiment analysis. QVEC was proposed as an intrinsic evaluation benchmark for estimating the quality of word vectors, and in particular whether (and how much) they predict lexical properties, such as words belonging to one of the fifteen verb supersenses contained in WordNet Miller (1995). As is customary in the literature, we compute Pearson correlation with respect to these predefined semantic properties, and measure how well a given set of word vectors is able to predict them, with higher being better. For this task we compare the 300-dimensional word embeddings of all models (without concatenating them with standard word embeddings), as the evaluation measure only assures a fair comparison for word embedding models of the same dimensionality.

First, we provide an analysis based on the nearest neighbours of selected words in the vector space. Table 4 shows nearest neighbours of our relational word vectors and the standard FastText embeddings.¹³¹³13Recall from Section 4 that both were trained on Wikipedia with the same dimensionality, i.e., 300. The table shows that our model captures some subtle properties, which are not normally encoded in knowledge bases. For example, geometric shapes are clustered together around the sphere vector, unlike in FastText, where more loosely related words such as “dimension” are found. This trend can easily be observed as well in the philology and assassination cases.

In the bottom row, we show cases where relational information is somewhat confused with collocationality, leading to undesired clusters, such as intersect being close in the space with “tracks”, or behaviour with “aggressive” or “detrimental”. These examples thus point towards a clear direction for future work, in terms of explicitly differentiating collocations from other relationships.

Unsupervised learning of analogies has proven to be one of the strongest selling points of word embedding research. Simple vector arithmetic, or pairwise similarities Levy et al. (2014), can be used to capture a surprisingly high number of semantic and syntactic relations. We are thus interested in exploring semantic clusters as they emerge when encoding relations using our relational word vectors. Recall from Section 3.2 that relations are encoded using addition and point-wise multiplication of word vectors.

Table 4 shows, for a small number of selected word pairs, the top nearest neighbors that were unique to our 300-dimensional relational word vectors. Specifically, these pairs were not found among the top 50 nearest neighbors for the FastText word vectors of the same dimensionality, using the standard vector difference encoding. Similarly, we also show the top nearest neighbors that were unique to the FastText word vector difference encoding. As can be observed, our relational word embeddings can capture interesting relationships which go beyond what is purely captured by similarity. For instance, for the pair “innocent-naive” our model includes similar relations such as vain-selfish, honest-hearted or cruel-selfish as nearest neighbours, compared with the nearest neighbours of standard FastText embeddings which are harder to interpret.

Interestingly, even though not explicitly encoded in our model, the table shows some examples that highlight one property that arises often, which is the ability of our model to capture co-hyponyms as relations, e.g., wrist-knee and anger-despair as nearest neighbours of “shoulder-ankle” and “shock-grief”, respectively. Finally, one last advantage that we highlight is the fact that our model seems to perform implicit disambiguation by balancing a word’s meaning with its paired word. For example, the “oct-feb” relation vector correctly brings together other month abbreviations in our space, whereas in the FastText model, its closest neighbour is ‘doppler-wheels’, a relation which is clearly related to another sense of oct, namely its use as an acronym to refer to ‘optical coherence tomography’ (a type of x-ray procedure that uses the doppler effect principle).

One of the main problems of word embedding models performing lexical inference (e.g. hypernymy) is lexical memorization. Levy et al. (2015b) found that the high performance of supervised distributional models in hypernymy detection tasks was due to a memorization in the training set of what they refer to as prototypical hypernyms. These prototypical hypernyms are general categories which are likely to be hypernyms (as occurring frequently in the training set) regardless of the hyponym. For instance, these models could equally predict the pairs dog-animal and screen-animal as hyponym-hypernym pairs. To measure the extent to which our model is prone to this problem we perform a controlled experiment on the lexical split of the HyperLex dataset Vulić et al. (2017). This lexical split does not contain any word overlap between training and test, and therefore constitutes a reliable setting to measure the generalization capability of embedding models in a controlled setting Shwartz et al. (2016). In HyperLex, each pair is provided by a score which measures the strength of the hypernymy relation.

For these experiments we considered the same experimental setting as described in Section 4. In this case we only considered the portion of the HyperLex training and test sets covered in our vocabulary¹⁴¹⁴14Recall from Section 4 that this vocabulary is shared by all comparison systems. and used an SVM regression model over the word-based encoded representations. Table 5 shows the results for this experiment. Even though the results are low overall (noting e.g. that results for the random split are in some cases above 50% as reported in the literature), our model can clearly generalize better than other models. Interestingly, methods such as Retrofitting and Attract-Repel perform worse than the FastText vectors. This can be attributed to the fact that these models have been mainly tuned towards similarity, which is a feature that loses relevance in this setting. Likewise, the relation-based embeddings of Pair2Vec do not help, probably due to the high-capacity of their model, which makes the word embeddings less informative.