Multiple Word Embeddings for Increased Diversity of Representation

Much of the recent work in NLP has focused on better feature representations via contextual word embeddings Peters et al. (2018, 2017); Radford et al. (2018); Akbik et al. (2018); Devlin et al. (2019). These models vary in architecture and pre-training objective but they all encode the input based on the surrounding context in some way. These papers normally compare to baselines like a bidirectional LSTM-CRF (biLSTM-CRF) where words are represented by a single pre-trained word embedding.

Peters et al. (2018, 2017) and Akbik et al. (2018) pre-train large language models based on LSTMs. Task-specific architectures are then built on top of these pre-trained models. Peters et al. (2018) introduce a technique for extracting word representations as a linear combination of layers in the pre-trained model. Gradient updates are only applied to this weighting factor, which simplifies the training to some extent, but forward propagation is still required for the full network which makes the model slow to train and evaluate.

Radford et al. (2018), followed by Devlin et al. (2019), pre-train deep transformers Vaswani et al. (2017) on massive corpora. They both use a simple output layer on top of the pre-trained model and tune the parameters of the whole model. In this case, training requires the forward and backward pass of the entire pre-trained model, which has a significant impact on size and speed. Devlin et al. (2019) used specialized hardware which may be unrealistic for many inference scenarios.

The prevailing wisdom is that, because these pre-trained models are contextual, they can create representations of a word that is different in different contexts. For example, a polysemous word can be represented by different vectors when its context suggests a different sense of a word, while context-independent word vectors need to represent a mix of all the senses of a word. The majority of NLP models have a similar “contextualization” step, typically done via a biLSTM, convolutional layers, or self-attention, but it is only learned from a smaller, task-specific corpus in contrast to the massive corpora used by contextual embeddings.

Contextual embeddings and transfer learning architectures are slow to train and evaluate, which may make them infeasible for many types of deployments. Using multiple pre-trained embeddings trained on different datasets, we can exploit the bias in different datasets that results in different representations of the same word. By combining these embeddings, we can create richer representations of the word without the high computational overhead required by contextual alternatives. We find that the concatenation of multiple pre-trained word embeddings show consistent improvements over single embeddings yielding results much closer to contextual alternatives.

We use three sequential prediction tasks to test the performance of our concatenated embeddings: NER (CoNLL 2003 Tjong Kim Sang and De Meulder (2003), WNUT-17 Derczynski et al. (2017), and OntoNotes Hovy et al. (2006)), Slot filling (Snips Coucke et al. (2018)) and POS tagging (TW-POS Gimpel et al. (2011)). We also show results on three classification datasets: SST2 Socher et al. (2013), Snips intent classification Coucke et al. (2018), and AG-News¹¹1http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html. For each (task, dataset) pair we use the most common embedding used in literature, for example, GloVe embeddings were used for CONLL 2003 in Ma and Hovy (2016) and Senna embeddings in Chiu and Nichols (2016); Peters et al. (2018). Embeddings were also chosen based on how well the embedding training data fit the task, i.e., we used GloVe vectors trained on twitter for the twitter part of speech tagging task. Once we developed tests for which embeddings worked together in Section 3 we checked if there were any more embeddings combinations we should try but did not find any additional combinations. For all tagging tasks, a biLSTM-CRF model with convolutional character compositional inputs, following Ma and Hovy (2016), is used. For all classification tasks, a single layer LSTM model is used except for the Snips classification dataset, where a convolutional word-based model Kim (2014) is used. The hyperparameters are omitted here for brevity but can be found in our implementation.

The results are presented in Table 1. 6B, 27B and 840B are well-known, pre-trained GloVe embeddings Pennington et al. (2014) distributed via the authors site, w2v-30M Pressel et al. (2018) and GN Mikolov et al. (2013) are Word2Vec embeddings trained on a corpus of 30 million tweets and Google News respectively, and the Senna embeddings were trained by Collobert et al. (2011).

We leverage multiple pre-trained embeddings in a model by creating one embeddings table per pre-trained embedding. Each input token in embedded into each vector space and the resulting vectors are concatenated into a single vector. This means that it is possible for there to be a type that is unattested in one pre-trained embedding vocabulary but present in the other. This results in a pre-trained vector from one embedding being concatenated with a randomly initialized vector form the other embedding space.

As hypothesized, we see improvements across tasks, datasets, and model architectures when using multiple embeddings.

Models using the concatenation of pre-trained and randomly initialized embeddings do $0.6$% worse on average compared to models that only use a single pre-trained embedding. This demonstrates that the performance gains are from the combination of different pre-trained embeddings rather than the increase in the number of parameters in the model. In some cases we were able to improve results further by adding several sets of additional embeddings.

Table 2 summarizes the results of using the multiple embedding approach on internal datasets. These datasets are drawn from the tasks defined earlier and span a variety of specialized domains. Due to the nature of the datasets the results are presented as the relative change in performance. Table 3 is provided to help frame the relative performance numbers from the internal datasets.

The models were trained with MEAD/Baseline Pressel et al. (2018), an open-source framework for developing, training, and deploying NLP models.

There are three logical places where the observed improvements could come from. 1) The use of multiple pre-trained embeddings creates a slightly larger model, increasing the network capacity—the embeddings are larger and therefore the projection from the embeddings to the first layer of the model will also be slightly bigger. 2) The use of a second pre-trained embedding increases the vocabulary size and more words are attested. A word that has a pre-trained representation will start the model in a better spot than a randomly initialized representation. 3) The second set of pre-trained embeddings gives a different perspective of the words. Most pre-trained embeddings are trained on different data and encode different biases and senses into the embedding that reflect the quirks and unique contexts found in the pre-training data. This representational diversity will allow a model to capitalize on different senses, or the combination of senses, that would not be present when using a single embedding.

In order to tease apart which of these factors are at play we designed a series of models that aim to isolate each effect and report results in Table 4. First, we train a model that uses a single pre-trained embedding and a second set of vectors that are initialized randomly. If the main improvement is due to increased model capacity this configuration should perform well. The second model uses a special version of the second pre-trained embedding where we remove all the words that already appear in the original pre-trained vocabulary. In this second set of embeddings, randomly initialized vectors are used for the words that are covered in the original vocabulary in order to keep the embeddings size consistent with the previous model. If the main reason for improvement is the increased vocabulary coverage, this model should perform well. The final version of the model also uses a customized version of the second pre-trained embedding. This time we only keep embeddings that are already represented in the original vocabulary. This is designed to test if the main source of improvement is the difference in the representations each pre-trained embedding brings to the table.

From our ablation studies using the above variations on both the SST2 and CoNLL datasets and find that the most important thing is the representational diversity in the pre-trained embeddings. This dovetails nicely with our observation that embeddings trained on distinct datasets tend to perform well together. To further test this hypothesis, we look at the “similarity” of various pre-trained embeddings. We define “similarity” using the overlap of nearest neighbors in the embedding space as in Wendlandt et al. (2018). Specificly we use the average Jaccard overlap percentage between the 10 nearest neighbors for each of the top 200 words in the dataset by frequency. Table 5 shows the overlap of different embeddings with the Glove 6B 100 dimension embedding and how their combination affects the performance. As it can be seen, Senna has the lowest overlap and causes the biggest performance gain.

However, this does not hold for the GoogleNews embedding which also has a low overlap yet the combination actually causes a drop in performance. This can be explained by coverage—the percentage of unique types in the data that are attested in the pre-trained vocabulary. That number is surprisingly low for GoogleNews and causes the GoogleNews representations to be used so rarely they actually cause a drop in performance.

In summary, one should look for two characteristics when combining embeddings: the word representations should have low “similarity” and the unique types in the dataset should be highly attested in both pre-trained vocabularies.

Recent large-scale, contextual, pre-trained models are exciting but produce relatively slow models. We propose a simple, lightweight technique: concatenation of pre-trained embeddings. We show that this technique has a significant impact on error reduction and a negligible effect of speed.

However, the concatenation on any two random pre-trained embeddings is not guaranteed to work well. From our analysis, we are able to suggest a recipe for finding an effective combination: there should be a high degree of coverage of the unique types in each of the pre-trained embedding vocabularies and the word vectors should exhibit representational diversity. In future work, we intend to try other methods of embeddings combination while remaining computationally cheap. We also plan to find more principled ways to quantify the diversity in pre-trained embeddings, which can suggest ways to induce representational diversity into the embedding pre-training procedure itself.