Infusing Collaborative Recommenders
with Distributed Representations

The Continuous Bag of Words model (CBOW) is a neural network composed from an input, projection, and output layer [17]. Input words are represented using one-hot encoding, where the dimensionality of the input vector is the number of words in the vocabulary, $V$. The projection layer is of dimensionality $R$, where $R$ is the dimensionality of the target distributed representation. The input to projection layer weight matrix (of dimensionality $V\times R$) is shared across input words, so order does not matter—all words are projected into the same space. The output layer is of dimension $V$.

The training of a CBOW model is optimized to predict the one-hot representation of a target word $w_{k}$ given the surrounding context words in a sentence $w_{k-c}$, $...$, $w_{k-1}$, $w_{k+1}$, $...$, $w_{k+c}$, where $c$ is a fixed window size generally on the order of a small positive integer. Each word in a sentence is sequentially treated as a target word. Multiple sentences comprise the data used to train a CBOW model. The flow of data is described in Figure 1. The one-hot encodings of the context words are averaged and multiplied by the projection matrix to form the hidden layer activations.

The model is trained by optimizing the likelihood function formed by feeding the output layer activations into the softmax function below, where $\textbf{w}_{c}$ is the collection of context words, and $\sigma(k)$ is the output activation from the $k$th word in the vocabulary:

The error vector is produced by differencing the one-hot encoding and the vector produced by the softmax function. Training then proceeds by backpropagation, generally using a method like stochastic gradient descent for optimization [26, 17].

The Skip-gram model, also described in Figure 1, is structured in a way similar to the CBOW model, except the inferential task is reversed: given the target word $v_{k}$, the Skip-gram model is optimized to predict the surrounding context words. The dimensionality of the output layer and input layer are switched, and the hidden to output layer weight matrix shares weights for every output word vector representation. The softmax function acts on the output layer activations for each context word $w_{c,i}$ fed into the network. The softmax functions takes the following form, where the $w_{c,i}$ is the $i$th context word for the target word $w_{k}$, and $\sigma(c,i)$ is the activation corresponding to $w_{c,i}$:

The error vector for a set of output activations is computed by summing the difference vectors for each word activation in the output layer.

Various schemes have been used to address the significant computational overhead of calculating the updates for each of the output vector representations in the hidden to output weight matrix every time a word is run through a CBOW or Skip-gram model. Two are commonly used in practice: hierarchical softmax and negative sampling [17]. Each method has a unique effect on the accuracy of the neural network, and some empirical work has been done as to understanding which one performs best in a given predictive task [18].

Hierarchical softmax uses a binary tree to efficiently represent words in a vocabulary [21, 20]. Instead of using a matrix of vectors to represent words, each word is instead represented as a path from the root to a unique leaf in the tree, and normalization can be computed in $O(\log V)$ time. Similar words are clustered using simple word-feature algorithms in order to make updates computationally efficient.

Negative sampling is inspired by noise contrastive estimation [10]. It deals with the update problem by only updating a sample of the output vectors: the updated word, along with a number of negative samples. A noise distribution, $P_{n}(w)$ is used to provide the samples for the current word vector representation being updated. An appropriate distribution is chosen empirically; for example, a unigram distribution raised to the power of $\frac{3}{4}$ has been used in practice [18]. Updates for backpropagation are derived from the gradients of a loss function based on the noise distribution.

There exist many ways to construct user representations in a web application including recency, authority, linkage or vector space models. In this work we focus on the vector space representation [27] and describe users as:

When rated items are used to represent the user, these weights can be taken as the user’s ratings for the items. Similarly, items can be represented as a vector over the user space and the weights are taken as the ratings users have given the item:

In this work, we compute alternative user and item representations using the Continuous Bag of Words and Skip-gram models. A user representation takes the same form:

User-based collaborative filtering (UBCF) is a commonly used recommendation algorithm. Given a user $u$ and item $i$ a predicted rating $p(u,i)$ is produced by identifying a neighborhood, $N$, of the $k$ most similar users and leveraging their ratings, $r(n,i)$, on the item:

The influence of the neighbors in the final prediction is weighted by their similarity to the query user, nearest neighbors receiving more importance.

Item-Based Collaborative Filtering (IBCF) computes the similarity between items rather than between users. It predicts a user’s rating for an item by exploiting ratings that the user gave to similar items:

We extend user-based and item-based collaborative filtering. The algorithms work as before except that the user and item representations are substituted with those learned from the Continuous Bag of Words and Skip-gram models.

When computing the similarity between users based on the Continuous Bag of Words representations, we abbreviate the model as UBCB. When representing users as the representations from the Skip-gram model we abbreviate it as UBSG.

Likewise, two extensions of item-based collaborative filtering, IBCB and IBSG, are produced by replacing the item representation with those learned in the Continuous Bag of Words and Skip-gram models.

The MovieLens 10M dataset [25] is made available from GroupLens Research Group. Released in 2009, it contains 10 million ratings and 100,000 tag applications applied to 10,000 movies by 72,000 users. After identifying movies in the MovieLens data, we extracted the actors, directors, and other metadata from the Internet Movie Database (IMDb) ¹¹1www.imdb.com.

In order to generate distributed representations, we first merged these two datasets to create artificial sentences. The sentences included a user ID, movie ID, set of tags applied by the user to the movie, director, and leading actors. These sentences were input into the Continuous Bag of Words and Skip-gram models

Each user profile was divided equally across five partitions. Four folds were used as training to build the recommenders. The fifth was used to tune the model hyperparameters including the size of the user and item neighborhoods, the length of the distributed representations, the number of epochs to train the representations and the optimal weights for the linear weighted hybrid. The CBOW and Skip-gram models were both trained using Negative Sampling.

The fifth fold was then discarded, so that the hyperparameters tuned on the fifth fold would not be used to create predictions for it. Four-fold cross-validation was then used with the tuned hyperparameters to create predictions for the remaining folds. The results were averaged across all four folds.

The recommenders were evaluated based on their ability to predict a user’s affinity for a movie. Possible metrics include mean absolute error, recall or precision. In this work, we report results using root mean square error (RMSE).

For each example, in the testing data $T$ consisting of a user $u$, item $i$ and rating $r(u,i)$, we pass the user and item to a recommender which returns a prediction $p(u,i)$. The RMSE is then computed as:

By squaring the difference between the observation and the prediction, RMSE amplifies and severely punishes large errors.

As baseline models, we use the predictions from the traditional UBCF and IBCF models outlined above.

By training distributed representations on the MovieLens and IMDb dataset, we hope to capture meaningful relationships between users, movies, directors, actors, and tags. Consider two similar movies. They may be similar with regard to the actors and directors of the movie, with regard to the tags users have assigned them, with regard to the users that have watched them, or with regard to all of these dimensions. If two movies are perceived by users to be similar, then their learned representations should also be similar.

For example in Table 3 we have listed the five most similar movies, directors, actors and tags to “Harry Potter and the Philosopher’s Stone” using representations learned from the Skip-gram model. The resulting movies are all from the Harry Potter franchise. Notice that these movies do not occur together in any training example, since each example contains a single movie. Alfonso Cuaròn and David Yates directed most of these movies. The list of actors include Daniel Radcliffe, the lead actor in the series, as well as supporting actors, Pam Ferris and Harry Melling, whose most popular roles occur in these movies. The resulting tags include subjective tags such as “love this movie so much” which are difficult to judge, but also relevant descriptive tags such as “potter” and “emma thompson”. Users are able to annotate movies with any tag they wish including actors’ names.

It is also possible to combine distributed representations to produce a new representation. In this way, representations can be retrieved that share characteristics from both of the originals. The ability to manipulate representations in this way might enable users to search through information space by selecting multiple movies, directors or actors.

For example, we have added together the representation of the actor Tom Hanks with the director Steven Spielberg. The result is a new vector of the same dimension. The most similar movies, actors, directors, and tags are shown in Table 3. The first three movies were directed by Spielberg and starred Hanks. As the similarity decreases, we find Forrest Gump starring Hanks and Lincoln directed by Spielberg. The nearest director to the addition of “Tom Hanks” and “Steven Spielberg” is not surprisingly Spielberg himself. Afterward, there is a sharp drop off in the similarity measure from 0.79 to 0.54. The remaining directors have all worked with Hanks. Likewise, the most similar actor is Hanks. Again there is sharp decline to the second most similar actor, 0.79 to 0.59. The remaining actors worked with either Hanks or Spielberg. The nearest tags are again subjective (“best war cinematography”) and descriptive (“tom hanks”).

The representations can also be combined in some surprising ways. Table 3 presents our final example. “Sean Connery” is subtracted from the popular James Bond movie, “Dr. No”. Then, “Roger Moore”, the actor who replaced Sean Connery in the Bond franchise, was added. The top four resulting movies are Bond films starring Roger Moore. John Glen and Peter Hunt have both directed Bond films. Not surprisingly, the representation for Roger Moore is the vector most similar to the new representation. The remaining actors have all appeared in Bond films. The resulting tags include the names of actors including Desmond Llewelyn, best known for his role as Q in 17 of the James Bond films.

This subjective evidence suggests that the distributed representations are capturing interesting relationships between the movies, directors, actors, and tags. It is difficult to quantify this evidence as there is, for example, no definitive similarity between “Harry Potter and the Philosopher’s Stone” and “Harry Potter and the Prisoner of Azkaban”. However, user ratings offer the opportunity to empirically test the quality of the distributed representations. We now turn our attention to experimental results.

Our experimental results are presented in Figure 2. Among the traditional recommender engines we use as our baselines, item-based collaborative filtering (IBCF) performed the best achieving an RMSE of 1.01 when the size of the neighborhood is set to 5. In both the user-based and item-based recommender systems, smaller neighborhoods produce better results.

When user and item representations are replaced with representations learned through the Continuous Bag of Words model (UBCB and IBCB), the item-based approach again outperforms the user-based approach. Compared to IBCF, IBCB produces significantly better results with an RMSE of 0.905 when the neighborhood size is set to 100. In both UBCB and IBCB, it appears a larger neighborhood offers better results.

The Skip-gram representations (UBSG and IBSG) improve upon the traditional approaches, but not as much as the Continuous Bag of Words representations. Once again, the item-based algorithm outperforms the user-based algorithm achieving an RMSE of 0.956 for a neighborhood of size 100. Larger neighborhoods appear preferred.

A comparison of these results offers several insights. First, the distributed representations appear to not only capture semantic information as suggested above, but to represent the interests of users and characteristic of items better than the commonly employed user-based and item-based collaborative filtering techniques.

Second, all item-based approaches outperform their user-based counterpart. This suggests that, in this domain at least, similarity between items is more informative. It may be that users watch several different types of movies and consequently create noisy profiles. In domains where users stick to narrower topics, such as when researchers bookmark journal articles, user-based systems might see an improvement.

Third, traditional approaches prefer small neighborhoods of users while the approaches utilizing the distributed representations prefer larger neighborhoods. This suggests that traditional user-based and item-based collaborative filtering algorithms struggle to find relevant neighbors and end up introducing noise into the prediction when the neighborhood size is increased. On the other hand, models incorporating distributed representations appear to benefit from large neighborhoods. It appears they are able to identify many relevant neighbors.

We now turn out attention to the linear weighted hybrid. It outperforms all other approaches reducing the RMSE down to as much as little as 0.858. The hybrid appears able to exploit the strengths of multiple approaches achieving better results than any individual component can achieve alone.

The weights associated with the component recommenders are displayed in Table 4. It is not surprising that the best performing individual recommender, IBCB, receives the most weight, 0.419, in the hybrid. The second best performing component, IBSG, receives the next highest amount, 0.225. The two user-based recommenders relying on the learned distributed representation, UBCB and UBSG, receive modest weights. Little weight is given to either of the two traditional recommenders, UBCF and IBCF.

Several conclusions can be drawn from these observations. First, since the linear weighted hybrid is trained to optimize the RMSE of the predicted ratings and more weight is given to IBCB than any other component, it seems this model provides the most information. However, other components garner significant weights as well. While IBCB is superior to the others, it seems the others still have something to offer. The contribution of multiple approaches is what allows the hybrid to outperform the individual components.

Second, while it appears the Continuous Bag of Words model is better suited to predict ratings in this domain, the Skip-gram model also outperforms the traditional techniques. Moreover, since the linear weighted hybrid draws upon the Skip-gram models significantly (0.122 and 0.225), it suggests that the Skip-gram model is learning something that the Continuous Bag of Words model is not.

Finally, since such little weight is given to the models using traditional representations of users and items, it seems that they add little additional information to the hybrid. In short, the distributed representations contain most of the information of the traditional ones and more.