MV-RNN: A Multi-View Recurrent Neural Network for Sequential Recommendation

The multi-modal features consist of visual and textual features ($\mathbf{f}$ and $\mathbf{g}$) in our work. They are obtained by GoogLeNet [44] and GloVe [45] weighted by TF-IDF respectively. The two kinds of features are $1024$-dimensional and $100$-dimensional vectors respectively. Due to the difference of $\mathbf{f}$ and $\mathbf{g}$, we learn two linear embedding matrices $\boldsymbol{E}$ and $\boldsymbol{V}$ to transform the original high-dimensional features to embedded low-dimensional visual and textual features ($\boldsymbol{i}_{\mathrm{f}}$ and $\boldsymbol{i}_{\mathrm{g}}$):

Sequential recommendation usually encounters the cold start problem as feedbacks are too sparse to learn fine representations of users and items. Modeling multi-view features is an effective way to alleviate this issue. These features are usually obtained from different data sources, and have different numerical ranges as well as different dimensions. Therefore, the raw features need be normalized to a same range to obtain $\boldsymbol{x}$, $\mathbf{f}$ and $\mathbf{g}$, and should better be embedded to $d$-dimensional vectors to obtain $\boldsymbol{i}_{\mathrm{x}}$, $\boldsymbol{i}_{\mathrm{f}}$ and $\boldsymbol{i}_{\mathrm{g}}$. None of them is sequence data and they are aligned with each other by the item ID.

Although concatenation and fusion are easy to utilize, they still have three issues. First, both concatenation and fusion do not have an explicit objective which is able to explore correlations across modalities [41]. Second, they are unhandy to use in such a situation where items in the test set have missing modalities [41]. Third, no matter the combination of $\boldsymbol{i}_{\mathrm{f}},\boldsymbol{i}_{\mathrm{g}}$ is concatenation or fusion, useful information is entered into the model as well as noise. Therefore, more robust structures and parameters ($\boldsymbol{E},\boldsymbol{V}$) need to be learned.

This method is based on the mDAE model [42]. It learns a linear mapping $\boldsymbol{M}$ and minimizes the reconstruction loss $l(\boldsymbol{t},\boldsymbol{M}\tilde{\boldsymbol{t}})$, where $\tilde{\boldsymbol{t}}$ is the corrupted version of original feature $\boldsymbol{t}$. However, mDAE has no hidden layer. Later, the coupled mDAE [43] modifies the original mDAE with two mappings in a linear way $l(\boldsymbol{t},\boldsymbol{M}^{\mathrm{T}}\boldsymbol{M}\tilde{\boldsymbol{t}})$. $\boldsymbol{M}\tilde{\boldsymbol{t}}$ and $\boldsymbol{M}^{\mathrm{T}}\boldsymbol{M}\tilde{\boldsymbol{t}}$ represent the encoding and decoding processes respectively. Based on these works, we introduce a multi-modal mDAE model, called 3mDAE, to learn fusion feature. Details are as follows.

We apply one RNN unit to model the latent feature and apply another RNN unit to model the multi-modal fusion feature. Formulation is defined by:

where $\boldsymbol{h}_{\mathrm{x}}^{t}$ and $\boldsymbol{h}_{\mathrm{m}}^{t}$ are defined as a user’s latent interest and multi-modal fusion interest at the $t$-th input. $\boldsymbol{U}_{\mathrm{x}}$ is a set of four matrices: $\boldsymbol{U}_{\mathrm{x1\sim 4}}\in\mathbb{R}^{d\times d}$. Similarly, $\boldsymbol{W}_{\mathrm{x}},\boldsymbol{b}_{\mathrm{x}},\boldsymbol{U}_{\mathrm{m}},\boldsymbol{W}_{\mathrm{m}}$ and $\boldsymbol{b}_{\mathrm{m}}$ are sets of three matrices or vectors, where subscripts $\mathrm{x}$ and $\mathrm{m}$ represent the latent modeling and multi-modal modeling.

Multi-view user representation is the concatenation of hidden states from the two RNN units. They are linked together at every time step in our work.

where $\boldsymbol{h}^{t}$ is the user’s general interest. But it may not be able to leverage the connection between multi-view features, as we model them in two RNN units separately and build discrete user’s interests. Thus we tend to develop a single RNN unit to handle multi-view features simultaneously.

Via the 3mDAE model and the united structure, we finally model the item’s multiple (latent, visual and textual) features and the user’s interest in the same feature space. Our MV-RNN model benefits from this united viewpoint.

The visual feature is obtained by the GoogLeNet [44] implemented by BVLC Caffe deep learning framework [49]. This network has 22 layers and has been pre-trained on 1.2M ImageNet ILSVRC2014 images [50]. We apply the output of layer $pool5/7x7\_s1$ to obtain $1024$-dimensional visual features. They are all positive and are normalized to range $[0,0.5]$.

To generate the textual features of items, a text description of each item is collected firstly. On Taobao, we directly use item titles which have already been segmented and disordered by the data provider. On Amazon, we combine each item’s category and title as its text data. Then we adopt the GloVe model [45] weighted by TF-IDF [51] to obtain each word’s feature and weight. Finally, the weighted feature for each item is computed to obtain $100$-dimensional textual features. Their values are in the vicinity of zero and are normalized to range $[-0.5,0.5]$.

Data is divided by time. We use feedbacks in first 60% of the time for training, 20% for validation and the rest 20% for test. Same as p-RNNs, hyperparameters are optimized on the validation set, and all models are retrained on the full training set (training and validation sets) before obtaining final results on the test set.

We design 3 combinations of input and 2 structures for the hidden state. There are 4 variants implemented as MV-RNN-Con., MV-RNN-Fus., MV-RNN-3mDAE-1U and MV-RNN-3mDAE-2U. The former 3 variants are built by the united structure, while the last one has the separate structure. The prefix ‘MV-RNN-’ can be omitted, and the 4 variants can be abbreviated as Con., Fus., 3mDAE-1U and 3mDAE-2U respectively. The Con. has the highest dimension of hidden state ($\boldsymbol{h}\in\mathbb{R}^{3d}$), while the rest has the same dimension ($\boldsymbol{h}\in\mathbb{R}^{2d}$). Additionally, we need to initialize parameters $\Theta$ to the same range, e.g., uniform distribution $[-0.5,0.5]$. The initial hidden state $\boldsymbol{h}^{0}$ of each sequence is always zero. The learning rate is fixed at $\alpha=0.1$ for all methods. Besides, the mini-batch size for training is set as 4 and users with similar lengths are grouped into one batch. This length-adjustment can greatly speed up training [53]. Complete codes for all models are written by using Theano and are available on GitHub⁴⁴4https://github.com/cuiqiang1990/MV-RNN. All experimental results are also listed on this website.

In this optimization process, $\lambda_{\Theta}$ is firstly selected based on basic methods (BPR, GRU and LSTM), then $\lambda_{\mathrm{ev}},r_{\mathrm{a}}$ are chosen by grid search. The ranges of these three parameters are $\lambda_{\Theta},\lambda_{\mathrm{ev}}\in\left[0.001,0.0001,0.00001,0.0\right]$ and $r_{\mathrm{a}}\in\left[0.001,0.0001,0.00001\right]$. With the reduction of data size from Taobao to Amazon, the best $\lambda_{\Theta},\lambda_{\mathrm{ev}},r_{\mathrm{a}}$ almost all get bigger.

Obviously, the performance of MV-RNN based on LSTM is better than that based on GRU in most cases, except the Con. and Fus. based on LSTM on Taobao. Although LSTM has more parameters, it also has the better model capacity. As a long as the model size is not significantly bigger, we should always consider the model with the best architecture. Therefore, in all the following experiments, we consider LSTM as the baseline instead of GRU and our MV-RNN is based on LSTM.

With the increasing of dimension, performances of top-30 metrics have similar trends with each other on both datasets. BPR and VBPR tend to get worse. They have similar trends as well as absolute values. It is difficult to tell the difference between VBPR and BPR on Recall, MAP, and NDCG, especially on Taobao. The p-RNN model is not sensitive to dimension. The LSTM and MV-RNN models obtain better performance with the increasing of dimension on Taobao, while they almost do not change with the dimension on Amazon. On the other hand, AUCs of all models are much stable with different dimensions. VBPR has obviously better performance than BPR on both datasets. The 4 variants of MV-RNN are nearly coincident with each other. The AUC is not sensitive to the dimension.

Generally, it is obvious that LSTM is a very strong baseline. Apparently, our MV-RNN model is the best. The optimal dimension is chosen as $d=20$ and it is applied to other experiments.

The denoising AE is first proposed for image classification on the MNIST database. It can make features more robust and avoid learning the identity function by using corrupted input. Identity function means just mapping the original input to its copy, which happens in the encoding process in AE (e.g., $\mathbf{f}\rightarrow\boldsymbol{E}\mathbf{f}$). It is easy to obtain a denoising AE just by a stochastic corruption operation on input. The original corruption mechanism randomly sets some of an input feature to zero with probability $0\leqslant p<1$. While in our experiment, we make feature itself corrupted.

This $denoising$ is conducted for 3mDAE. In this setting, we make some multi-modal data corrupted in the encoding process and still reconstruct both modalities in the decoding step. Training 3mDAE still requires all the data in Table LABEL:tablesub:datasets_5_core. The corruption levels are set as $p=\left[0.0,0.2,0.3,0.4\right]$ and $p=\left[0.0,0.1,0.2,0.3\right]$ for Taobao and Amazon respectively. If $p=0.0$, the input data in the encoding process is complete. The results are still obtained on the original test set where all items have all features. Results are shown in eight rows at the bottom of the Table V.

Obviously, performance can become better than the original ($p=0\%$) by $denoising$, especially the Recall, MAP and NDCG. More importantly, 3mDAE-1U performs best. It is able to be better than Con., although Con. has the highest hidden state dimension. When we randomly reset some features to zero in the encoding process, the noise in the whole input data is reduced. However, by reconstructing both modalities in the decoding step, the fusion feature of our 3mDAE can still keep the useful information in both modalities. Our 3mDAE can acquire more robust features. The best corruption levels for 3mDAE-1U/2U are $p=0.3/0.4$ and $p=0.1/0.1$ on two datasets respectively.

The 3mDAE-1U/2U are a one-unit model with the united structure and a two-unit model with the separate structure respectively. In Table V, the one-unit model outperforms the two-unit model. A united inner structure can better leverage the advantage of multi-view features. The separate structure may be not able to well model the connection between different views.

Reducing the number of items (search space) may be helpful. We extract three sub-datasets from Taobao by increasing the filtering strategy as $[10,15,20]$-core. The statistics are shown in Table LABEL:tablesub:datasets_bigger_core. In this way, the search space is greatly reduced. Then, we perform experiments by using LSTM and Con.. Accordingly, we need to re-select the best parameters and the results are shown in Table VI. With the increasing of $k$, the three metrics get bigger. Metrics of Taobao (20-core) are obviously bigger than that of the other datasts. This may be because the sparsity of Taobao (20-core) is clearly small. At the same time, our method Con. is always better than LSTM, which shows the effectiveness of our MV-RNN. Therefore, although the absolute values on Taobao are small, they are related to the dataset itself (e.g., sparsity).

In summary, our MV-RNN model is better than the others. MV-RNN can well model multi-view features and achieves the best and stable performance in different situations. The $denoising$ of 3mDAE is a good setting to improve performance. Besides, special strategies used in p-RNN are not necessary for handling multi-view features. Feature concatenation is natural but very useful. A united structure with simultaneous training strategy is easy to use and is better than the separate subnets built for each view in p-RNN. These conclusions of joint learning are also confirmed by the previous works, like a multi-view model for cross-domain user modeling [54].

From the perspective of basic performance, as shown in Tables LABEL:tablesub:cold_start_taobao and LABEL:tablesub:cold_start_amazon, the best values are scattered in four variants. It is difficult to draw a consistent conclusion.

As for the improvement shown in Table IX, most improvements on $cold$-$start$ are higher than those on $whole$, and are much higher than those on $active$. Comparatively, the basic model like LSTM has difficulty in predicting cold start items, while it is easier to obtain good performance on active items. Thus on the contrast, it is easy to design a model to substantially enhance the performance on $cold$-$start$, while it is more difficult to acquire obvious improvement on $active$. Under such situation, our MV-RNN still performs very well on $active$. For example, most improvements of MV-RNN are over 10% on $active$. MV-RNN not only has a significant improvement on cold start items but also has a sufficient improvement on active items.

In Table IX, there are some surprising improvements about Recall@30 on Amazon. We specify the improvement of MV-RNN over LSTM as $3\times 10^{\text{4}}\%$, because the performance of LSTM on $cold$-$start$ is zero. This poor performance of LSTM can be explained from the perspective of probability. When we train a sequence, we practically apply LSTM to model a joint probability $p(x_{1},\cdot\cdot\cdot,x_{t})$, where $x_{i}$ represents an item. When we predict $n$ items in corresponding test sequence, we actually predict a conditional probability $p(x_{t+1},\cdot\cdot\cdot,x_{t+n}|x_{1},\cdot\cdot\cdot,x_{t})$. Because the 81.4% cold start items and the corresponding 50.1% feedbacks on Amazon result in limited interactions among users and items, both probabilities are very small. Therefore, it is very hard to make accurate recommendation under Recall@30 on Amazon. After we incorporate the additional content information, 4 variants of MV-RNN have performances of 0.398%, 0.328%, 0.623% and 0.319% respectively. The absolute values are small, but we obtain very large but reasonable improvements. This strange and extreme phenomenon exactly reflects the great power of additional content information and the powerful modeling capability of MV-RNN.

When the bin number increases, performance becomes better, as seen from Tables LABEL:tablesub:cold_start_taobao and LABEL:tablesub:cold_start_amazon. That is because it is easier to predict frequent items. On Taobao, there is a strange phenomenon. Performance decreases first on a few bins in the front and then increases. As the decrement is not significant, we can still think the performance is growing. Then we mainly focus on the analysis of improvements. For better representation, improvements are illustrated by curves in Figure 5.

These growth curves do not always have the same change on two datasets. On Taobao, curves tend to be flat. On Amazon, as the bin has a larger proportion of cold start items (seeing from the right side of a figure to its left side), the improvement almost becomes larger. This indicates that multi-modal information is indeed beneficial to relieve cold start. In other words, when the cold start problem gets worse on small bins with a bigger proportion of cold start items, multi-modal information can significantly relieve this problem. Because cold start items have few interactions with users, directly related multi-modal information would effectively represent the item’s characteristics and the user’s interest.

AUC is much more stable than Recall@30. We consider the difference of user’s preferences towards positive and negative items in AUC, and the BPR training process exactly maximizes this difference. For Recall@30 curves, there is a large difference between Taobao and Amazon. These curves on Taobao are separate from each other, but they almost come together in the last interval $all$. Perhaps because of the small proportion of feedbacks on the interval $[1,18]$ (27.0%), there would be some fluctuations in the performance of each model. These curves on Amazon have an obvious increasing law when the bin number gets smaller. For AUC curves, the situation is much better. On Taobao, most improvements are stable. For example, improvements of MV-RNN are around 30%. On Amazon, the smaller the bin number, the larger the improvement.

These curves, especially those on Amazon, can greatly support the following conclusion. Multi-modal information can significantly relieve the item cold start problem. Besides, the worse the cold start, the more powerful the multi-modal information.

Obviously, the similar items under different kinds of features vary greatly, and the multi-view (latent, visual, textual) features are complementary to each other. (1) For the latent feature $\boldsymbol{i}_{\mathrm{x}}$, the similar items are greatly different from each other as $\boldsymbol{i}_{\mathrm{x}}$ are just learned by the feedback. If the latent features of two items are similar, probably because they were both purchased by many people. (2) Whether it is item itself or the background in the image, the top-5 items based on the visual feature $\boldsymbol{i}_{\mathrm{f}}$ are very similar in appearance. However, the second and the forth items in this line obviously belong to other categories. The visual feature is powerful but can not reflect the intrinsic characteristics of items, like material of clothes. (3) On the other hand, the textual feature $\boldsymbol{i}_{\mathrm{g}}$ is acquired by the item description. It can truly reflect what the product is and can ignore the effect of the background in an image, but it is not intuitive to show the color, shape, etc. (4) The fusion feature $\boldsymbol{i}_{\mathrm{m}}$ is a combination of $\boldsymbol{i}_{\mathrm{f}}$ and $\boldsymbol{i}_{\mathrm{g}}$. It mainly integrates the external and intrinsic characteristics of the item, such as the style and material of clothes. However, such characteristics can not generate precise recommendation because there is no one-to-one match between each characteristic and each item. (5) The final item feature $[\boldsymbol{i}_{\mathrm{x}};\boldsymbol{i}_{\mathrm{m}}]$ fuses $\boldsymbol{i}_{\mathrm{x}},\boldsymbol{i}_{\mathrm{f}},\boldsymbol{i}_{\mathrm{g}}$. It can fully reflect the characteristics of an item and help to understand the user’s overall interest. In summary, multi-view features $\boldsymbol{i}_{\mathrm{x}},\boldsymbol{i}_{\mathrm{f}},\boldsymbol{i}_{\mathrm{g}}$ used in our work are complementary.