Generating Negative Samples for Sequential Recommendation

Sequential recommendation aims to accurately characterize users’ dynamic interests by modeling their past behavior sequences. Early works on SR usually models an item-to-item transaction pattern based on Markov Chains (Rendle, 2010; He and McAuley, 2016). FPMC (Rendle et al., 2010) combines the advantages of Markov Chains and matrix factorization to fuse both sequential patterns and users’ general interest. With the recent advances of deep learning, many deep sequential recommendation models are also developed (Tang and Wang, 2018; Hidasi et al., 2015; Kang and McAuley, 2018; Sun et al., 2019b). GRU4Rec (Hidasi et al., 2015), Caser (Tang and Wang, 2018), and SASRec (Kang and McAuley, 2018) explore the potential of encoding user sequential behaviors via an RNN, CNN, and Transformer, respectively. FDSA (Zhang et al., 2019), TiSASRec (Li et al., 2020) and S${}^{3}\text{-Rec}$ (Zhou et al., 2020) leverage side information (e.g., time-interval, item categories.) for a comprehensive representation. BERT4Rec (Sun et al., 2019b) replaces next-item prediction (NIP) task with a masked-item prediction task (Taylor, 1953) to capture contextual information. With the success of contrastive self-supervised learning, several works (Ma et al., 2020; Xie et al., 2020; Qiu et al., 2021b, a) propose different contrastive SSL paradigms as a complement or a replacement task of NIP for a more comprehensive learning. LSAN (Li et al., 2021) also improves SASRec from efficiency serving perspective Nevertheless, most existing works ignore the importance of quality of sampled negative items and view the item randomly sampled from user non-interacted item set or all items in the same training batch as negative items.

Word2vec (Mikolov et al., 2013) first proposes to sample negative items based on the word frequency distribution proportional to the 3/4 power to train the skip-gram language models. Later works in NLP and Social Networks often follow such setting (Pennington et al., 2014; Tang et al., 2015; Grover and Leskovec, 2016). In graph mining, RotatE (Sun et al., 2019a) first proposes to sample negative items based on model’s prediction and then MCNS (Yang et al., 2020) proposes to further improve its efficiency via the Markov Chain based Metropolis-Hastings algorithm. However, these methods only consider neighborhoods of the nodes on graph while ignore the sequential dynamic of the data. Another line of works improves the Sampled Softmax (Zhou et al., 2021; Blanc and Rendle, 2018) to better approximate to the full Softmax. In contrast, our work study the SR methods that trained under NCE framework, which trains a sequential binary classifier to distinguish target and negative items. Several GAN-based (Goodfellow et al., 2014) methods are proposed for application such as information retrieval (Wang et al., 2017; Park and Chang, 2019) and graph node embeddings (Cai and Wang, 2017). However, GAN-based methods are often hard to train and the additional training of generator also makes the sampling inefficient for SR models. In recommendation, Bayesian Personalized Ranking (Rendle et al., 2012) first proposes to sample negative items uniformly from user non-interacted items for training factorization machines. Dynamic negative sampling(DNS) (Zhang et al., 2013) develops a ranking-aware negative sampling strategy for improving collaborative filtering based methods. PinSAGE (Ying et al., 2018) consider items with high PageRank scores as “hard-negative” samples with curriculum learning scheme to train large scale graph neural networks. Despite of their success in their own domain, these methods ignored the importance of the sequential dynamics of users’ interests thus are not ideal to be adopt for sequential recommendation.

SR is usually formulated as a next item prediction (NIP) task. Formally, in a recommender system, there is a set of users and items denoted as $\mathcal{U}$ and $\mathcal{V}$ respectively. Each user $u\in\mathcal{U}$ is associated with a sequence of interacted items sorted in chronological order $S^{u}=[s^{u}_{1},\dots,s^{u}_{t},\dots,s^{u}_{|S^{u}|}]$ where $|S^{u}|$ is the number of interacted items and $s^{u}_{t}$ is the item $u$ interacted with at step $t$. We denote $\mathbf{S}^{u}$ as the embedded representation of $S^{u}$, where $\mathbf{s}^{u}_{t}$ is the $d$-dimensional embedding of item $s^{u}_{t}$. In practice, sequences are truncated with maximum length $T$. If the sequence length is larger than $T$, the most recent $T$ actions are considered. If the sequence length is smaller than $T$, “padding” items will be added to the left until the length is $T$ (Hidasi et al., 2015; Tang and Wang, 2018; Kang and McAuley, 2018). For each user $u$ at time step $t$, the goal of SR is to predict the item that the user $u$ would be interested in at step $t+1$ among the item set $\mathcal{V}$, given her past behavior sequence $\mathbf{S}^{u}_{1:t}$.

To train an SR model, a standard learning procedure fits the sequential data following the maximum likelihood estimation principle. Specifically, for each user $u$ at position step $t$ in a mini-batch $\mathcal{B}$, we want to learn a parametric function $f_{\theta}$ that maximize the probability of the target item:

where

where $\mathbf{h}^{u}_{t}=f_{\theta}(S^{u}_{1:t})$ is the encoded user’s interest representation at time $t$, $Z_{\theta}(\mathbf{h}^{u}_{t})=\sum_{v\in V}(\mathbf{h}^{u}_{t}\cdot\mathbf{v})$ is the partition function that normalizes the score into a probability distribution, and $\exp(\mathbf{h}^{u}_{t}\cdot\mathbf{s}^{u}_{t+1})$ is a similarity score of a user’s preference toward the target item. Unfortunately, computing this probability as well as its derivatives are infeasible since the $Z_{\theta}(\cdot)$ term requires summing over all items in $\mathcal{V}$, which is generally of large-scale in sequential recommendation.

Hence, existing methods (Kang and McAuley, 2018; Li et al., 2020; Ma et al., 2020; Xie et al., 2020) commonly adopt an approximation via Noise Contrastive Estimiation (NCE) (Gutmann and Hyvärinen, 2010). NCE is based on the reduction of density estimation to probabilistic binary classification. It provides a stable and efficient way to avoid computing $Z_{\theta}(\cdot)$ while estimating the original goal. The basic idea is to train a binary classifier to discriminate between samples from the positive data distribution and samples from a “noise” (negative sampling) distribution. Specifically, given the encoded user interest $\mathbf{h}^{u}_{t}$, we view the next item $\mathbf{s}^{u}_{t+1}$ as its positive item and the sampled $k$ negative items from a pre-defined distribution function $Q(\cdot)$ (e.g., a uniform distribution over all other items in $V$). We train the SR model with the following loss function:

and

where $P(D=1|\mathbf{h}^{u}_{t},\mathbf{s}^{u}_{t+1})=\sigma(\mathbf{h}^{u}_{t}\cdot\mathbf{s}^{u}_{t+1})$, $\sigma$ is sigmoid function, and $\mathbf{s}^{u}_{-,t+1}$ is the sampled negative item at $t+1$. This loss decreases when $\mathbf{h}^{u}_{t}\cdot\mathbf{s}^{u}_{t+1}$ increases and $\mathbf{h}^{u}_{t}\cdot\mathbf{s}^{u}_{-,t+1}$ decreases. In other words, optimizing this loss function is equivalent to pulling the sequence embedding $\mathbf{h}^{u}_{t}$ closer to the positive item $\mathbf{s}^{u}_{t+1}$ whilst pushing away from sampled negative items, thus being contrastive. To make NCE approximate to maximum log-likelihood (Eq 2) closer, one needs to either sample more negative items or improve the quality of the negative sampling distribution $Q(\cdot)$. Surprisingly, neither of them is paid enough attention by existing methods.

The above theorem shows that $k$ is an importance factor of making SR models well trained (proof given in Appendix A). Empirically, naively increasing $k$ though trivial, but is not a good choice in recommendation tasks. Because under random sampling, most of the sampled items can be uninformative with the training going on while training time cost is linearly increased. Because of that, existing SR models often keep the default number $k=1$. Without naively increasing the number of negative items $k$, designing a good negative item distribution function $Q$ is crucial to make SR models successful.

Theorem 2 indicates that the optimal embeddings are dependent on both data distribution $P(\cdot)$ and the negative sampling distribution $Q(\cdot)$ (proof given in Appendix B.). As such, it is necessary to sample items from true negative sampling distribution, which would otherwise yield sub-optimal results.

The two theorems motivate us to improve sampling process of negative items for sequential recommendation as in following sections. Hereafter, we propose a novel negative item generator as well as a strategy to further improve its efficiency.

GenNi introduces $\alpha$ to control how often hard negatives are sampled. But we must still manually tune $\alpha$. Curriculum learning (Bengio et al., 2009; Kumar et al., 2010) allows neural networks to begin by understanding easy negative samples followed by hard ones. We further reduce this rule to let the model itself adjust $\alpha$. Specifically, we use the loss value in each batch as the critic to see if the current curriculum is too hard or too easy. When the previous loss is larger than the current one, we increase $\alpha$, otherwise we decrease $\alpha$. In this way, $\alpha$ is self-adjusted with the online loss value as feedback, which reduces human effort in choosing the initial $\alpha$ (see Section 5.5.1 for more detail).

The computation costs of $\textsf{GenNi}_{\text{SA}}$ and $\textsf{GenNi}_{S^{3}}$ are similar to SASRec and S${}^{3}\text{-Rec}$ except that our methods use GenNi instead of uniform sampling. The overall computation cost is mainly from Transformer, the feed-forward network and GenNi, which is $\mathcal{O}(T^{2}\cdot d+T\cdot d^{2}+\beta\cdot|\mathcal{V}|\cdot T)$. The dominant term is typically $\mathcal{O}(|T|^{2}d)$ from Transformer when $\beta$ is small. Though GenNi requires high computational cost when $\beta\cdot\mathcal{|V|}$ are large, however, our proposed acceleration strategy of it ensures faster convergence as well as better performance (see Section 5.3). The proposed two strategies of choosing $\beta$ in Section 3.3.3 also help to balance the effectiveness and efficiency of GenNi. More details regarding convergence analysis are provided in Appendix C.

Though our method is induced from NCE paradigm in SR, GenNi also has the ability to improve other training framework built upon pair-wise ranking loss, e.g., sequential BPR (Rendle et al., 2012). Previous work (Hidasi and Karatzoglou, 2018) justifies that optimizing a recommender model with a BPR loss results in gradient vanishing issue if introducing more than one negative samples. The reason is that after several epochs of training, those uniformly sampled negative items already have lower scores than the target due to their easiness to identify. As a result, gradients towards those negative items gradually diminish. Instead, GenNi generates informative negative items during each epoch of training, which alleviates the gradient vanishing issue of BPR. We conduct experiments to verify this claim in Section 5.5.2.

Table 1 shows overall recommendation performance of all models on the four datasets. We observe that:

• •

  Our methods $\textsf{GenNi}_{\text{SA}}$ and $\textsf{GenNi}_{S^{3}}$ both consistently outperform existing methods on all datasets by a large margin. The average improvement compared with the best baseline ranges from 6.15% to 49.05%. Specifically, compared with SASRec and S${}^{3}\text{-Rec}$, our approaches simply replacing their original uniform sampler with GenNi, achieve 96.02% and 107.66% average performance improvements on four datasets over SASRec and S${}^{3}\text{-Rec}$ at NDCG@5, respectively. This observation clearly shows that sampling informative negative items is as important as other components in making SR successful and also demonstrates the effectiveness of our proposed sampler GenNi.

• •

  Transformer is an effective way of encoding user sequential dynamic patterns. Compared with GRU4Rec, Caser, SASRec and S${}^{3}\text{-Rec}$ we can see that SASRec and S${}^{3}\text{-Rec}$ that utilize a Transformer-based encoder can consistently achieve better performance compared to CNN/RNN-based encoders: Caser and GRU4Rec. S${}^{3}\text{-Rec}$ performs better than SASRec in most datasets because it fuses additional item attributes during pre-training. However, all these methods sample negative items randomly from user non-interacted item sets, yielding to a sub-optimally trained model.

• •

  For different negative sampling strategies, SASRec${}_{\text{pop}_{\gamma}}$ performs slightly better than SASRec, indicating that the popularity-based method can help improve model learning. However, this strategy is static and does not consider the penalization of each user behavior, resulting in a large performance gap compared to $\textsf{GenNi}_{\text{SA}}$. DSSRec, CL4SRec, and MMInfoRec proposes different contrastive self-supervised learning paradigms can outperform other baselines that only train with an NIP objective. This observation demonstratives the effectiveness of contrastive self-supervised learning. These three methods commonly consider items in the whole training batch as negatives, and MMInfoRec also proposes a heuristic hard negative mining strategy with a memory bank to further improve the quality of the samples. Their successes suggest that sample more negative items and hard negative mining also benefits model learning. Although MMInfoRec is the best baseline method, it still performs worse than our approaches. The reason might be twofold. First, considering all items in the training batch as negative items can introduce false-negative samples. Second, heuristic hard negative mining (e.g., considering user historical interacted items as hard negatives in MMInfoRec) is not adaptive to model parameters. As a result, the sampled hard negatives can gradually become uninformative to the model.

SASRec has proven to be an order of magnitude faster than CNN and RNN-based recommendation methods (Kang and McAuley, 2018), such as Caser and GRU4Rec. In this section, we evaluate the efficiency of $\textsf{GenNi}_{\text{SA}}$ (on the Beauty dataset) by comparing with the most efficient baseline SASRec and the best performing baseline MMInfoRec (See Appendix F for result comparisons on other datasets). We omit the comparisons of $\textsf{GenNi}_{S^{3}}$ as it has the same computation cost as $\textsf{GenNi}_{\text{SA}}$ in its training stage. The only difference is that $\textsf{GenNi}_{S^{3}}$ requires a pre-training stage to fuse item attributes in the model.

Figure 4 shows the performance w.r.t. training (wall-clock) time as well as the computation cost per epoch. We can see that replacing the uniform sampler with GenNi does introduce additional computation cost; for example, SASRec spends 2.44 seconds on model updates for one epoch while $\textsf{GenNi}_{\text{SA}}$ ($\beta=1$) requires 6.30s/epoch. However, $\textsf{GenNi}_{\text{SA}}$ converges to much higher performance and requires fewer training epochs to converge. What’s more, as we reduce $\beta$ to 0.1, $\textsf{GenNi}_{\text{SA}}$ ($\beta=0.1$) only needs 2.47 seconds to update the model for one epoch, which is close to SASRec (2.44s/epoch), and still performs better than SASRec. Although MMInfoRec is the best performing baseline, it requires 34.22 seconds on model updates for one epoch. Our method $\textsf{GenNi}_{\text{SA}}$ ($\beta=1.0$) and $\textsf{GenNi}_{\text{SA}}$ ($\beta=0.1$) are over 5.42 and 13.85 times faster and also perform better than MMInfoRec.

GenNi introduces two hyper-parameters $\alpha$ and $\beta$ that controls the difficulty of sampled negatives and the negative item generation computation cost. The number of negative samples is set as $k=1$, which is the same as the original SASRec’s setting for fair comparison. We also study model sensitivity to the number of negative samples $k$, the embedding size, and learning rate.

Impact of the informative of negative items $\alpha$. Figure  5 shows the influence of $\alpha$ on model performance over four datasets. We can see that the model performance increases as $\alpha$ increases at the beginning, and then the performance reaches a peak. Specifically, when $\alpha=2.5$, the model performs best on Beauty,while $\alpha=4.4$, the model performs best on Yelp. Note that when $\alpha=0$, GenNi becomes a uniform sampler. The large $\alpha$ shows that randomly sampled items can be uninformative as training proceeds, while considering items that are currently hard to be correctly classified can further improve the model. Similar observations are found on Sports and Toys.

Impact of $\beta$ for accelerating generation. Figure 6 shows model performance w.r.t a fixed $\beta$ value. We interestingly find that there is an elbow point of $\beta$ that balances the effectiveness and efficiency of GenNi well. For example, when $\beta=0.1$, it reduces about $90\%$ computation cost of GenNi while the model can still achieve about 95% performance (e.g., NDCG@5) of its original version ($\beta=1.0$) in Beauty. On one hand, it shows the superiority of GenNi, which takes the efficiency of randomly sampling to pre-select a certain portion of items in the first stage and then concentrates on finding informative ones with a slower but more accurate sampling strategy. On the other hand, the decreasing of performance with small $\beta$ also indicates that with the training goes, the number of informative items also decreasing so too small $\beta$ can filter out all these items in pre-selection stage. As introduced in Section 3.3.3, we also report the results that gradually increasing the $\beta$ value via Eq 7 in Table 2. We can see that gradually increasing $\beta$ can achieves the similar effect as of a fixed $\beta=1.0$ because the informative items are decreasing along with training goes and $\beta$ can be small while still capture informative items in early training stage. This strategy reduces the computation cost while achieving same effect comparing with a fixed $\beta=1.0$.

Impact of the number of negative samples $k$. Figure 7 shows the impact of the number of negative samples. We can observe a diminishing return in the performance improvement for both SASRec and $\textsf{GenNi}_{\text{SA}}$. However $\textsf{GenNi}_{\text{SA}}$ can consistently outperform SASRec, which further verifies the importance of sampling informative negative items. Note that training with additional negative samples linearly increases the time cost. While $\textsf{GenNi}_{\text{SA}}$ can even achieve better performance with only 1 negative sample compared with SASRec that uses 9 negative samples on Beauty and Sports. See Appendix G for additional results on Toys and Yelp, and the sensitivity to the embedding size, and learning rate.

We conduct a case study on the Sports dataset (McAuley et al., 2015) to show examples of dynamically changing informative negative items. Figure 9 visualizes the informative items to the SR model. When the user reviews a water bottle, the cup holder is the most informative item; the user reviews earphones instead, and the most informative items changes to a gym bike (etc.). We can also observe that the informative negative distribution is close to uniform initially, and gradually diversifies as training goes.