Uniform Sequence Better: Time Interval Aware Data Augmentation
for Sequential Recommendation

Sequential recommendation has become an important research task to predict an accurate item to access at the next time step for a user based on her historical data, which is often organized as a sequence of interacted items. Most existing works attempt to learn user preference as a transition pattern from the sequential data, i.e., how the previous item can be properly transited to the next item. The ordering of interacted items matters a lot in this task (Jiang et al. 2021). Typical networks such as Transformer are applied to model sequential data and produce recommendations (Kang and McAuley 2018; Sun et al. 2019; Li, Wang, and McAuley 2020; Ma et al. 2020; Jiang et al. 2021).

However, the time interval between two consecutive items (in a sequence) does not receive much attention in the literature, and we believe this factor plays an important role in learning user preferences, especially when considering the issue of preference drift, i.e., user preference may drift away as time goes by. As a matter of fact, we observe that the time interval in a sequence may vary significantly different, which may lead to the ineffectiveness of user modeling merely based on the ordering of interacted items. Figure 1 shows a typical example that we drew from a real dataset (Amazon Beauty). Both users have a sequence of interacted items with the same length, but the time intervals for user $u_{1}$ are distributed more uniformly than those for user $u_{2}$. Most products purchased by user $u_{1}$ lie in the category of Skin / Facial Care, while user $u_{2}$ begins to purchase products in different features/categories every after a large time interval (e.g., from Hair Spray to Mascara, from Mascara to Moisturizer). It is intuitively believed that user preference can be better learned for user $u_{1}$ since her sequence is more uniformly distributed than the other user. We denote such a kind of sequence for user $u_{1}$ as ‘uniform sequence’ and that for user $u_{2}$ as ‘non-uniform sequence’.

In this paper, we conduct an empirical study to further validate this intuition. Specifically, we design two strategies to split the original sequential data into uniform and non-uniform sequences, and then compare the performance of four competing recommenders under different settings. The experimental results show that performance can be significantly improved with uniform sequences rather than non-uniform sequences. However, it is a commonplace that real sequences may not be uniformly distributed in terms of the time interval. To this end, we propose five operators of data augmentation (denoted as Ti-Crop, Ti-Reorder, Ti-Mask, Ti-Substitute, and Ti-Insert) to transform non-uniform original sequences to uniform sequences with the consideration of standard deviation of time intervals. Besides, we apply contrastive learning to ensure that augmented sequences preserve a high similarity with the original ones. Finally, we implement these improvements on a state-of-the-art model CoSeRec (Liu et al. 2021) and thus form a time interval aware recommendation model called TiCoSeRec.

To sum up, the main contributions of this paper can be presented as follows:

• •

  We conducted an empirical study to verify the idea that uniform sequences are more beneficial for performance improvement than non-uniform sequences. To the authors’ best knowledge, we are the first to study the distribution of time intervals for sequential recommendation.

• •

  We propose five operators of data augmentation to transform non-uniform sequences to uniform ones, and generate new sequences of high similarity to the original ones. Besides, we also devise a control strategy to deal with data augmentation for short sequences.

• •

  We conducted extensive experiments on four real datasets and demonstrated that our approach TiCoSeRec can achieve a significant performance improvement in comparison with other 11 competing models.

In this section, we will conduct an empirical study to verify our assumption that recommenders can get better performance with uniform sequences than non-uniform ones.

Therefore, the sequential recommendation task in this paper can be formulated as follows. Given the sequences of interacted items $S_{u}$ and time intervals $T_{u}$, how to accurately predict the most possible item $v$ that user $u$ will interact with at time step $N$ + 1, given by:

We can find a few observations from Table 2.Firstly, the performance with uniform subsets ($\mathbb{U}$) is generally better than that with non-uniform subsets ($\mathbb{N}$) under both strategies (S, I) and across all the datasets. Specifically, there are 54 cases (out of 64 pairs of comparison) with improvements greater than 15%, and 33 cases (around 51.6% of all cases) with improvements greater than 20%, and the maximum improvement goes up to 32.41%. There are only 3 cases (out of 64) lower than 10%, and the minimum improvement is 8.22%. Secondly, the performance of subset ‘random’ lies in the middle of subsets $\mathbb{U}$ and $\mathbb{N}$ under both strategies S and I. This is because subset ‘random’ can be regarded as a random mixture of subsets $\mathbb{U}$ and $\mathbb{N}$. Lastly, the best selection between strategies S and I is dataset specific. Specifically, strategy I gets greater percentage of performance improvements than strategy S on three datasets (Beauty, Sports, Yelp), but gets worse results on the rest dataset (Home).

As we previously discussed, we generally label the top $\sigma$ ratio of ranked item sequences as uniform, and the rest as non-uniform sequences that need to be augmented for better performance. Assume we have $m$ item sequences in total, then there are $m(1-\sigma)$ item sequences to be selected as non-uniform sequences.

For data augmentation, there are five widely used operators in the literature, including Crop, Mask, Insert, Reorder and Substitute. These operators perform data augmentation and generate new sequences by randomly selecting target items to modify. However, such a process cannot guarantee that a newly generated sequence will possess a uniform distribution of time intervals, i.e., becoming a uniform sequence. Therefore, in this section, we will introduce five new data operators with time intervals (Ti-Crop, Ti-Mask, Ti-Insert, Ti-Reorder, Ti-Substitute) that aim to generate better item sequences by carefully selecting the most proper target items to operate. A typical comparison between our operators and traditional ones is given in Figure 3.

As illustrated in Figure 3(a), we have a given sequence containing two sub-sequences: one uniform and the other non-uniform. According to the time intervals among items (in unit $d$), we may classify these items into three categories: $A,B,C$. Thus, the preference pattern can be represented by the transfer of item categories, i.e., $A\rightarrow B\rightarrow C$.

In Figure 3(b), since the traditional approach (Insert) selects the target position randomly, the newly generated sequence may destroy the uniform sub-sequence after inserting item $v_{i}$, resulting in an even worse situation. In contrast, our approach (Ti-Insert) will insert a similar item $v_{ti}$ to the right position such that the non-uniform sub-sequence becomes more uniform. The underlying assumption is that item $v_{ti}$ was accessed by the user in the middle timestamp between those of items $v_{6}$ and $v_{7}$. In this way, the original time interval (denoted by $d$) between items $v_{6}$ and $v_{7}$ is substituted by the new time intervals (value $d/2$) between $v_{6}$, $v_{7}$ and $v_{ti}$ in the new sequence, leading to a more uniform distribution of time intervals. To this end, we first sort the time intervals of sequence $T_{u}$ from large to small (denoted by function $\mbox{descend}(T_{u})$), and then preserve the top-$k$ indices as the target positions. Formally, our positions are determined by:

Item masking will increase the distance (time interval) between the items before and after the masked item, and thus change the characteristics of overall standard deviation. Take Figure 3(d) as an example, traditional Mask selects item $v_{6}$ to mask, obtaining a new sequence with standard deviation increasing from $0.810d$ to $1.084d$. As a matter of fact, the distance (i.e., $3d$) between items $v_{6}$ and $v_{7}$ is the greatest one among all the time intervals. User preference may drift to some extent in this case. Masking item $v_{6}$ will make it harder to learn user preference as the new maximum distance increases from $3d$ to $3.75d$. The larger distances between two items, the greater possibility they differ in more aspects. Differently, Ti-Mask operates on the items with much smaller time intervals, and the standard deviation gets decreased from $0.810d$ to $0.871d$ after masking item $v_{3}$, i.e., becoming a more uniform sequence.

Comparison methods. To evaluate the effectiveness of our data augmentation, we apply it to a state-of-the-art sequential model CoSeRec (Liu et al. 2021) and relabel this variant as TiCoSeRec for ease of discussion. Then, we compare with the following three groups of recommendation models.

• •

  Non-sequential models include BPR (Rendle et al. 2012) and LightGCN (He et al. 2020).

• •

  Sequential models without data augmentation include GRU4Rec (Hidasi et al. 2015b), STAMP (Liu et al. 2018), SASRec (Kang and McAuley 2018), BERT4Rec (Sun et al. 2019), TiSASRec (Li, Wang, and McAuley 2020) and LightSANs (Fan et al. 2021).

• •

  Sequential models with data augmentation include CL4SRec (Xie et al. 2020), CoSeRec (Liu et al. 2021) and DuoRec (Qiu et al. 2022).

Our approach TiCoSeRec is built upon CoSeRec, which consists of three components. (a) Recommendation by SASRec. (b) Data augmentation by five operators without considering time intervals. (c) Contrastive learning to maximize the agreement between the augmented and original sequences. Hence, the main contributions (and difference) of TiCoSeRec lie in the proposed five data operators with the consideration of time interval.

We set the embedding dimension size as 128 for all models, and take the suggested settings for other parameters as reported in the original papers. For our TiCoSeRec model, we set the number of layers and heads in the Transformer to 2. The maximum sequence length is set to 50. We set parameter $\sigma=0.2$, i.e., top 20% ranked item sequences are treated as uniform, and no data augmentation will be forced on them. We apply grid search to find the proper values for each data operator’s sampling ratio (i.e., $\mu,\eta,\beta$) in the range $[0.1,0.9]$ (stepping by 0.1) as well as the threshold $K$ to determine short sequences in $\{4,8,12,16,20\}$.

For non-sequential models, BPR produces the worst performance, indicating the value of item sequence information. Nevertheless, LightGCN obtains relatively good performance by considering the structural neighborhood information instead. Among all the sequential models without data augmentation, LightSANs achieves the best performance because of the powerful self-attention mechanism in the Transformer structure. Another interesting observation is that TiSASRec works worse than SASRec, though the former takes time interval as additional information for model learning. It implies that time interval has not been well studied in the literature. Our approach opts to use time interval for data augmentation rather than model building.

Further improvements are obtained by integrating data augmentation with sequential models. It is observed that our approach TiCoSeRec consistently performs the best across all the datasets, and the relative improvements are important and range from 5% to 18%. The improvements can be even more significant when compared with CoSeRec, since our main difference is the data augmentation with time intervals rather than traditional operators. Our data augmentation can not only transform non-uniform sequences into uniform ones, but also ensure the newly generated sequences are of high similarity to the original ones.

The recommendation accuracy will be deteriorated when degenerating our data operator to a traditional one, indicating the effectiveness of our data operator by time intervals. Nevertheless, all variants of our approach still outperform the baseline CoSeRec. Hence, it is beneficial for performance improvement to consider all kinds of data operators. Among all the variants, TS$\rightarrow$S has the worst performance while TR$\rightarrow$R reaches the best hit ratio. In other words, the TS operator has the greatest impact on the recommendation performance while the TR operator has the smallest effect.

The results on the four datasets followed similar trends. That is, as $\sigma$ increases, the performance improves gradually and then reaches the maximum value when setting to 0.2 or 0.3. Generally, $\sigma=0.2$ leads to the best performance. In other words, it is the best to consider the top 20% ranked item sequences as uniform, and the rest needs to be transformed for better accuracy. Further increasing the value of $\sigma$ will not improve performance to a larger extent, but cause much worse accuracy when $\sigma>0.4$. To conclude, a proper setting of parameter $\sigma$ is important for recommendation.

In this paper, we explored the impact of time interval on sequential recommendations. Our basic idea is that uniform sequences are more valuable for next-item prediction. This assumption was validated by an empirical study. Then, we proposed five data operators to augment item sequences in the light of time intervals. Our experiments on four public datasets have verified the effectiveness of our proposed operators. For future work, we intend to further consider the factor of item category for data augmentation.

This work is partially supported by the National Natural Science Foundation of China under Grant (No. 62032013, 61972078), and the Fundamental Research Funds for the Central Universities under Grant No. N2217004.