Learning an Adaptive Meta Model-Generator for Incrementally Updating Recommender Systems

Methods developed for continual learning generally fall under three categories: experience replay, regularization-based methods, and model fusion. Experience replay prevents forgetting by reusing past samples together with the new data to update the model (Robins, 1995). Generative methods are later on developed to alleviate the burden of storing real data (Shin et al., 2017). Regularization-based methods retain past knowledge by constraining the parameters update based on some measure of importance (Kirkpatrick et al., 2017; Zenke et al., 2017). Some works also employ distillation techniques to regularize the update direction (Rebuffi et al., 2017; Wen et al., 2018). Model fusion supports continual learning of tasks by gradually incorporating sub-networks. Both expandable (Rusu et al., 2016) and fixed-size network methods (Mallya and Lazebnik, 2018) are developed. Though these methods have shown promising results in CV and NLP, how to adapt them to incrementally update RS models is non-trivial, as they lack mechanism to explicitly optimize for the future performance. To overcome the specific forgetting problem in RSs incremental update, two lines of research have been developed in recent years, which are closely related to some of the methods in continual learning (Mi et al., 2020).

This line of works relies on reusing historical samples to preserve past memory, analogous to experience replay in continual learning. Most often, a reservoir is maintained to keep a random sample of history (Diaz-Aviles et al., 2012). After that, heuristics are designed to select samples from the reservoir for model updating, prioritising recency (Chen et al., 2013; Zhao et al., 2020) or the extent of being forgetten (Guo et al., 2019; Wang et al., 2018; Qiu et al., 2020). Sample-based approach can be seen as an intermediate between incremental update and training on full historical data, aiming to find a balance between short-term interest and long-term memory. However, an apparent limitation of this approach is that the individual samples, though carefully selected with some measure of informativeness, are insufficient to represent the big picture of overall distribution. In contrast, the historical models learned from the historical data can be seen as a better summarization of the past knowledge. Hence, model-based approach which focuses on transferring knowledge between past and present models was later on developed.

This approach aims to overcome forgetting by transferring knowledge between past and present models. Similar to the regularization-based methods in continual learning, (Wang et al., 2020) incorporates a knowledge distillation loss to regularize the model update, using previous incremental model as the teacher model. (Xu et al., 2020) and (Mi et al., 2020) also employ the distillation techniques and specifically target at GNN-based and session-based RSs respectively. However, simply constraining the change in networks does not guarantee that the knowledge useful for future prediction is extracted. A recently proposed method termed Sequential Meta-Learning (SML) (Zhang et al., 2020) achieves this by introducing a mechanism to explicitly optimize for the next period data. It devises a transfer module to combine the past and present models, which is adaptively trained in a sequential manner to optimize for future serving. However, the major limitation of both SML and the distillation methods is that they only consider transfer between models of two consecutive incremental periods, ignoring the long-term sequential patterns that may be highly valuable for generating a better model for future serving.

Conventional incremental update of RS models involves restoring the last period model as initialization and updating it with the newly arrived data to obtain the most recent model. Formally, let $D_{t}$ denote the dataset collected from period $T_{t}$ for $t\in\{1,2,3,...\}$, whereby a period can be a certain length of time (e.g., a day, a week) or until a certain amount of data is collected. The model $\Theta_{t}$ updated from $\Theta_{t-1}$ is obtained by minimizing loss on $D_{t}$:

where $\mathcal{L}(\Theta|D_{t},\Theta_{t-1})$ is the loss on $D_{t}$ with $\Theta_{t-1}$ as initialization (for recommendation tasks, the loss can be pointwise log loss or pairwise BPR loss). The initial model $\Theta_{0}$ can be obtained by random initialization or pre-training with sufficient amount of historical data. Without any post-processing, the updated model $\Theta_{t}$ is directly deployed to serve for period $T_{t+1}$, from which $D_{t+1}$ can be obtained for training the next period model $\Theta_{t+1}$. The overall procedure of conventional incremental update is illustrated in Figure 1(a).

Although this updating method is widely adopted due to its efficiency and effectiveness, the model trained in this manner can easily overfit to the current data and forget past patterns learned. To address this issue, we propose an Adaptive Sequential Model Generation (ASMG) framework to apply on a sequence of incrementally updated models.

For the ASMG framework, instead of direct deployment of the updated model, we introduce a meta generator to take in a sequence of historical models (including the newly updated one), and generate a better model to be deployed for serving the next period. The input sequence of models are obtained from regular incremental update, which means that the model at each period is only trained on the data collected from that period. This allows us to obtain a sequence of models that are highly representative of the respective period. Mining the temporal trends exist in this sequence can have great potential in generating a better model for the next period’s serving. Formally, let $\mathcal{M}_{t}$ denote the meta generator that takes in a sequence of models until $\Theta_{t}$, the output model $\Theta^{*}_{t}$ used for serving $T_{t+1}$ is obtained by:

where $\mathbf{\Theta}_{1:t}$ is a sequence of models of length $t$ from $\Theta_{1}$ to $\Theta_{t}$. Note that $\Theta_{0}$ is excluded from the input sequence, as it is obtained from random initialization or pre-training instead of regular incremental update.

The desired properties of an ideal meta generator include 1) good capability of modeling the temporal dependencies exist in the sequence, and 2) stable performance in generating a good model for the next period’s serving. The first property is related to the specific network design of the meta generator, which will be discussed in details in the next section. The second property, on the other hand, can be achieved via proper optimization process of the meta generator.

To generate model that is particularly good at serving the next period, we need to ensure that past knowledge that is specially useful for the next period’s serving is extracted. To achieve this, we propose to update the meta generator by adaptively optimizing the output model towards the next period data. More specifically, let $\omega_{t}$ denote the parameters of $\mathcal{M}_{t}$, i.e., $\mathcal{M}_{t}=\mathcal{M}_{\omega_{t}}$, the parameters $\omega_{t+1}$ are updated from $\omega_{t}$ by optimizing $\Theta^{*}_{t}$ towards $D_{t+1}$:

where $\mathcal{L}(\Theta^{*}_{t}|D_{t+1},\omega_{t})$ is the loss on $D_{t+1}$ with $\omega_{t}$ as initialization. The loss function here is the same as that in (1).

To ensure that the meta generator is sufficiently trained before it can be deployed for online model generation, we perform $\tau$ periods of meta generator warm-up training prior to the online deployment. During the online deployment phase, the meta generator will still need to be adaptively updated in order to keep up with the most recent sequential trends. The overall procedure of the ASMG framework is depicted in Figure 1(b) and described in Algorithm 1.

There are many possible designs for the meta generator. In this work, we propose to employ Gated Recurrent Unit (GRUs) (Chung et al., 2014) to leverage its ability to capture the sequential patterns. For simplicity, we apply the GRU meta generator coordinate-wise on the base model $\Theta$. That is to say, for each individual $\theta\in\Theta$, we generate the output parameter $\theta^{*}$ by applying the GRU meta generator on a sequence of its historical values, independently of parameters at all the other coordinates. The final serving model is therefore $\Theta^{*}=\{{\theta^{*}}^{\prime}s\}$. More specifically, at each step $i\in\{1,...,t\}$, the hidden state $h_{i}\in\mathbb{R}^{d}$ is computed from the last step hidden state $h_{i-1}\in\mathbb{R}^{d}$ and $\theta\in\mathbb{R}$ of the current step input model $\Theta_{i}\in\mathbb{R}^{n}$:

where $r_{i},z_{i}\in\mathbb{R}^{d}$ are gates that control how much of past and present information to be retained, $W_{r},W_{z},W_{\tilde{h}}\in\mathbb{R}^{d\times(d+1)}$ are learnable weights, $\sigma(\cdot)$ is the sigmoid function, $[\cdot,\cdot]$ denotes concatenation, and $\odot$ denotes element-wise multiplication. The initial hidden state $h_{0}$ is zero-initialized.

The output parameter $\theta^{*}\in\mathbb{R}$ of the final serving model $\Theta_{i}^{*}\in\mathbb{R}^{n}$ at step $i$ is obtained from the respective hidden state $h_{i}$ via a linear transformation:

where $W\in\mathbb{R}^{d}$ and $b\in\mathbb{R}$ are also learnable parameters. All the learnable parameters are shared across $t$ steps.

If we assign one unique GRU meta generator (i.e., unshared set of learnable parameters) at each coordinate in the base model, the space complexity of the GRU meta generator will be $\mathcal{O}(nd^{2})$ where $d$ is the GRU hidden size and $n$ is the base model parameter size. Note that the size of the GRU meta generator is unrelated to the sequence length.

In this section, we introduce two training strategies specifically tailored to the GRU meta generator to further improve its training efficiency and sequential modeling ability.

The proposed ASMG-GRU is model-agnostic. To test its effectiveness, we instantiate it on a general deep learning-based Embedding&MLP model, a network architecture that most of the Click-Through Rate (CTR) prediction models developed in recent years are based on (Shan et al., 2016; Cheng et al., 2016; Guo et al., 2017; Zhou et al., 2018). The model mainly comprises embedding layers that transform high-dimensional sparse features into low-dimensional dense vectors, and Multi-Layer Perceptron (MLP) layers that learn the interaction of the concatenated feature embeddings.

More specifically, given user $i$ with $P$ categorical features, the concatenated feature embedding $u_{i}$ is obtained by:

where $E_{up}\in\mathbb{R}^{d_{e}\times d_{up}}$ with $d_{e}\ll d_{up}$ is the embedding matrix for categorical feature $p\in\{1,\cdots,P\}$ of user, and $z_{ip}\in\mathbb{R}^{d_{up}}$ is the one-hot or multi-hot vector for categorical feature $p$ of user $i$.

Similarly, for item $j$ with $Q$ categorical features, the concatenated feature embedding $v_{j}$ is obtained by:

where $E_{vq}\in\mathbb{R}^{d_{e}\times d_{vq}}$ with $d_{e}\ll d_{vq}$ is the embedding matrix for categorical feature $q\in\{1,\cdots,Q\}$ of item, and $z_{jq}\in\mathbb{R}^{d_{vq}}$ is the one-hot or multi-hot vector for categorical feature $q$ of item $j$.

The predicted preference of user $i$ on item $j$ is then obtained by passing the concatenated feature embeddings through a two-layer MLP, followed by a sigmoid function to normalize the score:

The parameters of the Embedding&MLP base model $\Theta$ therefore include both the embedding matrices $\{\mathbf{E}_{u},\mathbf{E}_{v}\}$, and the weights and biases of $\text{MLP}(\cdot)$. For CTR prediction, we adopt log loss to update the base model $\Theta$ as described in (1):

where $D^{-}_{t}$ is obtained by randomly sampling 1 unobserved interaction for each observed interaction in $D^{+}_{t}$.

To apply the proposed ASMG-GRU, we assign one unique meta generator to each individual weight and bias parameter in the MLP layers. However, for the embedding layers of sparse ID features (i.e., user ID and item ID), the parameter size is large while the training samples for each ID are limited. Assigning one unique meta generator for each parameter in the embedding layers significantly increases the complexity and can easily lead to overfitting. Hence, we assign one unique meta generator at each dimension of the embedding vector and share it across all IDs. Take for instance, for the $p$-th user embedding matrix $E_{up}\in\mathbb{R}^{d_{e}\times d_{up}}$, we assign $d_{e}$ distinct meta generators along the first dimension, and share them across $d_{up}$ IDs along the second dimension. The space complexity of the GRU meta generators for $E_{up}$ will therefore be significantly reduced from $\mathcal{O}(d_{e}d_{up}d^{2})$ to $\mathcal{O}(d_{e}d^{2})$, as $d_{up}$ is often large.

Table 1 presents the overall performance for the compared methods, from which we have the following observations:

• •

  BU methods with $w\geq 2$ are not comparable to IU in terms of serving for the next period, and the degradation increases for larger window size. This observation is consistent with (Frigó et al., 2017; Wang et al., 2020), which have identified that in non-stationary environment, batch update is inferior as compared to incremental update. This may be due to that the newly collected data can better represent the latest trends, which are highly related to the upcoming period. Expanding the window size affects the recency of the training data, and hence will be less effective in capturing the latest trends and fast-changing short-term user interests for the near future prediction.

• •

  SPMF and IncCTR only improve over IU marginally. The former is a sample-based approach. The latter is a model-based approach but only utilizes the historical model indirectly via knowledge distillation. The fact that SML performs the best among all the baseline methods demonstrates that directly incorporating the previous model to derive the transfer patterns can be very effective for generating a better current model. Moreover, the mechanism of SML to train the transfer module by optimizing towards the next period data helps extract knowledge that is particularly useful for future prediction.

• •

  SML-MF is not comparable to other methods due to a different choice of base model. This justifies our choice of a deep learning-based Embedding&MLP model which is more sophisticated in modeling complex feature interaction. Another benefit of this base model of our choice is that it is general in the sense that many state-of-the-art RS models are developed from the same architectural paradigm. Being able to achieve success on this model implies that similar effects may also be obtained for other deep learning models with similar architectures.

• •

  Under the proposed ASMG framework, the improvement of ASMG-Linear over IU is minor, and it is sometimes outperformed by other baselines that do not make use of the historical models. This shows that choosing the right design for the meta generator is important, and a simple linear combination is not suitable/sufficient to model the sequential effects. Also, having to drop the models outside of the sliding window limits its ability to accumulate longer-term information.

• •

  Both ASMG-GRUsingle and ASMG-GRUmulti outperform all the baselines by a significant margin of around $1\%$ in AUC on all four datasets. Note that in the industry, $1\%$ increase in offline AUC is considered very significant (Guo et al., 2017; Zhou et al., 2018), and it is likely to translate to greater improvement in online CTR (Cheng et al., 2016). The strong results demonstrate the benefits of 1) directly incorporating the historical models and learning by optimizing towards the future (vs SPMF and IncCTR), 2) involving a longer sequence for more informative sequential mining (vs SML), and 3) employing the GRU-based meta generator which is a better design for sequential modeling (vs ASMG-Linear).

Figure 3 shows the disentangled performance at each test period. We can see that both ASMG-GRUsingle and ASMG-GRUmulti consistently outperforms all the other methods at each test period. We purposely divide the periods of different datasets to have different time spans, so that we can test the robustness of our method to different model updating frequencies. Furthermore, we observe that there are drastic fluctuations across periods, which indicates the dynamically changing sequential trends and validates the necessity to adaptively update the meta generator.

In this section, we focus on ASMG-GRUmulti and test the efficacy of its various designs in regard to the training of GRU meta generator. We compare ASMG-GRUmulti with the following variants, each disables a different design:

• •

  ASMG-GRUfull: This variant trains GRU meta generator on the full sequence of historical models. Illustration of this variant is shown in Appendix A Figure 5.

• •

  ASMG-GRUzero: This variant uses zero-initialized input hidden state for every period’s training instead of continuing on a previously learned hidden state. Illustration of this variant is shown in Appendix A Figure 5.

• •

  ASMG-GRUunif: This variant assigns uniform weights to the loss at different time steps. Illustration of this variant is the same as ASMG-GRUmulti, as the weights assigned are not explicitly shown in the diagram.

• •

  ASMG-GRUsingle: This variant performs single-step training at the last output model towards the newly collected data only. Illustration of this variant is shown in Figure 2(b).

We further conduct sensitivity analysis to investigate the effects of two important factors, input sequence length and GRU hidden size, on the performance of ASMG-GRUmulti. Here we only present the analysis in AUC. For results in LogLoss, see Appendix B.