MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction

The task of CTR prediction in online advertising is to build a prediction model to estimate the probability of a user clicking on a specific ad. Each instance can be described by multiple fields such as user information (“User ID”, “City”, “Age”, etc.) and ad information (“Creative ID”, “Campaign ID”, “Title”, etc.). The instantiation of a field is a feature. For example, the “User ID” field may contain features such as “2135147” and “3467291”. Table 1 shows some examples.

We define the cross-domain CTR prediction problem as leveraging the data from a source domain (or source domains) to improve the CTR prediction performance in a target domain. In news feed advertising (e.g., UC Toutiao shown in Figure 1), the source domain is the natural news feed and the target domain is the advertisement. In this scenario, the source domain and the target domain share the same set of users, but there are no overlapped items.

In order to effectively leverage cross-domain data, we propose the Mixed Interest Network (MiNet) in Figure 2, which models three types of user interest: 1) Long-term interest across domains. It is modeled by the concatenation of user profile feature embeddings $\mathbf{p}_{u}$, which is jointly learned based on cross-domain data. 2) Short-term interest from the source domain. It is modeled by the vector $\mathbf{a}_{s}$, which aggregates the information of recently clicked news in the source domain. 3) Short-term interest in the target domain. It is modeled by the vector $\mathbf{a}_{t}$, which aggregates the information of recently clicked ads in the target domain.

In MiNet, we apply two levels of attentions, one on the item level and the other on the interest level. The aim of the item-level attention is to dynamically distill useful information (relevant to the target ad whose CTR is to be predicted) from recently clicked news / ads and suppress noise. The aim of the interest-level attention is to adaptively adjust the importance of the three types of user interest (i.e., $\mathbf{p}_{u}$, $\mathbf{a}_{s}$ and $\mathbf{a}_{t}$) and to emphasize more informative signal(s) for the target ad. In the following, we detail our model components.

We first encode features into one-hot encodings. For the $i$th feature, its one-hot encoding is $\mathbf{v}_{i}=\textrm{one-hot}(i)$, where $\mathbf{v}_{i}\in\mathbb{R}^{N}$ is a vector with 1 at the $i$th entry and 0 elsewhere, and $N$ is the number of unique features. To enable cross-domain knowledge transfer, the set of unique features includes all the features in both domains.

We then map sparse, high-dimensional one-hot encodings to dense, low-dimensional embedding vectors suitable for neural networks (Mikolov et al., 2013). In particular, we define an embedding matrix $\mathbf{E}\in\mathbb{R}^{D\times N}$ (to be learned; $D$ is the embedding dimension; $D\ll N$). The $i$th feature is then projected to its corresponding embedding vector $\mathbf{e}_{i}\in\mathbb{R}^{D}$ as $\mathbf{e}_{i}=\mathbf{E}\mathbf{v}_{i}$. Equivalently, the embedding vector $\mathbf{e}_{i}$ is the $i$th column of the embedding matrix $\mathbf{E}$.

For each ad instance, we split its features into user features and ad features. We take out all ad features and concatenate the corresponding feature embedding vectors to obtain an ad representation vector $\mathbf{r}_{t}\in\mathbb{R}^{D_{t}}$ in the target domain. Similarly, we can obtain a news representation vector $\mathbf{r}_{s}\in\mathbb{R}^{D_{s}}$ in the source domain.

For a user $u$, her long-term interest representation vector $\mathbf{p}_{u}\in\mathbb{R}^{D_{u}}$ is obtained by concatenating the corresponding user feature embedding vectors ($D_{t}$, $D_{s}$ and $D_{u}$ are data-related dimensions). For example, if user $u$ has features “UID = u123, City = BJ, Gender = male, OS = ios”, we have

where $\|$ is the vector concatenation operation.

This long-term interest representation vector $\mathbf{p}_{u}$ is shared across domains and is jointly learned using the data from both domains. The detailed process will be described in §2.8 and §2.9.

Given a user, for each target ad whose CTR is to be predicted in the target domain, the user usually viewed pieces of news in the source domain. Although the content of a piece of news can be completely different from that of the target ad, there may exist certain correlation between them. For example, a user has a high probability to click a Game ad after viewing some Entertainment news. Based on such relationships, we can transfer useful knowledge from the source domain to the target domain.

Denote the set of representation vectors of recently clicked news as $\{\mathbf{r}_{si}\}_{i}$. Because the number of pieces of clicked news may be different from time to time, we need to aggregate these pieces of news. In particular, the aggregated representation $\mathbf{a}_{s}$ is given by

where $\alpha_{i}$ is a weight assigned to $\mathbf{r}_{si}$ to indicate its importance during aggregation. The aggregated representation $\mathbf{a}_{s}$ reflects the short-term interest of the user from the source domain.

The problem remaining is how to compute the weight $\alpha_{i}$. One simple way is to set $\alpha_{i}=1/|\{\mathbf{r}_{si}\}_{i}|$. That is, each piece of clicked news has equal importance. This is clearly not a wise choice because some news may not be indicative for the target ad.

Alternatively, the attention mechanism (Bahdanau et al., 2015) offers us a better way of computing $\alpha_{i}$. It is firstly introduced in the encoder-decoder framework for the machine translation task. Nevertheless, how to use it is still flexible. One possible way is to compute $\alpha_{i}$ as

where $\mathbf{W}_{s}$ and $\mathbf{h}_{s}$ are model parameters; $ReLU$ is the rectified linear unit ($ReLU(x)=\max(0,x)$) as an activation function. Nair and Hinton (Nair and Hinton, 2010) show that ReLU has significant benefits over sigmoid and tanh activation functions in terms of the convergence rate and the quality of obtained results.

However, Eq. (3) only considers each piece of clicked news $\mathbf{r}_{si}$ alone. It does not capture the relationship between a piece of clicked news and the target ad. Moreover, Eq. (3) is not tailored to the target user as well. For example, no matter the target ad is about coffee or clothing, or the target user is $u_{a}$ or $u_{b}$, the importance of a piece of clicked news keeps the same.

Given a user, for each target ad whose CTR is to be predicted, the user also has recent behavior in the target domain. What ads the user has recently clicked may have a strong implication for what ads the user may click in the short future.

Denote the set of representation vectors of recently clicked ads as $\{\mathbf{r}_{tj}\}_{j}$. We compute the aggregated representation $\mathbf{a}_{t}$ as

where $\mathbf{W}_{t}\in\mathbb{R}^{D_{h}\times(3D_{t}+D_{u})}$ and $\mathbf{h}_{t}\in\mathbb{R}^{D_{h}}$ are parameters to be learned. The representation $\mathbf{a}_{t}$ reflects the short-term interest of the user in the target domain.

Eq. (6) considers the following aspects:

• •

  The clicked ad $\mathbf{r}_{tj}\in\mathbb{R}^{D_{t}}$ in the target domain.

• •

  The target ad $\mathbf{q}_{t}\in\mathbb{R}^{D_{t}}$ in the target domain.

• •

  The target user $\mathbf{p}_{u}\in\mathbb{R}^{D_{u}}$.

• •

  The interaction $\mathbf{r}_{tj}\odot\mathbf{q}_{t}\in\mathbb{R}^{D_{t}}$ between the clicked ad and the target ad. Because they are in the same domain, no transfer matrix is needed.

Similarly, the computed $\tilde{\beta}_{j}$ ($\beta_{j}$) is not only a function of the clicked ad which needs to be assigned a weight, but is also aware of the target ad and the target user.

After we obtain the three types of user interest $\mathbf{p}_{u}\in\mathbb{R}^{D_{u}}$, $\mathbf{a}_{s}\in\mathbb{R}^{D_{s}}$ and $\mathbf{a}_{t}\in\mathbb{R}^{D_{t}}$, we use them together to predict the CTR of the target ad $\mathbf{q}_{t}\in\mathbb{R}^{D_{t}}$. Although $\mathbf{p}_{u}$, $\mathbf{a}_{s}$ and $\mathbf{a}_{t}$ all represent user interest, they reflect different aspects and have different dimensions. We thus cannot use weighted sum to fuse them. One possible solution is to concatenate all available information as a long input vector

However, such a solution cannot find the most informative user interest signal for the target ad $\mathbf{q}_{t}$. For example, if both the short-term interest $\mathbf{a}_{s}$ and $\mathbf{a}_{t}$ are irrelevant to the target ad $\mathbf{q}_{t}$, the long-term interest $\mathbf{p}_{u}$ should be more informative. But $\mathbf{p}_{u}$, $\mathbf{a}_{s}$ and $\mathbf{a}_{t}$ have equal importance in $\mathbf{m}$.

Therefore, instead of forming $\mathbf{m}$, we actually form $\mathbf{m}_{t}$ as follows

where $v_{u}$, $v_{s}$ and $v_{t}$ are dynamic weights that tune the importance of different user interest signals based on their actual values.

In particular, we compute these weights as follows:

where $\mathbf{V}_{*}\in\mathbb{R}^{D_{h}\times(D_{s}+2D_{t}+D_{u})}$ is a matrix parameter, $\mathbf{g}_{*}\in\mathbb{R}^{D_{h}}$ is a vector parameter and $b_{*}$ is a scalar parameter. The introduction of $b_{*}$ is to model the intrinsic importance of a particular type of user interest, regardless of its actual value. It is observed that these weights are computed based on all the available information so as to take into account the contribution of a particular type of user interest to the target ad, given other types of user interest signals.

It is also observed that we use $\exp(\cdot)$ to compute the weights, which makes $v_{*}$ may be larger than 1. It is a desirable property because these weights can compensate for the dimension discrepancy problem. For example, when the dimension of $\mathbf{q}_{t}$ is much larger than that of $\mathbf{p}_{u}$ (due to more features), the contribution of $\mathbf{p}_{u}$ would be naturally weakened by this effect. Assigning $\mathbf{p}_{u}$ with a weight in $[0,1]$ (i.e., replacing $\exp(\cdot)$ by the sigmoid function) cannot address this issue. Nevertheless, as these weights are automatically learned, they could be smaller than 1 as well when necessary.

In the target domain, we let the input vector $\mathbf{m}_{t}$ go through several fully connected (FC) layers with the ReLU activation function, in order to exploit high-order feature interaction as well as nonlinear transformation (He and Chua, 2017). Formally, the FC layers are defined as follows:

where $L$ denotes the number of hidden layers; $\mathbf{W}_{l}$ and $\mathbf{b}_{l}$ denote the weight matrix and bias vector (to be learned) in the $l$th layer.

Finally, the vector $\mathbf{z}_{L}$ goes through an output layer with the sigmoid function to generate the predicted CTR of the target ad as

where $\mathbf{w}$ and $b$ are the weight and bias parameters to be learned.

To facilitate the learning of long-term user interest $\mathbf{p}_{u}$, we also create an input vector for the source domain as $\mathbf{m}_{s}\triangleq[\mathbf{q}_{s}\|\mathbf{p}_{u}]$, where $\mathbf{q}_{s}\in\mathbb{R}^{D_{s}}$ is the concatenation of the feature embedding vectors for the target news. Similarly, we let $\mathbf{m}_{s}$ go through several FC layers and an output layer (with their own parameters). Finally, we obtain the predicted CTR $\hat{y}_{s}$ of the target news.

We use the cross-entropy loss as our loss function. In the target domain, the loss function on a training set is given by

where $y_{t}\in\{0,1\}$ is the true label of the target ad corresponding to the estimated CTR $\hat{y}_{t}$ and $\mathbb{Y}_{t}$ is the collection of true labels. Similarly, we have a loss function $loss_{s}$ in the source domain.

All the model parameters are learned by minimizing the combined loss as follows

where $\gamma$ is a balancing hyperparameter. As $\mathbf{p}_{u}$ is shared across domains, when optimizing the combined loss, $\mathbf{p}_{u}$ is jointly learned based on the data from both domains.

The statistics of the datasets are listed in Table 2.

1) Company News-Ads dataset. This dataset contains a random sample of news and ads impression and click logs from the news system and the ad system in UC Toutiao. The source domain is the news and the target domain is the ad. We use logs of 6 consecutive days in 2019 for initial training, logs of the next day for validation, and logs of the day after the next day for testing. After finding the optimal hyperparameters on the validation set, we combine the initial training set and the validation set as the final training set (trained using the found optimal hyperparameters). The features used include 1) user features such as user ID, agent and city, 2) news features such as news title, category and tags, and 3) ad features such as ad title, ad ID and cateogry.

2) Amazon Books-Movies dataset. The Amazon datasets (McAuley et al., 2015) have been widely used to evaluate the performance of recommender systems. We use the two largest categories, i.e., Books and Movies & TV, for the cross-domain CTR prediction task. The source domain is the book and the target domain is the movie. We only keep users that have at least 5 ratings on items with metadata in each domain. We convert the ratings of 4-5 as label 1 and others as label 0. To simulate the industrial practice of CTR prediction (i.e., to predict the future CTR but not the past), we sort user logs in chronological order and take out the last rating of each user to form the test set, the second last rating to form the validation set and others to form the initial training set. The features used include 1) user features: user ID, and 2) book / movie features: item ID, brand, title, main category and categories.

We compare both single-domain and cross-domain methods. Existing cross-domain methods are mainly proposed for cross-domain recommendation and we extend them for cross-domain CTR prediction when necessary (e.g., to include attribute features rather than only IDs and to change the loss function).

We set the dimension of the embedding vectors for each feature as $D=10$. We set $C=10$ and the number of FC layers in neural network-based models as $L=2$. The dimensions are [512, 256] for the Company dataset and [256, 128] for the Amazon dataset. We set $D_{h}=128$ for the Company dataset and $D_{h}=64$ for the Amazon dataset. The batch sizes for the source and the target domains are set to (512, 128) for the Company dataset and (64, 32) for the Amazon dataset. All the methods are implemented in Tensorflow and optimized by the Adagrad algorithm (Duchi et al., 2011). We run each method 5 times and report the average results.

1. (1)

   AUC: the Area Under the ROC Curve over the test set (target domain). It is a widely used metric for CTR prediction. It reflects the probability that a model ranks a randomly chosen positive instance higher than a randomly chosen negative instance. The larger the better. A small improvement in AUC is likely to lead to a significant increase in online CTR.

2. (2)

   RelaImpr: RelaImpr is introduced in (Yan et al., 2014) to measure the relative improvement of a target model over a base model. Because the AUC of a random model is 0.5, it is defined as: $RelaImpr=\left(\frac{\textrm{AUC(target model)}-0.5}{\textrm{AUC(base model)}-0.5}-1\right)\times 100\%$.

3. (3)

   Logloss: the value of Eq. (10) over the test set (target domain). The smaller the better.

Table 3 lists the AUC and Logloss values of different methods. It is observed that in terms of AUC, shallow models such as LR and FM perform worse than deep models. FM performs better than LR because it further models second-order feature interactions. Wide&Deep achieves higher AUC than LR and DNN, showing that combining LR and DNN can improve the prediction performance. Among the single-domain methods, DSTN performs best. It is because DSTN jointly considers various spatial and temporal factors that could impact the CTR of the target ad.

As to the cross-domain methods, CCCFNet outperforms LR and FM, showing that using cross-domain data can lead to improved performance. CCCFNet performs worse than other cross-domain methods because it compresses all the attribute features. MV-DNN performs similarly as MLP++. They both achieve knowledge transfer through embedding sharing. CoNet introduces cross connection units on MLP++ to enable dual knowledge transfer across domains. However, this also introduces higher complexity and random noise. CCCFNet, MV-DNN, MLP++ and CoNet mainly consider long-term user interest. In contrast, our proposed MiNet considers not only long-term user interest, but also short-term interest in both domains. With proper combination of these different interest signals, MiNet outperforms other methods significantly.

In this section, we examine the effect of the level of attentions in MiNet. In particular, we examine the following settings: 1) No attention, 2) Only item-level attention, 3) Only interest-level attention [with the exponential activation function as proposed in Eq. (2.7)], 4) Only interest-level attention but with the sigmoid activation function [i.e., replacing $\exp$ by sigmoid in Eq. (2.7)], and 5) Both attentions. It is observed in Figure 3 that “No attention” performs worst. This is because useful signals could be easily buried in noise without distilling. Either item-level attention or interest-level attention can improve the AUC, and the use of both levels of attentions results in the highest AUC. Moreover, “Interest attention (sigmoid)” has much worse performance than “Interest attention (exp)”. This is because improper activation function cannot effectively tackle the dimension discrepancy problem. These results demonstrate the effectiveness of the proposed hierarchical attention mechanism.

In this section, we examine the item-level attention weights in MiNet and check whether they can capture informative signals. Figure 4 shows two different target ads with the same set of clicked ads and clicked news for an active user in the Company dataset. For the privacy issue, we only present ads and news at the category granularity. Since ads are in the same domain, it is relatively easy to judge the relevance between a clicked ad and the target ad. Nevertheless, as ads and news are in different domains, it is hard to judge their relevance. We thus calculate the probability $p(Ad|News)$ based on the user’s behavior logs.

It is observed in Figure 4 that when the target ad is Publishing & Media (P&M), the clicked ad of P&M has the highest weight and the clicked news of Entertainment has the highest weight; but when the target ad is Game, the clicked ad of Game has the highest weight and the clicked news of Sports has the highest weight. These results show that the item-level attentions do dynamically capture more important information for different target ads. It is also observed that the model can learn certain correlation between a piece of clicked news and the target ad. News with a higher indication probability usually receives a higher attention weight.

In this section, we examine the effect of modeling different types of user interest in MiNet. We observe quite different phenomena on the two datasets in Figure 5. On the Company dataset, modeling short-term interest can lead to much higher AUC than modeling long-term interest, showing that recent behaviors are quite informative in online advertising. In contrast, on the Amazon dataset, modeling long-term interest results in much higher AUC. It is because the Amazon dataset is an e-commerce dataset rather than advertising and the nature of ratings are different from that of clicks. Nevertheless, when all these aspects are jointly considered in MiNet, we obtain the highest AUC, showing that different types of interest can complement each other and joint modeling can lead to the best and more robust performance.

We deployed MiNet in UC Toutiao, where the ad serving system architecture is shown in Figure 6. We conducted online experiments in an A/B test framework over two weeks during Dec. 2019 - Jan. 2020, where the base serving model is DSTN (Ouyang et al., 2019a). Our online evaluation metric is the real CTR, which is defined as the number of clicks over the number of ad impressions. A larger online CTR indicates the enhanced effectiveness of a CTR prediction model. The online A/B test shows that MiNet leads to an increase of online CTR of 4.12% compared with DSTN. This result demonstrates the effectiveness of MiNet in practical CTR prediction tasks. After the A/B test, MiNet serves the main ad traffic in UC Toutiao.