Exploiting Variational Domain-Invariant User Embedding for Partially Overlapped Cross Domain Recommendation

First, we describe notations. We assume there are two domains, i.e., a source domain $\mathcal{S}$ and a target domain $\mathcal{T}$. We assume $\mathcal{S}$ has $|\boldsymbol{U}^{\mathcal{S}}|$ users and $|\boldsymbol{V}^{\mathcal{S}}|$ items, and $\mathcal{T}$ has $|\boldsymbol{U}^{\mathcal{T}}|$ users and $|\boldsymbol{V}^{\mathcal{T}}|$ items. Unlike conventional cross domain recommendation, in POCDR, the source and target user sets are not completely the same. We use the overlapped user ratio $\mathcal{K}_{u}\in(0,1)$ to measure how many users are concurrence. Bigger $\mathcal{K}_{u}$ means more overlapped users across domains, while smaller $\mathcal{K}_{u}$ means the opposite situation.

Then, we introduce the overview of our proposed VDEA framework. The aim of VDEA is providing better single domain recommendation performance by leveraging useful cross domain knowledge for both overlapped and non-overlapped users. VDEA model mainly has two modules, i.e., variational rating reconstruction module and variational embedding alignment module, as shown in Fig. 3. The variational rating reconstruction module establishes user embedding according to the user-item interactions on the source and target domains. Meanwhile, the variational rating reconstruction module also clusters the users with similar characteristics or preferences based on their corresponding latent embeddings to enhance the model generalization. The basic intuition is that users with similar behaviors should have close embedding distances through this module with deep unsupervised clustering approach (Wang et al., 2019a; Ma et al., 2019; Wang et al., 2019b). The variational embedding alignment module has two parts, i.e., variational local embedding alignment and variational global embedding alignment, sufficiently aligning the local and global user embeddings across domains. The local embedding alignment can utilize the user-item ratings of the overlapped users to obtain more expressive user embeddings (Krafft and Schmitz, 1969). Intuitively, expressive overlapped user embedding can enhance model training of the non-overlapped users, since the non-overlapped users share the similar preference (e.g., romantic or realistic) as the overlapped ones (Zhang et al., 2018). Moreover, the global embedding alignment can also directly co-cluster both the overlapped and non-overlapped users with similar tastes, which can boost the procedure of knowledge transfer. To this end, the variational embedding alignment module can exploit domain-invariant latent embeddings for both overlapped and non-overlapped users. The framework of VDEA is demonstrated in Fig. 3 and we will introduce its two modules in details later.

To begin with, we first introduce the variational rating reconstruction module. We define $\boldsymbol{X}^{\mathcal{S}}\in\mathbb{R}^{N\times|\boldsymbol{V}^{\mathcal{S}}|}$ and $\boldsymbol{X}^{\mathcal{T}}\in\mathbb{R}^{N\times|\boldsymbol{V}^{\mathcal{T}}|}$ as the user-item interactions on the source and target domains respectively, and $N$ is batch size. We adopt dual variational autoencoder as the main architecture for accurately representing and clustering user embedding according to the rating interactions. We utilize two variational autoenocers in VDEA, i.e., a source variational autoencoder and a target variational autoencoder, for source and target domains, respectively. The source variational autoencoder includes the encoder $E_{\mathcal{S}}$ and decoder $D_{\mathcal{S}}$. Similarly, the target variational autoencoder includes the encoder $E_{\mathcal{T}}$ and decoder $D_{\mathcal{T}}$. The encoders $E_{\mathcal{S}}$ and $E_{\mathcal{T}}$ can predict the mean ($\boldsymbol{\mu}^{\mathcal{S}}$ and $\boldsymbol{\mu}^{\mathcal{T}}$) and log variance ($\log(\boldsymbol{\sigma}^{\mathcal{S}})^{2}$ and $\log(\boldsymbol{\sigma}^{\mathcal{T}})^{2}$) for the input data as $[\boldsymbol{\mu}^{\mathcal{S}},\log(\boldsymbol{\sigma}^{\mathcal{S}})^{2}]=E_{\mathcal{S}}(\boldsymbol{X}^{\mathcal{S}})$ and $[\boldsymbol{\mu}^{\mathcal{T}},\log(\boldsymbol{\sigma}^{\mathcal{T}})^{2}]=E_{\mathcal{T}}(\boldsymbol{X}^{\mathcal{T}})$. In order to guarantee gradient descent in back propagation, we also apply the reparamaterize mechanism (Liang et al., 2018; Shenbin et al., 2020) as $\boldsymbol{Z}^{\mathcal{S}}=\boldsymbol{\mu}^{\mathcal{S}}+\epsilon^{\mathcal{S}}\boldsymbol{\sigma}^{\mathcal{S}}$ and $\boldsymbol{Z}^{\mathcal{T}}=\boldsymbol{\mu}^{\mathcal{T}}+\epsilon^{\mathcal{T}}\boldsymbol{\sigma}^{\mathcal{T}}$, where $\epsilon^{\mathcal{S}},\epsilon^{\mathcal{T}}\sim\mathcal{N}(0,I)$ are randomly sampled from normal distribution, and $\boldsymbol{\mu}^{\mathcal{S}},\boldsymbol{\mu}^{\mathcal{T}},\boldsymbol{\sigma}^{\mathcal{S}}$, and $\boldsymbol{\sigma}^{\mathcal{T}}\in\mathbb{R}^{N\times D}$ have the same dimension $D$. Inspired by variational deep embedding (Jiang et al., 2016), we expect the latent user embedding conform to Mixture-Of-Gaussian distribution for grouping similar users. The dual variational autoencoder also reconstructs the user-item interaction as $\hat{\boldsymbol{X}}_{\mathcal{S}}=D_{\mathcal{S}}(\boldsymbol{Z}^{\mathcal{S}})$ and $\hat{\boldsymbol{X}}_{\mathcal{T}}=D_{\mathcal{T}}(\boldsymbol{Z}^{\mathcal{T}})$. In this section, we will first present the variational evidence lower bound according to the Mixture-Of-Gaussian distributions, and then illustrate the rating reconstruction and distribution matching for the total loss. By proposing the variational rating reconstruction module, the users with similar characteristics will be clustered in each domain, as can be seen in Fig. 4(a).

We then adopt Jensen’s inequality with the mean-field assumption to estimate the Evidence Lower Bound (ELBO) by maximizing $\log p_{\mathcal{S}}(\boldsymbol{X}^{\mathcal{S}})$. Following (Jiang et al., 2016), ELBO can be rewritten in a simplified form as:

where the first term represents the rating reconstruction loss that encourages the model to fit the user-item interactions well. The second term calculates the Kullback-Leibler divergence between Mixture-of-Gaussians prior $q_{\mathcal{S}}(\boldsymbol{Z}^{\mathcal{S}},C^{\mathcal{S}}|\boldsymbol{X}^{\mathcal{S}})$ and the variational posterior $p_{\mathcal{S}}(\boldsymbol{Z}^{\mathcal{S}},\boldsymbol{C}^{\mathcal{S}})$. Furthermore, in order to balance the reconstruction capacity (the first term) efficiently and disentangle the latent factor (the second term) automatically, we change the KL divergence into the following optimization equation with constraints:

where $\delta$ specifies the strength of the KL constraint. Rewriting the optimization problem in Equation (LABEL:equ:opt) under the KKT conditions and neglecting the small number of $\epsilon$, we have:

where $\beta$ denotes the hyper-parameter between rating modelling and latent distribution matching (Higgins et al., 2016).

As we have motivated before, the embeddings of similar users should be clustered through variational deep embedding to make them more discriminative, as is shown in Fig. 4. However, how to align these Mixture-Of-Gaussian distribution across domains remains a tough challenge. In order to tackle this issue, we propose variational embedding alignment from both local and global perspectives.

In order to obtain the alignment results effectively, we propose the Gromov-Wasserstein Distribution Co-Clustering Optimal Transport (GDOT) approach. Unlike conventional optimal transport (Santambrogio, 2015; Villani, 2008) that only aligns user embeddings, our proposed GDOT for domain adaptation performs user embedding distribution alignment in the source and target domains. The optimization of optimal transport is based on the Kantorovich problem (Angenent et al., 2003), which seeks a general coupling $\psi\in\mathcal{X}(\mathcal{S},\mathcal{T})$ between $\mathcal{S}$ and $\mathcal{T}$:

where $\mathcal{X}(\mathcal{S},\mathcal{T})$ denotes the latent embedding probability distribution between $\mathcal{S}$ and $\mathcal{T}$. The cost function matrix $\mathcal{M}(\mathbb{P}^{\mathcal{S}}_{i,i^{\prime}},\mathbb{P}^{\mathcal{T}}_{j,j^{\prime}})$ measures the difference between each pair of distributions. To facilitate calculation, we formulate the discrete optimal transport formulation as $\psi^{*}=\mathop{\arg\min}_{\boldsymbol{\psi}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{S}},\boldsymbol{\psi}^{T}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{T}}}\langle\mathcal{{M}}\otimes\boldsymbol{\psi},\boldsymbol{\psi}\rangle_{F}-\epsilon H(\boldsymbol{\psi})$, where $\boldsymbol{\psi}^{*}\in\mathbb{R}^{K\times K}$ is the ideal coupling matrix between the source and target domains, representing the joint probability of the source probability $\mathbb{P}^{\mathcal{S}}$ and the target probability $\mathbb{P}^{\mathcal{T}}$. The matrix $\mathcal{{M}}\in\mathbb{R}^{K\times K\times K\times K}$ denotes a pairwise Gaussian distribution distance among the source probability $\mathbb{P}^{\mathcal{S}}_{i},\mathbb{P}^{\mathcal{S}}_{i^{\prime}}$ and target probability $\mathbb{P}^{\mathcal{T}}_{j},\mathbb{P}^{\mathcal{T}}_{j^{\prime}}$ as:

Meanwhile $\otimes$ denotes the tensor matrix multiplication, which means $[\mathcal{{M}}\otimes\boldsymbol{\psi}]_{ij}=(\sum_{i^{\prime},j^{\prime}}\mathcal{{M}}_{i,j,i^{\prime},j^{\prime}}\boldsymbol{\psi}_{i^{\prime},j^{\prime}})_{i,j}$. The second term $H(\boldsymbol{\psi})=-\sum_{i=1}^{K}\sum_{j=1}^{K}\psi_{ij}(\log(\psi_{ij})-1)$ is a regularization term and $\epsilon$ is a balance hyper parameter. The matching matrix follows the constraints that $\boldsymbol{\psi}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{S}},\boldsymbol{\psi}^{T}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{T}}$ with $K$ denoting the number of clusters. We apply the Sinkhorn algorithm (Cuturi, 2013; Xu et al., 2020) for solving the optimization problem by adopting the Lagrangian multiplier to minimize the objective function $\ell$ as below:

where $\boldsymbol{f}$ and $\boldsymbol{g}$ are both Lagrangian multipliers. We first randomly generate a matrix $\boldsymbol{\psi}$ that meets the boundary constraints and then define the variable $\mathcal{G}=\mathcal{M}\otimes\mathcal{\psi}$. Taking the differentiation of Equation (12) on variable $\boldsymbol{\psi}$, we can obtain:

Taking Equation (13) back to the original constraints $\boldsymbol{\psi}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{S}}$ and $\boldsymbol{\psi}^{T}\boldsymbol{1}_{K}=\boldsymbol{\pi}^{\mathcal{T}}$, we can obtain that:

Since we define $\boldsymbol{u}={\rm diag}\left(\frac{\boldsymbol{f}}{\epsilon}\right)$, $\boldsymbol{H}=\exp\left(-\frac{\mathcal{{G}}}{\epsilon}\right)$, and $\boldsymbol{v}={\rm diag}\left(\frac{\boldsymbol{g}}{\epsilon}\right)$. Through iteratively solving Equation (14) on the variables $\boldsymbol{u}$ and $\boldsymbol{v}$, we can finally obtain the coupling matrix $\boldsymbol{\pi}^{*}$ on Equation (13). We show the iteration optimization scheme of GDOT in Algorithm 1. In summary, variational embedding alignment can obtain the optimal transport solution as $\boldsymbol{\psi}^{*}$ for each corresponding sub-distribution, as shown with dot lines in Fig. 4(b).

After that, the variational global embedding alignment loss is given as:

The bigger $\psi^{*}_{ij}$ for the matching constraint, the closer between the distributions $\mathcal{N}(\breve{\boldsymbol{\mu}}^{\mathcal{S}}_{i},(\breve{\boldsymbol{\sigma}}^{\mathcal{S}}_{i})^{2})$ and $\mathcal{N}(\breve{\boldsymbol{\mu}}^{\mathcal{T}}_{j},(\breve{\boldsymbol{\sigma}}^{\mathcal{T}}_{j})^{2})$.

The total loss of VDEA could be obtained by combining the losses of the variational rating reconstruction module and the variational embedding alignment module. That is, the losses of VDEA is given as:

where $\lambda_{VL}$ and $\lambda_{VG}$ are hyper-parameters to balance different type of losses. By doing this, users with similar rating behaviors will have close embedding distance in single domain through the variational inference, and the user embeddings across domains will become domain-invariant through GDOT which has been shown in Fig. 4(c).

We conduct extensive experiments on two popularly used real-world datasets, i.e., Douban and Amazon, for evaluating our models on POCDR tasks (Zhu et al., 2019, 2021b; Ni et al., 2019). First, the Douban dataset (Zhu et al., 2019, 2021b) has two domains, i.e., Book and Movie, with user-item ratings. Second, the Amazon dataset (Zhao et al., 2020; Ni et al., 2019) has four domains, i.e., Movies and TV (Movie), Books (Book), CDs and Vinyl (Music), and Clothes (Clothes). The detailed statistics of these datasets after pre-process will be shown in Table 1. For both datasets, we binarize the ratings to 0 and 1. Specifically, we take the ratings higher or equal to 4 as positive and others as 0. We also filter the users and items with less than 5 interactions, following existing research (Yuan et al., 2019; Zhu et al., 2021b). We provide four main tasks to evaluate our model, i.e., Amazon Movie $\&$ Amazon Music, Amazon Movie $\&$ Amazon Book, Amazon Movie $\&$ Amazon Clothes, and Douban Movie $\&$ Douban Book. It is noticeable that some tasks are rather simple because the source and target domains are similar, e.g., Amazon Movie $\&$ Amazon Book, while some tasks are difficult since these two domains are different, e.g., Amazon Movie $\&$ Amazon Clothes (Yu et al., 2020).

We randomly divide the observed source and target data into training, validation, and test sets with a ratio of 8:1:1. Meanwhile, we vary the overlapped user ratio $\mathcal{K}_{u}$ in $\{30\%,60\%,90\%\}$. Different user overlapped ratio represents different situations, e.g., $\mathcal{K}_{u}=30\%$ represents only few users are overlapped while $\mathcal{K}_{u}=90\%$ means most of users are overlapped (Zhang et al., 2018). For all the experiments, we perform five random experiments and report the average results. We choose Adam (Kingma and Ba, 2014) as optimizer, and adopt Hit Rate$@k$ (HR$@k$) and NDCG$@k$ (Wang et al., 2019c) as the ranking evaluation metrics with $k=5$.

We compare our proposed VDEA with the following state-of-the-art cross domain recommendation models. (1) KerKT (Zhang et al., 2018) Kernel-induced knowledge transfer for overlapping entities (KerKT) which is the first attempt on POCDR problem. It uses a shallow model with multiple steps to model and align user representations. (2) NeuMF (He et al., 2017) Neural Collaborative Filtering for Personalized Ranking (NeuMF) replaces the traditional inner product with a neural network to learn an arbitrary function on the single domain accordingly. (3) MultVAE (Liang et al., 2018) Variational Autoencoders for Collaborative Filtering (MultVAE) extends variational autoencoders for recommendation. (4) BiVAE (Truong et al., 2021) Bilateral Variational Autoencoder for Collaborative Filtering (BiVAE) further adopts variational autoencoder on both user and item to model the rating interactions. (5) DARec (Yuan et al., 2019) Deep Domain Adaptation for Cross-Domain Recommendation via Transferring Rating Patterns (DARec) adopts adversarial training strategy to extract and transfer knowledge patterns for shared users across domains. (6) ETL (Chen et al., 2020b) Equivalent Transformation Learner (ETL) of user preference in CDR adopts equivalent transformation module to capture both the overlapped and domain-specific properties. (7) DML (Li and Tuzhilin, 2021) Dual Metric Learning for Cross-Domain Recommendations (DML) is the recent state-of-the-art CDR model which transfers information across domains based on dual modelling.

We set batch size $N=256$ for both the source and target domains. The latent embedding dimension is set to $D=128$. We assume both source and target users share the same embedding dimension according to previous research (Liu et al., 2017; Chen et al., 2020b). For the variational rating reconstruction module, we adopt the heuristic strategy for selecting the disentangle factor $\beta$ by starting from $\beta=0$ and gradually increasing to $\beta=0.2$ through KL annealing (Liang et al., 2018). Empirically, we set the cluster number as $K=30$. Meanwhile, we set hyper-parameters $\epsilon=0.1$ for Gromov-Wasserstein Distribution Co-Clustering Optimal Transport. For VDEA model, we set the balance hyper-parameters as $\lambda_{VL}=0.7$ and $\lambda_{VG}=1.0$.

The comparison results on Douban and Amazon datasets are shown in Table 1. From them, we can find that: (1) With the decrease of the overlapped user ratio $\mathcal{K}_{u}$, the performance of all the models will also decrease. This phenomenon is reasonable since fewer overlapped users may raise the difficulty of knowledge transfer across different domains. (2) The conventional shallow model KerKT with multiple separated steps cannot model the complex and nonlinear user-item relationship under the POCDR setting. (3) Most of the cross domain recommendation models (e.g., DML) perform better than single domain recommendation model (e.g., NeuMF), indicating that cross domain recommendation can integrate more useful information to enhance the model performance, which is always conform to our common sense. (4) Although ETL and DML can get reasonable results on conventional cross domain recommendation problem where two domains are highly overlapped, it cannot achieve satisfied solutions on the general POCDR problem. When the majority of users are non-overlapped, ETL and DML cannot fully utilize the knowledge transfer from the source domain to the target domain and thus lead to model degradation. The main reason is that their loss functions only act on the overlapped users without considering the non-overlapped users. The same phenomenon happens on other baseline models, e.g., DARec. (5) VDEA with deep latent embedding clustering can group users with similar behaviors and can further enhance the performance under the POCDR settings. We also observe that our proposed VDEA can also have great prediction improvement even when the overlapped user ratio $\mathcal{K}_{u}$ is much smaller (e.g., $\mathcal{K}_{u}=30\%$). Meanwhile our proposed VDEA still obtains better results when the source and target domains are different (e.g., Amazon Movie $\rightarrow$ Amazon Clothes) which indicates the high efficacy of VDEA.