Cross-Network Social User Embedding with Hybrid Differential Privacy Guarantees

Differential Privacy (Dwork et al., 2006) has emerged as a strong privacy definition for statistical data release with the intuition that a randomized algorithm behaves similarly on neighbouring datasets.

Noted that early DP is oriented toward structured data. For the protection of graph data, the notion of $\epsilon$-edge-DP is proposed.

Being a very strong privacy notion, $\epsilon$-DP, however, is unsuitable to protect text privacy with utility (Yue et al., 2021). Because under $\epsilon$-DP protection, a word will be transformed to any other words with equal probabilities, no matter how unrelated they are. Thus a sanitized token may not capture the semantics. The relaxed notion of Metric DP (MDP) can address this problem.

For MDP, the indistinguishability of output distributions is further controlled by the corresponding distance between the inputs, and the metric $d$ needs to be defined according to the specific application. Considering different characteristics and privacy expectations of heterogeneous social networks, we introduce a novel hybrid-DP notion which preserves various graph properties through carefully designing the injected noise.

Generally, each online social network can be represented as a heterogeneous graph. In particular, as shown in Figure 1, we map every single network to a heterogeneous network $G=(V,E,R)$ containing two types of nodes: (i) users $v_{u}$; (ii) textual posts $v_{t}$ written by users, and two types of edge relationships: (i) friendship $r_{f}$ (user-user) and (ii) writing $r_{w}$ (user-post). The user-type nodes have their respective multi-dimensional user attribute features $\mathbf{X}$.

The problem of DP-CroSUE is to obtain more comprehensive representations of social users by combining multiple HSNs together without raw data leakage. Formally, this problem can be divided as follows: 1) Heterogeneous social graph protection with hybrid differential privacy guarantees: Given a social network company’s data which can be represented as a heterogeneous network $G$, we need to design a hybrid randomized mechanism $M_{h}$: $M_{h}(G)\to\hat{G}$ by fully considering the different characteristics and privacy expectations of different data types. That is to say, for each data type, we should adopt the proper DP notion and set a proper privacy budget. 2) Cross-network information transferring: Given two protected social networks $\hat{G}_{1}$ and $\hat{G}_{2}$, to transfer knowledge across these two networks, a set of anchor users need to be found. Next, with those anchor pairs working as a ”bridge”, we need to design a transferring architecture which propagates information across and within these two networks effectively.

As mentioned in Section 3.2, we consider privacy protection of three types of sensitive information in heterogeneous social networks - user attribute features, user friendship relationships and posts written by users. They correspond to three data formats respectively: multidimensional numerical and categorical data, graph edge data and textual data. Given the different characteristics of different data formats and users’ heterogeneous privacy expectations, treating all the data types as equally sensitive will add too much unneeded noise and sacrifice utility. For example, the strict $\epsilon-DP$ notion applied in text data will totally change the original semantics and result in low utility. Besides, the contributions of a user’s attribute information, the numerous posts he/she has posted and the friendship relations to the final prediction tasks and information leakage differ a lot. Thus, a hybrid differential privacy mechanism which carefully designs the injected noise is highly needed. Specifically, we adopt $\epsilon$-DP, $\epsilon$-Edge-DP and MDP for attribute data, graph edge data and textual data respectively. Meanwhile, we also give insights into allocating proper privacy budgets.

We first introduce the three strategies for different data formats. To protect user attributes, we directly inject noise to the user attribute vector containing both numerical and categorical features. The extended PM algorithm proposed in (Zhang et al., 2021) is adopted. According to the proof in (Zhang et al., 2021), it satisfies $\epsilon$-DP. For the protection of user friendship relations, we extract the homogeneous user graph and enforce edge-DP to it. Specifically, we adopt TmF (Nguyen et al., 2015) algorithm, which first computes a new, noisy number of graph edges, then utilizes a filter to decide whether the original edge should be preserved or not. As proved in (Nguyen et al., 2015), TmF satisfies $\epsilon$-Edge-DP.

To protect text with utility, we develop a sanitation mechanism with a MDP guarantee. First, we inject each word in the corpus $C$ to an embedding. We denote the injection function $\phi$, which can be any of the well-known word embedding algorithms (e.g., Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), or FastText (Bojanowski et al., 2017)). Here we select Word2Vec(Mikolov et al., 2013). For any two words $x$ and $x^{\prime}$, we define their distance $d(x,x^{\prime})=d_{euc}(\phi(x),\phi(x^{\prime}))$, where $d_{euc}$ represents the Euclidean distance. To achieve MDP, for each word $x$, we run the randomized mechanism $M$ to sample a sanitized $y$ with probability:

Proof of Theorem 4.1. Consider a sentence $D$ only having one word ¡$x$¿, another sentence $D^{\prime}$ which is ¡$x^{\prime}$¿ ($x^{\prime}\not=x$), and a possible output $y$. We set $C_{x}=({\sum_{y^{\prime}\in C}exp(-\frac{1}{2}\epsilon\cdot d_{euc}(\phi(x),\phi(y^{\prime})))})^{-1}$.

We now analyze the overall differential privacy guarantee by combining all the above three information perturbations. We have the following theorem:

Proof of Theorem 4.2. We assume the three kinds of data in the heterogeneous graph $G$ are independent and denote the three perturbation mechanisms for user attributes, graph structure and user posts as $M_{a}$, $M_{g}$, $M_{t}$, respectively. Meanwhile, we utilize $G_{a}$, $G_{g}$ and $G_{t}$ to represent the attribute features, graph structure and posts alone. $G_{a}^{\prime}$, $G_{g}^{\prime}$ and $G_{t}^{\prime}$ are the corresponding neighbouring datasets for each data type. Noted that the hybrid-DP mechanism $M_{h}$ is the combination of the three randomized mechanisms mentioned above and $G^{\prime}$ denotes the whole perturbed heterogeneous social network. Since the attribute part, graph edge part and textual part satisfies $\epsilon$-DP, $\epsilon$-Edge-DP and MDP, respectively, we have:

Note that these three perturbation strategies are designed according to the characteristics of different data formats. Additionally, how to set privacy budgets ($\epsilon_{a}$, $\epsilon_{g}$ and $\epsilon_{t}$) to achieve proper privacy levels as well as utility for heterogeneous graph properties is also challenging. We solve this by referring to the Task-relevance to Message-inference Ratio (TMR). The intuition behind is that less noise being injected into the extracted data type which is more relevant to the target task will bring out more utility. Meanwhile, more noise should be injected into those data types causing accurate personal message inference to satisfy privacy. In this work, we leverage the precision score obtained through a single piece of information in the interest prediction task to measure task relevance and use the sum of precision scores in inference attacks as the message-inference value. Since larger privacy budget means less injected noise, we set larger $\epsilon$ for data with high TMR values. More details can be seen in Section 5.2.

We leverage embedding-based alignment methods to find anchor user linkages, which is divided into two procedures: 1) heterogeneous social network embedding learning and 2) unsupervised user linkage prediction.

To fully encode all kinds of information in the perturbed heterogeneous social network $\hat{G}=(V,\hat{E},R)$, we adopt relation-specific transformations and the propagation function is:

where $\mathbf{h}_{i}^{l+1}$ is the updated representation in the $(l+1)$-th layer. $\mathcal{N}_{i}^{r}$ represents the set of one-hop neighbor indices of node $i$ under relation $r\in R$ and $c_{(i,r)}=|\mathcal{N}_{i}^{r}|$. Suppose the final user embedding is $\mathbf{z}_{v_{u}}$ ($v_{u}\in\mathcal{V}_{u}$), the loss function can be expressed as:

where $v_{u}^{\prime}$ is the one-hop neighbours of $v_{u}$. $P_{n}(V_{u})$ defines negative sampling distribution of user nodes. $Q$ is the number of negative samples.

Next, we leverage the obtained social user embeddings $\mathbf{Z}_{u1}$ and $\mathbf{Z}_{u2}$ to make the alignment. As the spaces of $\mathbf{Z}_{u1}$ and $\mathbf{Z}_{u2}$ are learnt independently, we need to learn the matrix $\mathbf{W}$ such that $\mathbf{W}=\mathop{argmin}||\mathbf{WZ}_{u1}-\mathbf{Z}_{u2}||$ to reconcile them. Here the source network is $\hat{G}_{1}$ and the target one is $\hat{G}_{2}$. We get $\mathbf{W}$ by Generative Adversarial Networks (GANs), where the generator learns the transformation matrix $\mathbf{W}$, ensuring that the transformed $\mathbf{WZ}_{u1}$ approximates $\mathbf{Z}_{u2}$ as closely as possible. The discriminator tries to classify whether the embeddings are real $\mathbf{Z}_{u2}$ or those transformed ones. When $\mathbf{W}$ is trained well, we utilize the similarity measure CSLS proposed in (Lample et al., 2018) to find equivalent users.

After finding anchor users, we further integrate the partially aligned homogeneous user networks $\hat{G}_{u1}=(V_{u1},\hat{E}_{u1})$ and $\hat{G}_{u2}=(V_{u2},\hat{E}_{u2})$. To integrate them together, we propose a novel cross-network GCN embedding model, which is composed by an inter-graph propagation layer and a hierarchy intra-graph propagation layer.

$\bullet$ inter-graph propagation layer: We first update those aligned user nodes in both networks by fusing their information together. Considering the existing network disparity of $\hat{G}_{u1}$ and $\hat{G}_{u2}$, we leverage transformation matrices $\mathbf{W}_{12}$ and $\mathbf{W}_{21}$ to project embeddings into the other network space. For example, $\mathbf{W}_{12}$ is used to project users of $\hat{G}_{u1}$ into the space of $\hat{G}_{u2}$. Suppose two social users $v_{u1}^{anchor}$ and $v_{u2}^{anchor}$ are aligned, the representations of them after inter-graph propagation are:

where $\sigma$ denotes the activation function.

$\bullet$ hierarchy intra-graph propagation layer: We set the two user embeddings $Z_{u1}^{\prime}$ and $Z_{u2}^{\prime}$ with those aligned users getting updated through inter-graph propagation, while others are still the initial representations learned by GNN model. Now those anchor users’ representations contain knowledge from both networks. Considering the number of anchor users is limited, propagating the cross-graph information from those aligned users to the entire graph needs more hops. Considering the over-fitting and over-smoothing problems of GCN (Liu et al., 2020a), we leverage a hierarchy intra-graph propagation layer which emphasizes the role of anchor users and directly transfers information to nodes within $k$-hop range. Suppose $\mathbf{\hat{A}}_{u1}$ and $\mathbf{\hat{A}}_{u2}$ are the normalized adjacency matrices of the two user networks, and $\mathbf{D}_{u1}^{anchor}$ and $\mathbf{D}_{u2}^{anchor}$ are the diagonal matrices with the positions of anchors being set to one while others are zeroes, the propagation function is:

To show the difference between the original iterative propagation layer in GCN and our hierarchy one clearly, we depict their propagation processes in Figure 2. As shown in Figure 2(a), for the iterative architecture, it needs $k$ transformations to transfer the carrying knowledge in the anchor user to its $k$-hop neighbours. For example, the forward propagation process of iterative architecture in $\hat{G}_{u1}$ is formulated as $\mathbf{O}^{(u1)}=\sigma\left(\mathbf{\hat{A}}_{u1}\left(\ldots\left(\sigma\left(\mathbf{\hat{A}}_{u1}\mathbf{Z}_{u1}^{\prime}\mathbf{W}_{u1}^{(0)}\right)\ldots\right)\mathbf{W}_{u1}^{(k-1)}\right)\right.$. Obviously, when $k$ becomes large, there will be too many transformation parameters $\mathbf{W}_{u1}^{(i)}$ which will make the network difficult to train. Meanwhile, according to  (Liu et al., 2020a), if we apply iterative architecture to a connected graph, when $k$ goes to infinity, nodes become indistinguishable. However, as for the hierarchy architecture, each anchor user can be transferred to other users inside $k$-hops directly. It only needs one transformation thus the knowledge of anchor users will not be changed too much when they are transferred to distant users. Meanwhile, the emphasis of anchor users can further help transfer cross-network knowledge. By considering the information of multiple hops together, the hierarchy architecture has more capacities in capturing its local information compared to the iterative architecture, which helps make users distinguishable.

$\bullet$ Objective Function: We jointly train these two networks together to integrate them. The loss includes three parts: the graph-based losses $\mathcal{L}_{\hat{G}_{u1}}$ and $\mathcal{L}_{\hat{G}_{u2}}$ computed as per Eq. 9, and the hard alignment regularization which measures the anchor nodes’ representation distance in the two networks to regularize the output representation spaces. Overall, the total loss is:

We select three online social platforms: Foursquare, Twitter and Weibo, and make two partially-aligned dataset pairs. One is the Foursquare-Twitter pair from (Zhang et al., 2017). For the other, we crawled the Weibo platform and divided it into two sub-networks containing a part of sharing users. We manually labelled users from 10 possible interest categories according to their biographies and posts, including: (1) travel; (2) art; (3) health; (4) food; (5) technology; (6) sports; (7) business; (8) politics; (9) game; (10) fashion. Each user may be classified into multiple interests (multi-label classification). Table 1 shows the detailed statistical information about these datasets. The corresponding user attribute information extracted form the datasets are: (1) user screen name (We decompose the name into sub-string sets and convert them into numerical values using SimHash (Sadowski and Levin, 2007).), (2) number of followers, (3) number of followees, (4) number of posts, (5) gender, (6) occupation.

We compare DP-CroSUE with the following methods.

$\bullet$ Single-view methods: In general, single-view methods can be divided into text-based ones and structure-based ones. We select Word2Vec (Mikolov et al., 2013), SANTEXT (Yue et al., 2021), and DeepWalk (Perozzi et al., 2014) as single-view baselines. Among them, Word2Vec maps a sequence of social media posts into a vector representation. SANTEXT gets text representations under differential privacy. DeepWalk learns the latent representations of users in the homogeneous user network.

$\bullet$ Multi-view methods: For multi-view methods, we choose SNE (Liao et al., 2018), UDMF (Zhang et al., 2018) and R-GCN  (Schlichtkrull et al., 2018) as multi-view baselines. SNE learns user representations by preserving both structural proximity and attribute proximity. UDMF is a novel hybrid DNN-based framework that fuses information across different modalities. R-GCN aggregates information on HSNs based on different relations.

$\bullet$ Variants of DP-CroSUE: We utilize R-GCN and DP-R-GCN to observe the performances of single-network user embeddings obtained on the raw heterogeneous social networks and the perturbed ones, respectively. We also create a non-private model called CroSUE, which is a variant of DP-CroSUE deleting the Hybrid DP perturbation mechanism, to show the performance of the cross-network user embeddings without privacy protection.

In DP-CroSUE, to get the perturbed heterogeneous social graph, we set the privacy budget $\epsilon_{a}=5$ for user attribute feature perturbation, $\epsilon_{g}=10$ for user graph edge perturbation and $\epsilon_{t}=7.5$ for textual data perturbation. Our guidance is the TMR values as shown in Table 2. The TMR values of attribute information and textual information are obtained through the corresponding attribute features and sentence embeddings (i.e., averaging word embeddings (Mikolov et al., 2013)), respectively, while the TMR of friendship information is obtained through the features learnt by DeepWalk (Perozzi et al., 2014). To get initial user embeddings, we use a two-layer R-GCN. The first layer embedding dimension is 256 and the second embedding dimension is 128. For the cross-network GCN model, we set $k=4$ and $\alpha=2$ in the hierarchy intra-graph propagation layer. The final embedding dimension is 128.

The evaluation metrics for user interest prediction and attribute inference attack are as follows:

User Interest Prediction. After getting the social user embeddings, we utilize the decision tree classifier to conduct multi-label user interest classification. We randomly pick 80% of data for training, while the rest 20% is for evaluation. To be more convincing, we report the mean and standard deviation of the results after repeating experiments for 5 times. The specific metrics we leverage to judge the quality are precision and micro-$F1$ score.

Attribute Inference Attack. To evaluate all models’ robustness and capabilities in preserving sensitive user attribute information, we quantify the privacy leakage in social user embeddings through two inference attacks: gender inference and occupation inference. For gender inference, it is a binary classification problem. We use Logistic Regression as the attacker to make classification. For the occupation inference attack, we use the decision tree classifier to make classification. Similarly, we repeat the experiments for 5 times.

All models’ performances on predicting user interests are summarized in Table 3. DP-CroSUE outperforms all baselines and is only next to CroSUE, which is the same method using the unsanitized network data. Generally, single-view based embedding methods - DeepWalk, SANTEXT and Word2vec get the worst performances. Compared to single-view methods, SNE which incorporates structure and attributes together works better. It achieves maximum relative improvements of 14.1% in precision on Twitter compared to DeepWalk. Moreover, UDMF and R-GCN which incorporate more kinds of information perform even better than SNE on most datasets. For example, compared to SNE, R-GCN gets an improvement of 13.4% in precision on Sub-Weibo1. These observations demonstrate that incorporating more user information can help predict user interests. Compared with R-GCN, DP-R-GCN gets worse precision and Micro-$f1$ score. This is reasonable because DP-R-GCN extracts user information from the perturbed heterogeneous social network. These methods mentioned above are all single-network methods. Obviously, the best single-network method is R-GCN applied in real social networks. As observed, DP-CroSUE consistently outperforms R-GCN on all the datasets. For example, the precision of DP-CroSUE is 2.0% higher than R-GCN on Sub-Weibo1 dataset. This indicates that DP-CroSUE, though operating on perturbed networks for the purpose of information protection, can achieve satisfactory results. In addition, CroSUE achieves the best performance. This further confirms the effectiveness of fusing networks together. However, since CroSUE works directly on real social networks, it faces user information leakage problems.

Table 4 shows the results of the attribute inference attack models depicted in Section 5.3 on all the datasets. Note that lower scores show higher resistance to inference attacks. As shown in Table 4, all baselines operating on real social networks except for DeepWalk achieve high precision and Micro-$f1$ score in attribute inference attacks. For example, the precision scores of DeepWalk on all datasets are nearly 50.0%. That means DeepWalk has no ability to infer user’s gender. This is predictable as DeepWalk only extracts network structure information but does not involve user personal attribute features at all. It is also worth noting that DP-CroSUE can achieve the second or third best results on almost all datasets. Given that DeepWalk does not have a chance to learn user attribute features during training, the result that DP-CroSUE is only worse than DeepWalk highly validates the capability of DP-CroSUE in private attribute protection. Meanwhile, the better performance of DP-R-GCN compared to R-GCN also indicates the effectiveness of our hybrid DP mechanism. For example, compared to R-GCN, the precision of DP-R-GCN drops significantly on Foursquare dataset in both gender inference and occupation inference (22.3% and 2.6% reduction). In addition, we noticed that DP-CroSUE, which incorporates information from two perturbed networks, performs even better in resisting attribute inference attacks than DP-R-GCN on most datasets. The reason is that if a user’s attribute is disturbed into fake in one network, but remains true in another, linking these two nodes may result in both being inferred to false. This further shows the superiority of DP-CroSUE in privacy protection.

To investigate how does each part of the perturbation paradigm of DP-CroSUE affect its performance on user interest prediction and privacy protection, we independently vary the privacy budget $\epsilon_{a}$, $\epsilon_{g}$ and $\epsilon_{t}$, and report the new results in Figure 3.

$\bullet$ Impact of user attribute perturbation privacy budget $\epsilon_{a}$. We vary the privacy budget $\epsilon_{a}\in\{1,5,10,15\}$. The results are shown in Figure 3(a). In general, for all the datasets, the results of user interest prediction get improvement with the increase of $\epsilon_{a}$. Because larger $\epsilon_{a}$ means less privacy thus more accurate user attributes, which helps predict user interests. For example, if we know a user is a woman, she may be more interested in fashion compared to a man. Meanwhile, for all the $\epsilon_{a}$ we selected, DP-CroSUE consistently outperforms R-GCN in terms of user interest prediction. This further confirms the effectiveness of our model. As for the privacy protection capabilities, larger $\epsilon_{a}$, negatively influences the privacy protection results. Because more accurate user attributes are encoded into the final user representations. However, the gender and occupation information still can be effectively protected even when $\epsilon_{a}$ is set to a large value. From Figure 3(a), even when $\epsilon_{a}$ is set to 15, the precision scores of attribute inference attacks are lower than R-GCN. Our analysis is that by aligning and aggregating two perturbed datasets together, user’s gender and occupation information may be further masked. Since if a user’s gender or occupation is perturbed into fake in one network with the user attribute perturbation algorithm, his/her attribute in the other network may also be influenced. All the above results validate the superiority of DP-CroSUE in privacy protection.

$\bullet$ Impact of user edge perturbation privacy budget $\epsilon_{g}$. We choose the value of $\epsilon_{g}$ from $\{1,5,10,15\}$ and plot the results in Figure 3(b). Similarly, larger $\epsilon_{g}$ results in better user interest prediction performance. However, the prediction precision degrades significantly when the privacy budget is small. For example, when $\epsilon_{g}=1$, the precision is even lower than R-GCN. As we know, the nature of GCN is smoothing - making connected nodes similar. Real social networks have the homophily property. People tend to follow those who have similar interests with them. If we inject a large amount of noise into the network structure, the homophily level of the graph will be impacted. Thus user prediction results will decrease while the user information can be better protected. In this way, we should choose a proper $\epsilon_{g}$ to achieve a good trade-off between privacy protection and recommendation accuracy.

$\bullet$ Impact of user text perturbation privacy budget $\epsilon_{t}$. We study the impact of privacy budget in text perturbation with $\epsilon_{t}\in\{0.1,2.5,5,7.5\}$. As we can see from Figure 3(c), in general, larger $\epsilon_{t}$ brings higher user interest prediction precision. Meanwhile, we find that the improvement is relatively slow with the increase of $\epsilon_{t}$. The reason behind is the high utility property of the relaxed MDP notion applied in the textual data perturbation algorithm. Even when privacy budget is comparably small, with the computed substitution words, the original semantics may still be kept. Thus the user interest prediction performance can be guaranteed. What’s more, it is worth noting that the perturbation of user text effectively preserves user attribute information compared to R-GCN. Since users may declare their gender and occupation in the original posts, substituting the original words can prevent information leakage effectively. For example, the word ”girl” may be substituted by the word ”boy”.

To sum up, to make a good trade-off of interest prediction tasks and privacy protection, we recommend a comparably small attribute privacy budget, and comparably large edge and text privacy budgets. A good budget setting can be: $\epsilon_{a}=5,\epsilon_{g}=10$ and $\epsilon_{t}=7.5$. The total privacy budget is $\epsilon_{a}+\epsilon_{g}+\epsilon_{t}\cdot d_{euc}=15+7.5\cdot d_{euc}$.

To study the performance of the novel cross-network GCN embedding model in network integration and user interests prediction, we evaluate different degraded model versions and record the precision scores in Table 5. For convenience, we denote the original cross-network GCN model introduced in Section 4.3 as CroGCN. We create CroGCN-NH by replacing the hierarchy intra-graph propagation layer with iterative propagation layers like standard GCN model. Besides, we also remove the hard alignment regularization and denote the new version as CroGCN-NA. As shown in Table 5, on all the datasets, CroGCN gets highest user interest prediction scores. The better performance of CroGCN compared with CroGCN-NH demonstrates the superiority of the hierarchy intra-graph propagation layer in transferring knowledge to the whole network. Meanwhile, the worse performance of CroGCN-NA also proves the importance of the hard alignment regularization.