On the Reliability of Profile Matching
Across Large Online Social Networks

In this paper, our goal is to investigate the reliability of techniques for matching profiles across large real-world online social networks, such as Facebook and Twitter, using only publicly available profile attributes, such as names, usernames, location, photos, and friends. Reliability refers to the extent to which different profiles belonging to the same user can be matched across social networks, while avoiding mistakenly matching profiles belonging to different users. Matching schemes need to be highly reliable because incorrectly matched profiles communicate an inaccurate portrait of a user and could have seriously negative consequences for the user in many application scenarios. For example, Spokeo has been recently sued over providing inaccurate information about a person which caused “actual harm” to the person employment prospects [4]. We focus on publicly available profile attributes because data aggregators today can crawl and leverage such information for matching profiles.

Recently, a number of schemes have been proposed for matching profiles across different social networks  [22, 28, 34, 12, 21, 14] (we review them in §9.) The potential of these schemes to reliably match profiles in practice, however, has not been systematically studied. Specifically, it is not clear how or what properties of profile attributes affect the reliability of the matching schemes. Furthermore, the training and testing datasets for evaluating the matching schemes are often opportunistically generated and they constitute only a small subset of all user profiles in social networks. It is unclear whether the reliability results obtained over such datasets would hold over all user profiles in real-world social networks, where there are orders of magnitude more non-matching profiles than matching profiles (i.e., there is a huge class imbalance).

Our first contribution lies in defining a set of properties for profile attributes–Availability, Consistency, non-Impersonability, and Discriminability (ACID)–that are both necessary and sufficient to determine the reliability of a matching scheme (§3). Analyzing the ACID properties of profile attributes reveals the significant challenges associated with matching profiles reliably in practice (§4). First, data in real-world social networks is often noisy – users do not consistently provide the same information across different sites. Second, with hundreds of millions of profiles, there is a non-trivial chance that there exist multiple profiles with very similar attributes (e.g., same name, same location) leading to false matches. Finally, attackers create profiles attempting to impersonate other users, fundamentally limiting the reliability of any profile matching scheme.

Another key contribution lies in our method for carefully selecting the training and testing datasets for matching profiles (§5). When we evaluate the main types of matching schemes in the literature (based on binary classifiers) using a small random sample of Twitter and Facebook profiles (similar to how these schemes were evaluated originally), the schemes achieve over 90% recall and 95% precision (§2). Unfortunately, when we evaluate these schemes over datasets sampled carefully to preserve the reliability that the schemes would have achieved over the larger datasets (full Facebook database), their performance drops significantly (§6.2). We could obtain only a 19% recall for a 95% precision.

We then investigate if we could improve the reliability of matching schemes in scenarios where we know that there is at most one matching profile (see §7). In such scenarios, we propose a new matching scheme and show that it is indeed possible to improve the recall to 29% at 95% precision. This is still considerably lower than the high recall (90%) reported in the literature.

Thus, we discover a fundamental limitation in matching profiles across existing social networks using public attributes. To further confirm the inherent limits of reliably matching profiles in practice, we compare the reliability of automated matching schemes with that of human Amazon Mechanical Turk (AMT) workers. Under similar conditions, AMT workers are able to match only 40% of the profiles with a 95% precision. Our analysis is the first to highlight that achieving high reliability in matching profiles across large real-world social networks comes at a significant cost (in terms of reduced recall).

Note that we address here the problem of matching individual profiles, which is different from the problem of matching two entire social networks or databases. The difference is that we do not assume that we have access to all the data in $SN_{1}$ but only to one profile. For example, we cannot match profiles by exploiting patterns in the graph structure of $SN_{1}$ and $SN_{2}$, and we cannot optimize the matching of a profile in $SN_{1}$ based on the matchings of other profiles in $SN_{1}$. Thus, we cannot take advantage of some methods proposed for de-anonymizing social graphs [23, 15] and entity matching [9].

Our problem formulation is motivated by practical scenarios. There are many people search engines such as Spokeo that allow users to search for data about a particular person. These services gather data about a person by matching the profiles a person has on multiple social networks.

We are particularly interested in two instantiations of the problem that are motivated by practical scenarios: (1) the generic case – a profile can have multiple matching profiles in $SN_{2}$; and (2) the special case – a profile can have at most one matching profile. This case is suited for matching social networks such as Facebook or LinkedIn that enforce users to have only one profile.

This solution works well for the generic case of our matching problem. Given a profile $a^{1}$, we can use the binary classifier to check, for every pair of profiles $(a^{1},a^{2})$ such that $a^{2}\in SN_{2}$, whether it is matching or not. We can then output any profile $a^{2}$ that the binary classifier declares as matching. In this paper, we test such approach when we represent $(a^{1},a^{2})$ with five features, each corresponding to the similarity score between $a^{1}$ and $a^{2}$ for each of the five profile attributes: real name, screen name, location, photo, and friends.

For the special case of our matching problem, the previous approach is vulnerable to output many false matches. For this case, instead of independently judging whether each pair $(a^{1},a^{2})$ is a match or not, we can compare (for a given $a^{1}$) the probabilities $p$ for all pairs $(a^{1},a^{2})$ to judge which profile is most likely the matching profile of $a^{1}$. We discuss this case in more detail in §7.

Many previous studies used the true and the false positive rate to evaluate their matching schemes. The true positive rate is the percentage of matching profiles that are identified, while the false positive rate is the percentage of non-matching profiles that are false matches. The goal is to have a high true positive rate and a low false positive rate. These metrics are, however, a misleading indicator of the reliability of a matching scheme because they are not suited for scenarios with high class imbalance, i.e., the number of matching profiles is much lower than the number of non-matching profiles. For example, a matching scheme with a 90% true positive rate for a 1% false positive rate might seem reliable, however, if we use it in a scenario where we have 1,000 matching and 999,000 non-matching profiles, the matching scheme would output 900 true matches and 9,990 false matches, which is clearly unreliable. In real-world social networks, the class imbalance is even higher (e.g., for each matching profile we have over 1 billion non-matching profiles in Facebook) thus the scheme would output even more false matches.

This paper argues that better metrics to evaluate the reliability of a matching scheme are the precision and recall. The recall is the percentage of matching profiles that are identified, while the precision the percentage of all pairs returned by the matching scheme which are true matches. The goal is to have a high recall and a high precision. In the previous example, we would have 90% recall for a 8% precision, which shows the low reliability of the scheme (out of all matched profiles only 8% are true matches). Thus, the best way to show the reliability of a matching scheme is to evaluate its precision and recall with realistic class imbalance. In the rest of the paper, by reliable we mean a precision higher than 95%.

The ACID properties are clear and intuitive properties that help understand the potential of an attribute to perform reliable matching, as the following theorem formalizes. ¹¹1The proof can be found in the Appendix A.

In Theorem 1, the threshold parameter $th$ must be the same as the one in the definitions of $C$, $nI$ and $D$. Theorem 1-(i) shows that the classifier’s recall is simply the product of consistency and availability. Theorem 1-(ii) gives a simple upper bound of the precision as a function of the effective discriminability (which itself is a function of the discriminability and of the impersonability, see above). This upper bound gives a good order of magnitude for the precision; moreover, for high precision (which is what we aim), given the small number of false positives, the true precision should be close to the bound. Finally, Theorem 1-(iii) confirms that a high value of all four ACID properties is necessary and sufficient to obtain high precision and recall.

Properties $A$, $C$ and $nI$ are independent of the network scale, however, the discriminability very largely depends on the network scale since having more non-matching pairs decreases the probability that none of them has a high similarity score. This implies that we must estimate the precision and the recall of a matching scheme using datasets that accurately capture the ACID properties of profile attributes of the entire social network. Otherwise, the precision and the recall will be incorrect.

In practice different attributes satisfy the properties to different extent and the challenge is to combine different attributes with imperfect properties to achieve a reliable matching. The next section analyzes the ground truth for several large social networks to understand the limits of matching profiles across different sites.

Most previous studies sampled datasets by picking matching and non-matching profiles at random. Such random sampling fails to capture the precision and recall of matching schemes in practice because it severely over-estimates the discriminability of attributes found in the original social network (as seen in §4.4) and therefore it severely over-estimates precision. To estimate well the reliability of a matching scheme in practice, the sampled dataset needs to preserve the precision and recall of the original social network at least for high values of precision. The key to ensure this is to sample all potential false matches, i.e., all profiles that could be mistakenly matched by the matching scheme. Thus, we build two datasets: (1) a reliability non-preserving sampled dataset for comparison with previous techniques (as this is the standard evaluation method); and (2) a reliability preserving sampled dataset that strives to capture all possible false matches in a social network to better estimate the reliability of matching schemes in practice.

We generate a reliability preserving sampled dataset for matching Twitter and Facebook. Although building such dataset for other social networks is possible, the process is strenuous. Instead, we take two of the most popular social networks to show the limitations for matching profiles across real-world social networks.

Given that our analysis in §4.3 shows that most Twitter-Facebook matching profiles have consistent real names or screen names, we hope to build a comprehensive $C(a^{1})$ by exploiting the Facebook search API, which allows searching for people by name. For each Twitter profile, $a^{1}$ (we sample the same Twitter profiles from Random-Sampled), we generate $C(a^{1})$ using the Facebook search API to find profiles with the same or similar real name or screen name as $a^{1}$. The resulting dataset, which we call Emulated-Large, contains over 270,000 combinations of profiles $(a^{1},a^{2})$ where $a^{2}\in C(a^{1})$. Thus, for each Twitter profile the dataset contains in average 320 Facebook profiles with similar names.

Our analysis shows that the matching profile of $a^{1}$ is in $C(a^{1})$ (i.e., $\hat{a}^{2}\in C(a^{1})$) for 70% of Twitter profiles. This implies that for 70% of cases we selected at least all non-matching profiles with higher name similarity than the matching profile. Additionally, the median similarity of the least similar real name in $C(a^{1})$ is 0.5, while the median similarity of matching profiles is 0.97. This means that we also catch many Facebook profiles with lower name similarity than the matching profiles. Thus, the only possible false matches that we miss are the ones that have very different names.

Note that the reliability preserving sampling does not sample the matching profile when there is little chance for it to match (in 30% of the cases). We actually tried to train and test matching schemes with or without including the unsampled matching profiles in the Emulated-Large and the reliability did not differ significantly.

Our sampling strategy ensures that the discriminability and impersonability of real names and screen names found in the real-world datasets are preserved. It might over-estimate, however, the discriminability of location, friends, and profile photos since we do not sample in $C(a^{1})$ profiles with similar location, friends or photos if they do not also have similar names or screen names. Evaluating matching schemes over $C(a^{1})$ rather than all $SN_{2}$ could lead to an under-estimation of the false matches. Thus, the precision we obtain over this dataset is an upper bound on the precision in practice. This implies that the limits of reliably matching schemes in practice can only be worse than what we show in this paper. We believe, however, that our sampling strategy gives a very good idea of the precision and recall in the real-world datasets because there will be very few false matches (if any) with very dissimilar names even if they have similar location, photo or friends.

Another limitation of the dataset is that it does not contain cases where a profile $a^{1}$ has multiple matching profiles in $SN_{2}$. This is a consequence of our method to gather ground truth (§4.1) that only gives a single matching profile in $SN_{2}$ for each $a^{1}$. The implications of this limitation on our evaluation is that there might be some matching profiles that we consider as false matches whereas they are not. Since Facebook enforces the policy that users should only have one profile, we believe there are not many such cases and the reliability we measure is likely close to the real-world reliability.

In practice, there are Twitter users that do not have a matching Facebook profile, but our datasets do not contain such cases. To evaluate how matching schemes perform in such scenarios, we test in §7 the reliability of matching schemes when we remove the matching profile from Emulated-Large.

We first emulate the methodology employed by previous works: we train and test matching schemes with Random-Sampled, using all attributes. Since some profile attributes have a high discriminability in the dataset, it is straightforward to build a matching scheme with high reliability. On top of this, there is little difference between the reliability of naive classification techniques and more sophisticated ones.

We then investigating the reliability of matching schemes in practice by testing them over Emulated-Large. As expected, the precision of the previously built matching schemes drastically decreases to a point that makes them unusable. Thereafter, we investigate the reasons behind such poor reliability and we evaluate different strategies to increase the precision and recall in practice. The resulting schemes are able to achieve a good precision, but the recall is still low. These results show the inherent difficulty of matching profiles reliably in today’s large social networks.

Intuitively, there are two possible scenarios where the topmatch is a false match: The first is if an attacker creates an impersonating profile on $SN_{2}$ that is more similar than the true matching profile. It might be possible to detect these cases as both $p_{1st}$ and $p_{2nd}$ will be high and $(p_{1st}-p_{2nd})$ will be very small. The second is when the true matching profile $\hat{a}^{2}$ is in $C(a^{1})$ but a non-matching profile $a^{2}\in C(a^{1})$ is chosen as output because the classifier assigns it a higher probability $p$ of being the matching profile (due to the lack of attribute availability and/or consistency). Another case is when $\hat{a}^{2}$ does not exist, forcing the scheme to choose the non-matching profile that is the most similar to $a^{1}$ as the output. We might detect these cases as $p_{1st}$ and $p_{2nd}$ will not be very high (none of the profiles in $C(a^{1})$ are very similar to $a^{1}$) and $(p_{1st}-p_{2nd})$ will be again very small (none of the profiles in $C(a^{1})$ is much more similar than the rest).

To incorporate the above logic, we design the Guard which is a binary classifier that takes as input $p_{1st}$ and $p_{2nd}$ and outputs the probability that the topmatch is the matching profile. Figure 2d shows that the Guard increases the recall of the matching scheme to 29% for a 95% precision. Although 29% recall is a big improvement over the recall previously obtained, the recall is still low. This shows that in practice, it is hard to achieve a high recall if we want to have a high precision.

The matching schemes in §6 decide independently for each pair ($a^{1},a^{2}$) where $a^{2}\in C(a^{1})$ whether it is a match or not. In contrast, the strength of the Guard is that it exploits the structure of $C(a^{1})$ for a given $a^{1}$. In particular, since $C(a^{1})$ depends on $a^{1}$, for a given probability $p$ to be the matching profile of $a^{1}$, the topmatch profile $a^{2}$ will be declared a match for some $a^{1}$ if its attribute values are sufficiently unique, whereas the scheme will return nothing for other $a^{1}$ if the attribute values are too common (e.g., Jennifer Clark that lives in New York). This reduces considerably the false matches and, as we have shown, increases a lot the matching recall for a given precision.

For this we designed an AMT experiment. We randomly select 200 Twitter-Facebook matching profiles from Dataset FF (that are not used for training the matching schemes). In each assignment, we give AMT workers a link to a Twitter profile as well as links to the 10 most similar Facebook profiles (we shuffle their position) and we ask AMT workers to choose the matching profile. We allow workers to choose that they are unable to identify the matching profile. For each assignment we ask the opinion of three different workers. We present the results for majority agreement (two out of three workers decided on the same answer). We design two versions of the experiment: in the first one if the matching profile is not in $C(a^{1})$, the matching profile will not be in the list of 10 Facebook profiles; and a second version, where we always put the matching profile the list of 10 Facebook profiles.

In the first version of the experiment, AMT workers were able to match 40% of the Twitter profiles to their matching profiles and 4% are matched to the wrong Facebook profile. This means that AMT workers achieve a 40% recall for a 96% precision, which is better than the Guard, but far from a 100% recall. In the second version of the experiment, AMT workers were able to match 58% of Twitter profiles. Thus, even humans cannot achieve a recall close to 100% to match profiles in practice.

The closest to our work are a number of schemes that leverage information in the profiles of users similar to the attributes we use in this paper [22, 28, 20, 25, 5, 34, 33, 29, 24, 27, 18, 36, 35, 34, 17]. Most schemes work by training classifiers to distinguish between matching and non-matching profiles. We simulated these approaches in §6 and we saw that, because they did not consider the problems that come with matching in practice, the matching schemes are very unreliable when evaluated in real-world social networks. A few studies attempted to perform profile matching in practice [20, 25, 5]. These studies, however, just pointed out that profile matching in practice yields a large number of false matches. In contrast, we conduct a systematic analysis of the causes of such false matches and possible ways to eliminate them.

Other schemes use attributes extracted from user activities (i.e., the content users generate instead of attributes of the profile) [12, 21, 14]. These schemes reveal how even innocuous activities of users can help identify a user across social networks. However, these schemes explore attributes with either low availability or low discriminability, which makes them hard to use in practice without sacrificing reliability.

To understand how profile attributes used by matching schemes affect the overall matching reliability, we proposed a framework that consist of four properties – Availability, Consistency, Impersonability, and Discriminability (ACID). Our analysis showed that most people maintain the same persona across different social networks – thus it is possible to match the profiles of many users, however, in practice there can be a non negligible number of profiles that belong to different users but have similar attribute values, which leads to false matches.

We showed that the reliability of matching schemes that are trained and tested on reliability non-preserving sampled datasets is not indicative of their reliability in practice. In fact, traditional matching schemes based on binary classifiers can only achieve a 19% recall for a 95% precision to match Twitter to Facebook profiles in practice. To avoid these pitfalls we illustrated the right assumptions we can make about the matching problem and the correct methodology to evaluate matching schemes in realistic scenarios.

Finally, we proposed a matching scheme that is able to mitigate impersonation attacks and reduce the number of false matches to achieve a 29% recall for a 95% precision. Our matching scheme exploits a special case of the matching problem, namely that there exists at most one matching profile. Although we cannot claim that 29% is a high recall, humans cannot do much better (they only detect 40% of matching profiles).