Exploring Linkablility of Community Reviewing

For each account $IR$, we built an NB model, $P(token_{i}|IR)$, from its identified record. Probabilities are computed using the Maximum-Likelihood estimator bishopbook and Laplace smoothing mlearn1 as shown in 2. We then construct four models corresponding to the four aforementioned token types. That is, for each $IR$, we have $P_{unigram}$, $P_{digram}$, $P_{category}$ and $P_{rating}$.

To link an anonymous record $AR$ to an account $IR$ with respect to token type $R$, we first extract all $R$-type tokens from $AR$, $T_{R_{1}},T_{R_{2}},....T_{R_{n}}$ (Where $T_{R_{i}}$ is the $i$-th $R$ token in $AR$). Then, for each $IR$, we compute the probability $P_{R}(IR|T_{R_{1}},T_{R_{2}},....T_{R_{n}})$. Finally, we return a list of accounts sorted in decreasing order of probabilities. The top entry represents the most probable match.

We use symmetric KLD (see Section 2) to compute the distance between anonymous and identified records. To do so, we first compute distributions of all records, as follows:

To avoid division by 0, we smooth distributions via Laplace smoothing mlearn1 , as follows:

As before, we compute four distributions. To link $AR$ with respect to token type $R$, we compute $SymD_{kl}$ between the distribution of $R$ for $AR$ and the distribution of $R$ for each $IR$. Then, we return a list sorted in ascending order of $SymD_{KLD}(IR,AR)$ values. The first entry represents the account with the most likely match.

Figure 2 shows Top-1, Top-10, and Top-50 plots of the linkability ratios (LRs) for NB and KLD models for several anonymous record sizes when either rating or category is used as the token. Not surprisingly, an increase in the anonymous record size causes an increase in the LR. Figures 2 and 2 show LRs when rating token alone is used. In the Top-1 plot, LRs are low and the highest ratio is 2.5%/1.9% in NB/KLD for an anonymous record size of 60. However, in Top-10 and Top-50 plots, LRs become higher and reach 13.9%(11.9%) and 34.8%(33.3%) in Top-10 and Top-50 plots, respectively, in NB(KLD) for the same anonymous record size. Figures 2 and 2 show LRs for the category token. In Top-1, the highest LR is 13.6%/4.7% in NB/KLD for anonymous record size of 60. A significant increase occurs in LRs in Top-10 and Top-50 plots: 40.4%(17.4%) and 68.2%(38.3%), respectively, in NB(KLD) model. The category is clearly more effective than the rating token. Additionally, we observe that NB performs better than KLD model, especially, for the category token.

We conclude that rating- and category-based models are only somewhat helpful, yet insufficient to link accounts for many anonymous records. However, it turns out that they are quite useful when combined with other lexical tokens, as discussed in Section 5.3 below.

Figures 3 and 3 depict LRs (Top-1 and Top-10) for NB and KLD with the unigram token. As expected, with the increase in the anonymous record size, the LR grows: it is high in both Top-1 and Top-10 plots. For example, in Top-1 of both figures, the LRs are around: 19%, 59% and 83% for anonymous record sizes of 10, 30 and 60, respectively. Whereas, in Top-10 of both figures, the LRs are around: 45.5%, 83% and 96% for same record sizes. This suggests that reviews are highly linkable based on trivial single-letter distributions. Note that the two models exhibit similar performance.

Figures 3 and 3 consider the digram token. In both models, the LR is impressively high: it gets as high as 99.6%/99.2% in Top-1 for NB/KLD for an AR size of 60. For example, the Top-1 LRs in NB are: 11.7%, 62.9%, 87.5% and 97.1%, for respective AR sizes of 1, 5, 10 and 20. Whereas, in KLD, the Top-1 LRs for record sizes of 10, 30 and 60 are: 1.9%,74.9% and 99.2%, respectively.

Unlike unigrams – where LRs in both models are comparable – KLD in digram starts with LRs considerably lower than those of NB. However, the situation changes when the record size reaches 50, with KLD performing comparable to NB. One reason for that could be that KLD improves when the distribution of ARs is more similar to that of corresponding identified records; this usually occurs for large record sizes, as there are more tokens.

Not surprisingly, in both lexical and non-lexical models, larger AR sizes entail higher LRs. With NB, a larger record size implies that, a given AR has more tokens in common with the corresponding IR. Thus, an increase in the prediction probability $P(IR|T_{1},T_{2},...T_{n})$. For KLD, a larger record size causes the distribution derived from the AR to be more similar to the one derived from the corresponding IR.

This is straightforward in the NB. We simply increase the list of tokens in the unigram- or digram-based NB by adding the non-lexical tokens. Thus, for every AR, we have $P(lexical\_token_{i}|IR)$, $P(category\_token_{i}|IR)$ and $P(rate\_token_{i}|IR)$.

Combining non-lexical with lexical tokens in KLD is less clear. One way is to simply average $SymD_{KLD}$ values for both token types. However, this might degrade the performance, since lexical distributions convey much more information than their non-lexical counterparts. Thus, giving them the same weight would not yield better results. Instead, we combine them using a weighted average. First, we compute the weighted average of rating and category $SymD_{KLD}$:

Then, we combine the above with $SymD_{KLD}$ of the lexical tokens to compute the final weighted average:

Thus, our goal is to get the right $\beta$ and $\alpha$ values. Intuitively, $SymD_{kl\_lexical}$ should have more weight as it carries more information. Since there is no clear way of assigning weight values, we experiment with several choices and pick the one with the best performance; we discuss the selection process below. We experiment only within the IR set and then verify the results generalizes to the AR. This is done as follows:

First, for every IR, we allocate the last 30 reviews as a testing record and the remainder – as a training record. Then, we experiment with $SymD_{KLD\_r\_c}$ using several $\beta$ values and set $\beta$ to the value that yields the highest LR based on the tested records. Then, we experiment with $SymD_{KLD\_l\_r\_c}$ using several $\alpha$ values and, similarly, pick the one with the highest LR.

Since $\beta$ or $\alpha$ could assume any values, we need to restrict their choices. For $\beta$, we postulate that its optimal value is close to $0.5$ since LRs for rating and category are comparable. Thus, we experiment with a range of values, from $0$ to $1.00$ in $0.1$ increments. For $\alpha$, we expect the optimal value to exceed $0.9$, since the LR for lexical tokens is significantly higher than for non-lexical ones. Therefore, we experiment with the weighted average by varying $\alpha$ between $0.9$ and $0.99$ in $0.01$ increments.

If the values exhibit an increasing trend (i.e., $SymD_{KLD\_l\_r\_c}$ at $\alpha$ of 0.99 is the largest in this range) we continue experimenting in the $0.99--1.00$ range in $0.001$ increments. Otherwise, we stop. For further verification, we also experiment with smaller $\alpha$ values: $0.0,0.3,0.5,0.7,\mbox{and}\ 0.8$, all of which yield LRs significantly lower than 0.9 for both the unigram and digram. We acknowledge that we may be missing $\alpha$ or $\beta$ values that could further optimize $SymD_{KLD\_l\_r\_c}$. However, results in sections 5.3.2 and 5.3.3, show that our selection yields good results.

Figure 4 shows LRs (Top-1) for $\beta$ values. The LR gradually increases until it tops off at $3.4\%$ with $\beta=0.5$ and then it gradually decreases. Figure 4 shows LRs (Top-1) for $\alpha$ values in the unigram case. The LR has an increasing trend until it reaches $67.8\%$ with $\alpha=0.997$ and then it decreases. Figure 4 shows LRs (Top-1) for $\alpha$ values in the diagram case where it tops off at $75.9\%$ with $\alpha=0.97$. Thus, the final values are $0.5$ for $\beta$ and $0.997$/$0.97$ for $alpha$ in the unigram/digram case. Even though we extract $\alpha$ and $\beta$ values by testing on a record size of 30, the results in following sections show that the derived weights are effective when tested on ARs of other sizes.

Figures 5 and 5 show Top-1 plots for NB and KLD models when rating and category tokens are combined or used separately. Clearly, combining the tokens significantly improves LRs in several record sizes. In NB, the gain in Top-1 LRs ranges from 2.5-15.9%/3.5-27.1% over the category/rating based model for most record sizes. For example, LRs increase from 5.8(1.4)% and 13.6(2.5)% in category(rating) based model to 14.5% and 29.5% in NB combined model for record sizes of 30 and 60, respectively. In Top-50, the LR could reach as high as 87.7%, versus 68.2(34.8)% in the category (rating) based model for a record size of 60.

In KLD, the gain in Top-1 LRs ranges from 1.7-9.9%/1.6-12.7% over category/rating based model for most record sizes. For example, it leaps from 1.1(1.3)% and 4.7(1.9)% in category (rating) based model to 3.7% and 14.6% in KLD combined model for record sizes of 30 and 60, respectively. The gain is even higher in Top-50 where it reaches 69.1%, versus 38.3(33.3)% in the category (rating) based model for a record size of 60. These results show that combining rating and category tokens is very effective in increasing LRs in both NB and KLD models.

Figures 6 and 6 show Top-1 and Top-10 plots in NB and KLD models of unigram tokens before and after combining them with rating and category tokens. Adding non-lexical tokens to unigrams substantially increases LRs in several record sizes. In NB, the gain in Top-1 LRs ranges from 0.25-18.9% (1.4 - 15.7% for Top-10 LRs). In KLD, the gain in Top-1 LRs ranges from 2.5-11.9 (2-7.8% in Top-10 LRs) for most record sizes. These findings shows how effective is combining the non-lexical tokens with the unigrams. In fact, we can accurately identify almost all ARs.

Figures 6 and 6 show the effect of adding ratings and categories to digrams. The overall effect is minuscule: in NB (KLD) model, the increase in Top-1 LRs ranges from 0.3-1.8% (0.2-2.7%) for most record sizes. The increase is very similar in Top-10 plots.

In previous sections, our analysis was based on using the full data set. That is, except for the anonymous part of the data set, we use all of the user reviews as part of our identified set. Although the LR is high in many cases, it is not clear how the models will perform when we restrict the IR size. To this end, we re-evaluate the models with the same problem settings, however, with a restricted IR size. We restrict the IR size to the AR size; both randomly selected without replacement.

Figures 7 and 7 show two Top-1 plots in NB and KLD models: one plot corresponds to the restricted identified set and the other – to the full set. Tokens used in the models consist of digrams, ratings and categories (since this combination gives the highest LR). Unlike the previous sections, where NB and KLD behaved similarly, the two models now behave differently when restricting the identified set. While NB performs better than KLD on the full set, the latter performs much better than NB when the identified set is restricted. In fact, in some cases, KLD performs better when the set is restricted.

The reason for this improved KLD performance might be the following: in the symmetric KLD distance function, the distributions of both the IR and AR have to be very close in order to match regardless of the size of the IR; unlike the NB, where larger training sets would lead to better estimates of the token probabilities and thus more accurate predictions.

In KLD, we achieve high LRs for many record sizes. For example, Top-1 LRs in the restricted set are 74.5%, 88% and 97.1% when the anonymous (and identified) record sizes are 30, 40 and 60, respectively. Whereas, the LRs in the full set for the same AR sizes are: 76.5% , 93% and 99.4%. When the record size is less than 30, KLD performs better in the restricted set than the full one. For example, when the AR size is 20, the LR in the restricted set is 50.1% and 34.3% in the full set. In NB, Top-1 LR in the restricted set is lower than the full set. For instance, it is 20.8%, 35.3% and 62.4% for AR sizes of: 30, 40 and 60, respectively. Whereas, for the same sizes, the LR is more than 99% in the full set.

This result has one very important implication: even with very small IR sizes, many anonymous users can be identified. For example, with only IR and AR sizes of only 30, most users can be accurately linked (75% in Top-1 and 90% in Top-10). This situation is very common since many real-world users generate 30 or more reviews over multiple sites. Therefore, even reviews from less prolific accounts can be accurately linked.

We now experiment with another natural strategy of attempting to match all ARs at once.

In the previous section, we focused on linking one AR at a time. That is, ARs were independently and incrementally linked to IRs (accounts/reviewer-ids). One natural direction for potential improvements is to attempt to link all ARs at the same time. To this end, we construct algorithm $Match\_All()$ in Figure 8 as an add-on to the KLD models suggested in previous sections.

$SymD_{KLD}(IR_{j},AR_{i})$ symmetrically measures the distance between their ($IR_{j}$’s and $AR_{i}$’s) distributions. Since every $AR$ maps to a distinct $IR$ ($AR_{i}$ maps to $IR_{i}$), it would seem that lower $SymD_{KLD}$ would lead to a better match. We use this intuition to design $Match\_All()$. As shown in the figure, $Match\_All()$ picks the smallest $SymD_{KLD}(IR_{j},AR_{i})$ as the map between $IR_{j}$ and $AR_{i}$ and then deletes the pair $(IR_{j},V_{kj})$ from all remaining lists in $S_{L}$. The process continues until we compute all matches. Note that, for any $List_{AR_{k}}$, $(IR_{j},V_{kj})$ is deleted from the list only when there is another pair $(IR_{j},V_{lj})$ in $List_{AR_{l}}$, such that $SymD_{KLD}(IR_{j},AR_{l})\leq SymD_{KLD}(IR_{j},AR_{k})$, and $IR_{j}$ has been selected as the match for $AR_{l}$.The output of the algorithm is a match-list: $S_{M}=\{(IR_{i_{1}},AR_{j_{1}}),É(IR_{i_{n}},AR_{j_{n}})\}$.

We now consider how $Match\_All()$ could improve the LR. Suppose that we have two ARs: $AR_{i}$ and $AR_{j}$ along with corresponding sorted lists $L_{i}$ and $L_{j}$ and assume that $IR_{i}$ is at the top of each list. Using only KLD, we would return $IR_{i}$ for both ARs and thus miss one of the two. Whereas, $Match\_All$, would assign $IR_{i}$ to only one AR – the one with the smaller $SymD_{KLD}(IR_{i},...)$ value. We would intuitively suspect that $SymD_{KLD}(IR_{i},AR_{i})<SymD_{KLD}(IR_{i},AR_{j})$ since $IR_{i}$ is the right match for $AR_{i}$ and thus their distributions would probably be very close. If this is the case, $Match\_All$ would delete $IR_{i}$ (erroneous match) from the top of $L_{j}$ which could help clearing up the way for $IR_{j}$ (correct match) to the top of $L_{j}$.

We note that there is no guarantee that $Match\_All()$ will always work: one mistake in early rounds would lead to others in later rounds. We believe that $Match\_All()$ works better if $SymD_{KLD}(IR_{i},AR_{i})<SymD_{KLD}(IR_{j},AR_{i})$ ($j\neq~i$) holds most of the time.

In the next section, we show the results of $Match\_All()$ when we experiment with the KLD model with digram, rating and category tokens.²²2We also tried $Match\_All()$ with the NB model and it did not improve the LR..

Figures 9 and 9 show the effect of $Match\_All()$ on Top-1 LRs in both the restricted identified set and the full identified set, respectively. The combination of diagram, rating and category tokens are used. Each figure shows two Top-1 plots: one for the LR after using $Match\_All$ and the other – for the LR before using it. Clearly, $Match\_All$ is effective in improving the LR for almost all record sizes. For the restricted set, the gain in the LR ranges from 1.6-16.4% for nearly all AR sizes. A Similar increase is observed in the full set that ranges from 1-23.4% for most record sizes. This shows that the $Match\_All$ is very effective when used with diagram, rating and category tokens. The privacy implication of $Match\_All$ is important as it significantly increases the LR for small ARs in the restricted set. This shows that privacy of less prolific users is exposed even more with $Match\_All$.