A Computational Analysis of Polarization on Indian and Pakistani Social Media

In the first variant, we define $\mathcal{V}$ to be the set of all hashtags used in our data set. Then, we construct $G$ as a hashtag*hashtag co-occurrence network. Each node in $G$ corresponds to a hashtag. Edges occur between hashtags that co-occur in the same tweet, and edge weights are proportional to how frequently the hashtags co-occur. Then, we use the label propagation algorithm detailed in Algorithm 1 to infer polarity scores for $w\in\mathcal{V}$ from $S_{A}$ and $S_{B}$, where $a=-1$ and $b=1$. The algorithm uses a greedy approach to assign labels to each node in G. If all nodes connected to a node n have been labeled, then node n is assigned a weighted average of all the adjacent nodes. This step is repeated until the maximum possible number of nodes are labeled. A low value of $\gamma$ would label nodes neighboring unlabeled nodes, a high value would only label nodes neighboring unlabeled nodes after multiple iterations of the outer loop.

Our algorithm is similar to methods used to infer user-level polarities, in which a small seed of users is hand-annotated and a graph-based algorithm propagates labels to other users by assuming that users who retweet each other share the same views [21, 62]. For example, [29] quantify polarity based on a graph structure by assuming that the controversial topics induce clusters of discussions, commonly referred to as “echo-chambers”. However, we conduct propagation at a hashtag level, by assuming that hashtags that frequently occur in the same tweets indicate similar polarities. Also, our approach does not assume homophily in retweet network nor that user polarities are constant over time. Graph-based approaches have also been used to examine sentiment or for mixed tweet/hashtag/user-level analyses [19, 48].

The second variant is similar to the first; however, instead of restricting $\mathcal{V}$ to be the set of hashtags in the corpus, we define $\mathcal{V}$ to be the set of all tokens, including words and hashtags. We then construct G as a token*token co-occurrence network, and as above, we infer labels using Algorithm 1 and obtain token-level polarity scores in the range [-1, 1]. Expanding $\mathcal{V}$ to all tokens instead of just hashtags allows our algorithm to incorporate more information, but also risks introducing noise, as we do not attempt to process nuances in language like negation.

In the third variant, we define $\mathcal{V}$ to be the set of all tokens, as in the Network-based Word Propagation approach. Then, we train GloVe embeddings [47] over our entire corpus (limiting vocabulary size to 50K). We then use SentProp [31], a method for inferring domain-specific lexicons to infer labels over $\mathcal{V}$. In this method, as before, we construct a graph $G$ where each $w\in\mathcal{V}$ is a node. However, rather than relying on raw co-occurrence scores, SentProp uses embedding similarity metrics to define edge weights and a random-walk method to propagate labels. We implement SentProp using the SocialSent package [31], where $a=0$ and $b=1$.

Once we have obtained hashtag-level or word-level polarity scores, we infer the polarity of a tweet or a group of tweets (e.g. all tweets by a given user) by averaging the polarity scores inferred by our algorithms for all the hashtags and words used in data subset. This approach is similar to the aggregation conducted in [12], but our label propagation allows for the incorporation of thousands of words and hashtags, rather than relying on only a small hand-annotated set. If the data subset does not contain any of the keywords labelled by our algorithm (e.g. in a hashtag-based approach, the tweet contains no hashtags), we consider it unclassified. In some cases, primarily for evaluation, we convert the polarity scores into a ternary negative/neutral/positive position by using the cut-offs $\{<0,0,>0\}$ for the $[-1,1]$ scale and $\{<0.5,0.5,>0.5\}$ for the $[0,1]$ scale.

This methodology allows us to infer the polarity of any group of tweets along any dimensions, provided a small set of seed words or hashtags for each dimension. Thus, we can examine how polarities differed for different groups of users and how they changed over time. The two dimensions we focus on are Pro-India/Pro-Pakistan and Pro-Peace/Pro-Aggression. In practice we found that minor variations in the exact words in the seed set had no noticeable impact on our final results. For the network-based methods, we label Pro-India seeds as $+1$, Pro-Pakistan seeds as $-1$, Pro-Peace seeds as $+1$, and Pro-Aggression seeds as $-1$. For the embedding-based approach, we label Pro-India seeds as $+1$ Pro-Pakistan seeds as $0$, Pro-Peace seeds as $+1$, and Pro-Aggression seeds as $0$. For all word-based approaches, we limit the vocabulary size to 50K.³³3We provide our manually defined seed sets and label propagation code on https://github.com/amantyag/india_pakistan_polarization/.

We first evaluate our methods by focusing on the Pro-India and Pro-Pakistan dimension and assuming that popular users in India are more likely to post Pro-India content and popular users in Pakistan are more likely to post Pro-Pakistan content. From the Socialbakers.com platform, we identified the 100 most followed Twitter accounts in India and in Pakistan. 16 of the Indian accounts and 15 of the Pakistani accounts do not occur in our data, leaving 84 Indian accounts with 2,199 tweets and 85 Pakistani accounts with 1,456 tweets for evaluation. For each account, we average word and hashtag polarities over all tweets from the account, and binarize the resulting score into a Pro-Pakistan or Pro-India position.

Table 1 reports results. Both of the network-based approaches rely on hashtag or word co-occurrences to propagate labels. Thus, hashtags and words that do not have any co-occurrence links to the original seed list are unable to be labeled. For instance, in the hashtag propagation approach, our method labels 41,700 hashtags out of 67,059 total hashtags in the dataset. Any users who only use unlabeled words or hashtags are therefore unable to be classified by our algorithm, resulting in 88/169 unlabeled accounts for the hashtag approach and and 49/169 unlabeled accounts for the word approach (%Unk in Table 1). In contrast, SentProp obtains polarity scores for all accounts, as it relies on embedding similarity and can propagate labels between words, even if they do not ever co-occur.

However, although SentProp labels more accounts, its precision is much lower than the network-based methods. The network-based hashtag propagation approach overall obtains the highest precision and the least explicit errors – lower recall scores occur because of accounts that it leaves unlabeled, rather than because of accounts that it labels incorrectly. Although the word-propagation approach labels more accounts and works well over the Indian accounts, its classification of the Pakistani accounts is close to random. We suspect that our method works well for hashtags, because they tend to be strongly polar and indicative of the overall sentiment of the tweet. A word-based approach likely requires more careful handling of subtle language cues like negation or sarcasm.

In our subsequent analysis, we use the network-based hashtag propagation method in order to infer polarities, thus favoring high precision and strong polarization, and choosing not to analyze data where we cannot infer polarity with high-confidence. Additionally, in examining the data set, we found that many of the top-followed accounts in India and Pakistan consisted of celebrities who avoided taking stances on politicized issues, which makes the high number of unclassified accounts in this subset of the data unsurprising.

In order to further evaluate our methods, we compare the performance of the network-based hashtag model with a small sample of manually annotated tweets. We randomly sampled 100 users from our data set. For each user, we randomly sampled 1 day on which the user tweeted and aggregated all tweets from that day. Thus, we conduct this evaluation at a per-user-per-day level. Two annotators independently annotated each data sample as Pro-India/Pro-Pakistan/Neutral/Can’t Determine and Pro-Peace/Pro-Aggression/Neutral/Can’t Determine. For simplicity, we collapsed Neutral/Can’t Determine and Unclassified into a single “Neutral” label. Notably, the Pro-Peace/Pro-Aggression and Pro-India/Pro-Pakistan dimensions are distinct. For example, users may write tweets that are Pro-Peace and Pro-Pakistan: “Don’t let people pull you into their War, pull them into your Peace… Peace for our World, Peace for our Children, Peace for our Future !! #PakSymbolOfPeace #SayNoToWar” or that are Pro-Peace and Pro-India: “Very mature conciliatory speech by #ImranKhan. We now urge him to walk the talk. Please return our #Abhinandan safely back to us. This will go a long way in correcting perceptions and restoring peace. #SayNoToWar”.

Table 2 reports inter-annotator agreement, which is generally high. Additionally, most disagreements occurred when one annotator labeled Neutral/Can’t Determine and the other did not, meaning polar opposite annotations were rare. If we only count polar opposite labels as disagreements, the percent agreement rises to 94% for both dimensions.

Then, the two annotators discussed any data points for which they initially disagreed and decided on a single gold label for each data point. We compare performance of the network-based hashtag propagation method against these gold annotations in Table 2. In this 3-way classification task, the accuracy of random guessing would be 33%, which our method easily outperforms. In particular, the “Soft” accuracy, in which we only consider the model output to be incorrect if it predicted the polar-opposite label, meaning neutral/unclassified predictions are not considered incorrect, is high for both dimensions.⁴⁴4We provide the manual annotations as well as additional metrics on https://github.com/amantyag/india_pakistan_polarization.

In Table 3, we obtain polarity scores for each user and tweet and then ternarize them into Pro-India/Pro-Pakistan/Unclassified and Pro-Peace/Pro-Aggression/Unclassified as in §5. At the user level, the classified accounts are approximately balanced between Pro-India and Pro-Pakistan. However, at the tweet level, the classified data contains a high percentage of Pro-India tweets, suggesting Pro-India users tweeted about this issue more prolifically. Further, there is a much higher percentage of Pro-Peace users than Pro-Aggression users. This pattern also holds at the tweet level, where only a small percentage of tweets are unclassified.

Next, we examine the communication network between users, particularly prevalence of echo chambers. Did users with opposite positions interact? Figure 1 shows a 30-core all communication network constructed using ORA-PRO [14]. Accounts are colored based on their Pro-India/Pro-Pakistan polarity (left) and Pro-Peace/Pro-Aggression polarity (right). An edge occurs between two users if one user retweeted, mentioned, or replied to the other and users with $\leq 30$ links are not shown. Unsurprisingly, the Pro-India/Pro-Pakistan position is highly segregated, with little interaction between users with different positions. In contrast, the Pro-Peace/Pro-Aggression dimension is more mixed. Although there are some areas of high density for each position, there are interactions between users of different positions, which are potential avenues for users to influence each other’s views.

We investigate the polarities projected by different political entities: specifically BJP politicians (currently in power in India), INC politicians (largest opposition party), other Indian politicians, and Pakistani politicians. We used the Socialbakers.com platform to obtain the Twitter handles of the 100 most followed politicians in India and Pakistan. Our data contained tweets from 66 Pakistani and 78 Indian politicians, and our hashtag model inferred scores for 36 Pakistani and 33 Indian politicians. Figure 2 (left) reports aggregate polarity scores over all tweets from these politicians. Pakistani politicians were predominantly Pro-Peace, while Indian politicians expressed mixed polarities, yielding a near neutral score.

We then examined a broader set of Indian politicians, subdivided by political party based on a list of members running for parliament elections in 2019 [37]. Out of the 1,360 Twitter handles in the list, our data set contained activity from 316 BJP accounts, 281 INC accounts, 204 other Indian party accounts.

Figure 2 (right) shows the overall polarities, aggregated from all tweets by verified members of each party. Strikingly, members of the BJP party are positioned as much more Pro-Aggression than the members of either the INC or other parties, and the party overall obtains a Pro-Aggression polarity score. This score is not dominated by 1-2 strongly polarized members of the party: if we aggregate the polarity scores by individuals instead of by party, 15% of BJP members had net Pro-Aggression scores and 13% had net Pro-Peace scores, in comparison to 10% Pro-Aggression/25% Pro-Peace for INC, and 6% Pro-Aggression/29% Pro-Peace for other parties. The language used by BJP politicians was often openly Pro-Aggression: #IndiaWantsRevenge We need to give a befitting reply to Pakistan, we will strike back….

These results support observations made by journalists and community members about the role of the BJP party in these events. BJP is well-known for promoting nationalism, and several journalists have speculated that conflict with Pakistan would increase Prime Minister Modi’s chances of winning the upcoming elections in April and have accused the BJP of war-mongering [28, 41, 59].

Figure 3 shows how this polarity changed over the two-week period of events: we infer a Pro-Peace/Pro-Aggression polarity score for all tweets posted by members of the specified political subgroup, and we plot the average score across tweets posted each day.

Immediately following the initial attack on 2/14, the tweets from all Indian political party members are inclined towards Pro-Aggression, suggesting initial outrage. However, over the next few days, while tweets from INC and other Indian political party members switch towards Pro-Peace, tweets from BJP politicians remain consistently Pro-Aggression. There is high volatility between 2/20 and 2/26. However, there was a much lower volume of tweets about the Pulwama incidents during this time period,⁵⁵5Tweet volume is provided in our project repository. and we do not believe these fluctuations are meaningful. The volume of tweets increases once again following the Indian (2/26) and Pakistani (2/27) airstrikes. Tweets by Pakistani politicians generally fall on the Pro-Peace side, but they become more polarized after the Indian airstrike and reach a peak following the Pakistani airstrike. This is consistent with reported quotes by Pakistani officials (§2), saying that the airstrike was designed to avoid escalation. Similarly, tweets by Indian politicians from the INC and other parties become strongly Pro-Peace directly following the Indian airstrike, with polarity increasing after the Pakistani airstrike. In contrast, on the day of the Pakistani airstrike, tweets by BJP politicians remain Pro-Aggression, possibly focusing either on praise for the Indian airstrike or condemnation of the Pakistani airstrike. The polarity of the BJP tweets belatedly switches to Pro-Peace on the following day (2/28), though the strength of the Pro-Peace polarity still remains weaker for BJP tweets than for tweets by other politicians.