The wisdom_of_crowds: an efficient, philosophically-validated, social epistemological network profiling toolkit

Epistemology is a branch of philosophy which, simply put, “is concerned with how people should go about the business of trying to determine what is true” [18]. As per our Introduction (Section 1), social epistemology concerns the testimony of others embedded in social contexts [7]; in contrast with ‘individual’ epistemology which concerns how an individual conducts reasoning, abstracted away from their “social environment” [18].

In recent years, social epistemologists have moved away from considering dyadic relationships between individuals to consider the ways in which social epistemic networks shape the information we receive [15, 4]. Consider an epistemic network $G$ where nodes are epistemic agents and edges represent the relationship of receiving information via testimony. ‘Testimony’ is used broadly in social epistemology for any way in which one source delivers information to another, and includes speech, writing, and other forms of media. All things being equal, a node is better off receiving information from more and more diverse nodes. However, testimony is often transmitted in chains, and this transmission need carry only the content of the information, not (meta-)information about the original source or the intermediate links. This complicates the position of any individual who is trying to learn from multiple sources. For example, a piece of gossip heard from two people seems more reliable than from one, but that reliability is undermined if both heard it from the same person [3].

There are a few notes to make about the implications of the Sullivan et al. [19] core concepts in real social networks.

Regarding $m,k$-observers, while higher rankings are better and all the nodes with a specific value of $m$ and $k$ are members of an equivalence class, the framework does not posit which of two $m,k$-observers is better positioned if one has a higher $m$ value and the other a higher $k$ value—for instance, whether $2,3$-observers are better- or worse-placed than $3,2$-observers. The framework thus does not provide a total order but instead provides a collection of partial orders.

Intuitively, for instance, a $1,2$-observer is worse placed than a $2,3$-observer despite having neither an $m$ or a $k$ value in common, because a $2,3$-observer is better placed than a $2,2$-observer when considering $k$ values, which in turn is better placed than a $1,2$-observer when comparing $m$ values. Thus, a node’s $S$ value does provide a way to more easily compare the epistemic position of nodes, by combining their $m$ and $k$ values into a single value. There are concerns with it, though.

The choice of multiplying $m$ and $k$ values together to come to a single measure is ultimately arbitrary. The $S$ value contains less information than the $m$ and $k$ values does, because it fails to preserve the difference between being a $2,3$-observer and being a $3,2$-observer (the former has fewer independent sources, but these have a greater degree of independence from each other than in the latter’s). Simultaneously, it posits that a node which is a $3,3$-observer is determinately better placed than either a $4,2$-observer or a $2,4$-observer, but worse than a $5,2$-observer or a $2,5$-observer. It is unclear why we should believe that this is true in general.

In short, while a node’s $S$ value is one measure of the independence of its sources, it is unclear why we should use this measure rather than another. This was the impetus for us introducing a further measure, the h_measure(n) [8] which returns the highest $h$ such that $n$ is a $h,h$-observer, as discussed below.

The core of the wisdom_of_crowds is a class Crowd³³3Due to space constraints, we only discuss our substantial contributions within this paper. The full documentation for all implemented methods is available at https://github.com/cvklein/wisdom-of-crowds/blob/main/docs/wisdom_of_crowds.py.md. Crowd is initialized with a NetworkX graph (encapsulating the social network’s edges and nodes), and provides various functions to calculate the metrics defined above.

Calculating whether a node is an $m,k$-observer combines multiple shortest-path problems with clique-finding problems. Naïve approches to the latter have complexity $O(n^{k})$ [22]. Given that we are considering unweighted paths, the shortest-path problem has a reasonably efficient linear-time solution, but the requirement to remove $n$ from the distance calculations also means that network shortest paths cannot simply be calculated at the outset. In the worst case scenario, they must be recalculated for each pair of sources for each node.

Hence this is a computationally difficult problem to brute-force. In practice, efficient caching and testing of seen paths plus greedy $k$-clique algorithms means that worst-case performance can often be avoided for realistic networks: we memoize (i.e., cache) intermediate path values, trading-off space for a reduced processing time. As the envisioned use case for large graphs is for one-shot batch processing, our code allows for easy multithreading or multiprocessing (e.g. using multiprocessing or concurrent.futures) allowing it to attain substantial performance gains, in conjunction with the memoization mechanism.

The $m,k$-observer functionality is the basis for calculating $D$, $S$, and $\pi$. $D$ is calculated on node attributes, and users can specify the appropriate key for the attribute. If a single attribute is supplied, $D$ is calculated using the singleton set containing that attribute.

In addition to the standard measures, we also introduce an improvement: the h_measure() of a node $n$ as the smallest $x$ such that $n$ is an $x,x$-observer; comparable to the standard definition of Hirsch’s h-index in citation practices [8]. As suggested by [19], being a 3,3-observer is the minimal secure epistemic position, and the use of a single non-multiplicative standard may be useful for some cases.

Finally, the package includes two helper functions to allow for comparable reporting and display across different users. iteratively_prune_graph() takes a NetworkX graph, removes small-degree nodes, small-weight edges, and takes the largest connected component in what remains, iterating this process until the graph is stable. The thresholds are parameterized; the default is for $indegree+outdegree\leq 1$ and no edge culling, as per [19]. In the spirit of scientific reproducibility, make_sullivanplot() makes a summary figure of a whole network in the style of [19, refer to their Figure 7]. It can produce standalone plots or return a subplot in a specified matplotlib axis.

Firstly, to benchmark the ability of wisdom_of_crowds on different magnitudes of nodes, typical of modern-day datasets, we sought to batch calculate $S$ for different magnitudes of node sizes ($|N|$), given various probabilities of edge connections.

Figure 1 shows the efficiency of batch calculating $S$ for each node of a Crowd on random graphs with varying parameters for probability of edge connection. Random directed graphs were generated using the networkx generator
fast_gnp_random_graph() with the parameters indicated in the figure. Timing was done using the python $\verb+timeit+$ package over a single iteration.

As the log-linear plot in Figure 1 shows, there is a roughly exponential relationship between the number of nodes and runtime, with the exponent a function of the number of edges. This suggests that the efficiency of our code approaches what would be expected given the fundamental complexity of the clique-finding problem. Note that the exponential growth means that the boundary between computationally tractable and intractable graphs can be relatively tight. Judicious pruning often makes a difference.

Further, note that this was achieved in standard Python operating conditions, i.e. in the absence of any multiprocessing/multithreading support (see also Section 2.4). Performance gains will be attained for batch calculations on high-CPU (16-or-more CPU cores) systems.

Secondly, as part of our investigation of real-world phenomena, in line with [19], we replicate their findings on a real-world Twitter retweet network to examine the information-sharing dynamics during the Black Lives Matter movement. An earlier version of this dataset was used by [19] who were able to examine a network of 185 nodes. For further information on the dataset⁴⁴4We queried the Twitter Streaming API with a series of Black Lives Matter (BLM)-related keywords, hashtags, and short expressions in a window between January and July 2020.The dataset comprised $\sim$4.6M original tweets between January 13th and July 18th and $\sim$94.5M retweets from January 18th to July 23rd; these tweets were produced by $\sim$2.0M distinct authors. used, see [2]. For brevity, the key details are summarised here.

A retweet network [19, 2] was generated: i.e., a weighted directed network where nodes are authors and the weight of an edge from node $u$ to node $v$ represents the number of times that user $v$ retweeted user $u$. We took the largest connected component of this graph as the starting point for cluster-analysis [21, 6]. Following [2], We found first-level clusters using Modularity Vertex Partitioning, preserving clusters with more than 10% of the original nodes. This gave 4 clusters, covering 83% of the graph. Next, we manually inspected the 100 most-influential nodes within each group, characterizing the communities as Activists, Center-Left Democrats, Republicans, and a set of “Boosters” who mainly amplified the content of the first two groups. Topic models were fit using scikit-learn’s non-negative matrix factorization (NMF), fit on a tf-idf representation of the Twitter text (post-sanitization) with min_df=0.05. Finally, we used iteratively_prune_graph() with a node and weight threshold of 3, which resulted in a tractable subgraph with $|N|=16249$ nodes and $|E|=145246$ edges. This subgraph had very little representation from the ‘booster’ group, so they were omitted from further analysis. By comparison with [19], our analysis is an increase in the number of nodes by a magnitude of $\sim$100$\times$. Batch processing took about 6.25 hours on a 2017 desktop iMac.

Figure 2 plots $S$, $D$, and $\pi$ for the current experiment. For the left side of 2, $D$ was calculated using the cluster identity of the node. For the right side, $D$ was calculated using the argmax of the fitted and normalized $W$ matrix for the topic model. This gives the topic that is most distinctive of each user’s tweets. We examine both the network as a whole and three identified subclusters in the graph. The left half of Figure 2 shows $S$, $D$, and $\pi$ for the network, where $D$ is calculated via the subcluster identity of sources. The right half of Figure 2 recalculates $D$ and $\pi$ based on a 9-topic NMF topic model of aggregated tweets (compared with the 3-topic model of [19]).

The results illustrate the utility of profiling networks using our toolkit. On the left, one can see that Republicans appear to be in the worst epistemic position in terms of the other subgroups with which they interact: they have a generally low $D$, suggesting that they tend to listen mostly to in-group members. However, they have a relatively high $S$ and therefore a $\pi$ comparable to other subgroups. Compare this to the topic-wise graph, in which Republicans have a a relatively high diversity for topics, one at least as good as other groups. The activist group shows something of the inverse pattern. That is, they show a more varied range of $S$ and $D$ values for group-group interactions, but a comparatively lighter graph with fewer topics for the broad span.

These results might suggest that both groups are part of ‘echo chambers’ [1, 14], but in different ways: the right tends to be a monoculture socially but a polyculture topically, with a converse pattern on the left. Finally, we note that all subgroups, in both domains of measurement, tend to have an $S<10$ for more than half the population. This replicates the observations of [19], in which most participants end up in a comparatively poor epistemic position. However, most groups tend to contain at least a small sub-population which is well-connected and often with a relatively high $D$. We note that this is especially the case with our ‘Booster’ group, a small subset who seemed to be mainly concerned with amplifying and relaying the content of other groups.

Finally, we apply our wisdom_of_crowds analysis onto the email-Eu-core network, an existing dataset curated by [12, 23] on the Stanford Large Network Dataset Collection [11]. The rationale behind this is to apply our social epistemological lens to analyze an existing network which is hitherto (to our knowledge) never been examined using the tools we have at hand.

Briefly, email-Eu-core consists of “email data from a large European research institution” [11], represented as a digraph where an edge ($u$, $v$) exists “if [researcher] $u$ sent [researcher] $v$ at least one email”. The beauty of email-Eu-core lies in the fact that it only considers internal communications in the institution, ignoring any noise resulting from possible non-academic communication originating from/in reply to outside actors; and that the ground-truth membership of each node has already been established, i.e., as belonging to any “one of 42 departments at the research institute” [11].

Compared to more modern social networks, email-Eu-core has a comparatively small $|N|=1005$ and $|E|=25571$. However, again, we note that this is already $\sim$10$\times$ the magnitude as compared to the hitherto state-of-the-art [19]. The total running time on such a dataset is comparatively trivial (less than 5 minutes). As we do not have e-mail text associated with the original nodes in this dataset, we sought only to analyze the ‘overall epistemic picture’ of the network, as a whole. In the same vein as Section 3.2, Figure 3 illustrates the profile plot for email-Eu-core.

Again, the utility of our approach is evident. Given the results, we observe that the distribution of $S$ is fairly consistent for all the nodes (researchers) in the research institution’s academic network. However, the progressively darker bars illustrate that researchers who have connections with a more diverse amount of departments (thereby maximizing $D$) can vastly optimize their epistemic position. Roughly, about $\sim 40\%$ of researchers – i.e., the right-most data points in Figure 3 – have a $\pi$ of about 100 or more, despite having roughly the same $S$ values.

To our knowledge, this is the first time empirical social epistemological analyses in the spirit of [19] have been conducted on such email networks.