Countering Underproduction of Peer Produced Goods

Commons-based peer production is a term coined by Yochai Benkler to describe an emerging form of cooperative production made possible by new communication technology. Tasks are self-selected and combined through technology, resulting in information public goods. Examples of peer production include free/libre open source software (FLOSS) projects like GNU/Linux and Apache, Wikipedia, and OpenStreetMap (OSM). Benkler, (2006) argues that because individuals know their interests, knowledge, availability, and skills, peer production creates the potential for diminished separation between producers and consumers and that self-assignment to tasks supports efficient matching between contributors and tasks.

One way to assess the success of a peer production project is to examine how well it meets the needs of its users. Warncke-Wang et al., (2015) found that over 40% of views to English Wikipedia were to articles that were lower quality than one might expect given their popularity. Countries, religions, LGBT topics, psychology, pop and rock music, the Internet and technology, comedy, and science fiction were found to be disproportionately affected. The misalignment of quality and public interest in these topics is concerning because of their relevance to public affairs and modern culture.

To quantify misalignment, Warncke-Wang et al., (2015) propose that if a system is aligned, goods in the highest demand should be the highest quality while those in the lowest demand should be the lowest quality. When quality is low relative to demand, goods are described as underproduced. When quality is high relative to demand, they are overproduced. Although overproduction may not be harmful beyond the potential for wasted effort, underproduction can impact public knowledge goods and digital infrastructure (Champion and Hill,, 2021; Eghbal,, 2016).

To understand the sources of underproduction, we must consider why people participate in peer production. Studies have observed a range of motivations. Some are motivated to create public goods (Budhathoki and Haythornthwaite,, 2013), to address social issues (March and Dasgupta,, 2020), to help other community members (Wu et al.,, 2007), to enhance their own use of a public good (Krishnamurthy et al.,, 2014; Meng and Wu,, 2013), to enhance their reputation or qualifications (Silva et al.,, 2020; Xu and Li,, 2015; Oreg and Nov,, 2008), as an experience of self-efficacy (Yang and Lai,, 2010), as a professional responsibility (Farič and Potts,, 2014), for personal enjoyment and learning (Farič and Potts,, 2014), out of reciprocity (Xu and Li,, 2015), for a class assignment (McDowell and Vetter,, 2022; Xing and Vetter,, 2020; Coelho et al.,, 2018; Konieczny,, 2012), or as part of their job (Germonprez et al.,, 2019). The results of this body of literature suggest that contributor motivation in peer production tends to be complex and multidimensional and primarily, although not exclusively, intrinsic (Benkler et al.,, 2015; Rafaeli and Ariel,, 2008; Belenzon and Schankerman,, 2015; Kuznetsov,, 2006).

To the extent that peer production contributors do so because they are told to (e.g., by an employer or instructor in a class), extrinsic rewards and others’ interests may be primary motivators. For example, contributing to Wikipedia is an increasingly common assignment in college classes where instructors are driven by concerns about content gaps or their commitment to public goods (McDowell and Vetter,, 2022; Xing and Vetter,, 2020; Konieczny,, 2012), or by a desire to have students engage with their coursework in public (Gallagher et al.,, 2019). Firms may encourage their employees to contribute to FLOSS to enhance the firm’s use of the public good or to build their reputation (Germonprez et al.,, 2019).

Nonprofit organizations also encourage contributions to peer production projects as part of their missions. Examples include the Missing Maps project by the Red Cross and Doctors Without Borders encouraging contributions to OSM (Herfort et al.,, 2023), partnerships encouraging contributions to FLOSS projects targeting climate change,²²2e.g., Climage Triage https://perma.cc/9R8F-QQYY and galleries, libraries and museums partnering with Wikipedia (Karczewska,, 2023). Some non-profits seek to counter content and participation gaps in peer production. For example, WikiEdu supports instructors using Wikipedia in their course assignments to close content gaps (Wilfahrt and Michelitch,, 2022). The Outreachy initiative aims to increase diversity in open source (Ackermann,, 2023).

Within this complex motivational landscape, contributor task selection varies across users and within users over time. In their examination of OSM contributors, Budhathoki and Haythornthwaite, (2013) reported that high-volume contributors were likelier to seek community recognition, while low-volume mappers reported wanting to contribute to a free and open project. In contrast, Shah, (2006) showed that FLOSS contributors describe their first contributions as directly related to personal skills, needs, and priorities, while longer-term participants report working for the good of the project. Likewise, Bryant et al., (2005) found that while Wikipedia editors reported participating within their areas of expertise initially, they sought to build the Wikipedia community and serve the public good over time. Restivo and van de Rijt, (2014) found that the highest-volume contributors produce more after receiving informal social rewards, while lower-volume contributors did not and may even respond negatively to the same awards.

Previous work offers some additional clues to the relationship between experience and choosing to contribute to underproduced goods. New contributors initially select tasks that match their immediate needs, areas of knowledge, and skills (Preece and Shneiderman,, 2009; Bryant et al.,, 2005). Because people are interested in similar things, it seems likely that these areas of knowledge will, on average, be associated with high-interest subjects. On the other hand, contributors faced with high-quality artifacts may not see where their assistance is needed (Preece and Shneiderman,, 2009; Bryant et al.,, 2005). However, as they accumulate experience, we expect contributors’ skills to increase.

Although this literature points in conflicting directions, prior work has emphasized that skilled participants incorporate the public’s desire for information as a component of their task selection. As a result, we expect to find that the most experienced contributors are more likely to engage in the improvement of underproduced goods: H1: individuals with less experience will contribute to less underproduced goods than individuals with more experience.

In that experienced contributors may seek recognition for their work, social factors may act to disrupt the trend we describe in H1 (Oreg and Nov,, 2008). Once a participant is involved in a project, their ongoing participation may be encouraged by a desire for status (Willer,, 2009). To distinguish the extent to which social rewards drive task selection, we take advantage of the fact that receiving in-group social rewards partially depends on identifiability. Collaborators may seek to direct their responses to an attributed person. Previous research suggests that creating an account may be driven by a desire to obtain feedback or recognition (Forte et al.,, 2017). By contrast, Belenzon and Schankerman, (2015) argued that anonymous contributors to FLOSS projects were more motivated by a desire to contribute to the public interests, as measured by the fact that they were more likely to contribute to projects operated by a nonprofit than a for-profit entity, and those serving end users (rather than other developers). Because contributing without an account makes social rewards more difficult, we propose that H2: contributing without an account is associated with more underproduced goods than contributing with an account.

Finally, we consider two competing explanations for H1 and H2. An association between task selection and experience could be explained by contributor attrition (i.e., those who go on to make many contributions have always differed from those who make only a few) or by shifting motivations (i.e., persistence causes changes in task selection). In other words, if we find support for H1 and H2, is it due to differences caused by users being “born” or “made”? Support for the “born” explanation comes from past work that has found that individuals who go on to contribute at high volume seem to do so from the start (e.g., Panciera et al.,, 2009). On the other hand, contributors’ motivation appears to shift over time (e.g., Shah,, 2006; Bryant et al.,, 2005). Moreover, peer production projects have successfully invested in efforts to socialize and retain newcomers (e.g., Tan et al.,, 2020; Morgan and Halfaker,, 2018), suggesting that contributors can also be “made”, at least to some extent. Given this evidence of motivational shifts, we hypothesize within-person change as H3A: an individual will shift toward underproduced goods as they accumulate experience, and H3B: an individual not using an account will shift toward underproduced goods at lower experience levels than one using an account. Our reasoning for H3B is that because non-account-users are less likely to receive social rewards, they are even more likely to be motivated by the public interest.

This study is conducted in the context of Wikipedia, the Wikimedia Foundation (WMF) website. Wikipedia is one of the most popular websites in the world and is used as reference material, a fact-checking source, and an input to machine learning and AI systems (McMahon et al.,, 2017). Wikipedia projects in 326 different language editions received 29 billion page views in January 2024.³³3 WMF Statistics: https://perma.cc/H3NT-RP9N, Wikipedia page about Wikipedia: https://perma.cc/RNC3-4SLN The largest Wikipedia language edition, English, has more than 6.7 million articles. With some exceptions, clicking on an ‘Edit’ tab on every Wikipedia page creates an interface for revising the existing text. For most language editions, changes are immediately visible without review.

Our unit of analysis is the revision, and we use the complete revision history of Wikipedia, made available by the WMF, from its inception in January 2001 through July 2021. The complete revision history of Wikipedia contains a wide range of content, including discussion and personal pages.

Our hypotheses concern the relationship between experience and having an account with article underproduction, a function of viewership and quality. To test our hypotheses, we filtered the revision population in multiple ways before drawing our sample. We excluded work done by bots, revisions to non-article pages, vandalism and its removal, and purely administrative revisions. Because no viewership data are available before December 2007, we only considered contributions after December 2007. Details on these filters and the total revisions after each step are reported in our online supplement.

In Wikipedia, a small fraction of contributors make a large portion of all contributions. To capture variation based on editor experience, we drew two samples. For the revision sample used to test H1 and H2, we first stratified contributions by their ordinal position into increasingly large “buckets” increasing in size by $2^{x}$ (i.e., 1, 2, 4, 8, and so on). We then drew equal-sized random samples from each bucket, oversampling where the data was thinner, for a total of 192,672 revisions. The inverse of the resulting sampling proportions was used as weights during subsequent calculations (Tracy and Carkin,, 2014).

The second sample is a within-person sample, designed to extract the contribution history of editors who differed in their ultimate contribution level. To build this sample, we stratified editors based on their total contributions through July 2021, again by creating increasingly large ($2^{x}$) buckets. We then drew a random sample of editors from each. Again, we oversampled where data was thinner and retained proportions as weights. After selecting editors, we gathered all article contributions made by the selected editors for a total of 42,602,912 revisions. This approach gives us the contribution history of a diverse range of editors: those who went on to contribute a great deal and those who only contributed once or twice.

We have made our data and code available at https://doi.org/10.7910/DVN/UDQT6E.

Our dependent variable is underproduction factor. To construct this measure, we drew from elements of both Champion and Hill, (2021) and Warncke-Wang et al., (2015) and calculated the negative log of the ratio between quality rank and popularity rank. Both ranks were assigned such that high quality and high popularity receive high ranks, while low quality and low popularity receive low ones. A perfectly aligned article—one with the same popularity and quality rank—would have an underproduction factor of 0 since log(1) is 0. An overproduced article of high quality and low popularity would have a negative underproduction factor. An underproduced article of low quality and high popularity would have a positive underproduction factor.

We count views at the monthly level using data released by the WMF. We calculate quality at the monthly level using the ORES quality measure. ORES is a machine learning-based quality measure that reflects the structural characteristics of articles, such as the presence of references, section headings, and inter-wiki links (Halfaker et al.,, 2016). ORES is a continuous measure with separate predictions for six article quality levels. We treat these levels as equally spaced to create a single quality score between 0 and 5. More details are provided in the supplement.

We operationalize editor experience as revision count: the number of revisions the editor has made before and including the current revision. Therefore, before conducting the filtering described in the data and sample section, we calculated each revision’s ordinal position in its contributor’s edit history, ignoring any reverted contributions. To consider the influence of the potential for receiving social rewards, we also introduced a variable reflecting whether contributions were made by users who were not using accounts. Because these users are identified by IP address in Wikipedia, we call this measure IP-based.

We employ a hierarchical multiple regression model to test our hypotheses H1 and H2. We use revision count (logged), revision count (logged) squared, and IP-based as predictors and a random intercept term for users using the lmer package in R and a ‘raw’ specification for our polynomials.

Although our hierarchical models attempt to correct for repeated measures of users, this approach does not account for the fact that experienced users might edit different articles than less experienced users in ways that reflect underlying differences in the types of users who go on to become less or more experienced (Panciera et al.,, 2009). To answer H3A and H3B, we fit a second group of models with user-level fixed effects using the felm package in R and our within-person sample. Because we have no within-person variation in account use, we cannot include IP-based in the within-person model. Due to heteroskedasticity, we use robust standard errors in all our models.

This study was conducted entirely using publicly available data published by WMF and does not involve any interaction or intervention with human subjects. The IRB at our institution has reviewed this type of research using these data and determined it is not human-subject research. WMF fully anonymized article view data before release.

This article offers insight into community-wide trends and within-contributor changes in task selection behaviors. Our results in H1 and H3A suggest that as they increase in experience, contributors tend to shift their attention to underproduced content. This is consistent with the “made” argument that accumulating experience is associated with editing underproduced articles and, presumably, with increased exposure to socialization. Our results in H2 and H3B call this interpretation into question. Socialization suggests a social process, but contributors who participate without an account have less opportunity to build a social identity in Wikipedia. How does this occur if these contributors also shift towards underproduced articles?

Our models suggest that IP-based contributors are initially less inclined to select underproduced articles but are comparable to account-based contributors for moderate contribution levels (i.e., in the 100-200 range). This finding opens up a new set of challenges for social research. Given that contributors without accounts also move toward underproduced goods—and hence may represent a valuable subgroup to seek to influence and retain—how should we explain their persistence and changing task selection?

One way to understand this shift in the behavior of IP-based contributors is through an extension of learning theory—e.g., Preece and Shneiderman,’s (2009) reader-to-leader perspective and Lave and Wenger,’s (1991) legitimate peripheral participation. As participants gain experience, perhaps their awareness of underproduced articles increases. Given that this process is mediated through technology in rich social computing environments, this may be evidence of what we call “technosocial learning,” rather than social learning per se.

We suggest that technosocial learning may be an important mechanism to shape behavior in peer production environments. For example, consider the general phenomenon of feedback on a contribution. Someone contributing without an account might not receive such feedback. However, this contributor could notice whether their past contribution remains present.

Platforms offer other opportunities to learn from technical traces: contributors might experience success in mastering a particular formatting trick, observing the work of others, reading documentation, or seeing feedback others receive. A technosocial learning process, illustrated in Table 3, may provide a mechanism for the changes in task selection we observe.

Given the previous finding of widespread underproduction in peer production (Warncke-Wang et al.,, 2015; Champion and Hill,, 2021), countering it is an important challenge. One approach to addressing underproduction is to increase retention overall. Another approach is nudging contributors towards more severe areas of underproduction earlier: both when they are newer to platform, and more quickly in response to emerging needs. For example, peer production communities responded during the COVID-19 pandemic by developing FLOSS analytic and visualization tools (Wang et al.,, 2021), organizing to produce Wikipedia articles to provide reliable information at high speed (Avieson,, 2022), and producing personal protective equipment and filling supply chain gaps (Sarkar et al.,, 2023). Evaluation of recommender systems suggests they can influence contributors to improve underrepresented topics, providing the recommendation’s relevance is kept constant (Houtti et al.,, 2023). Creating interest groups within a larger project may also be an effective way to tackle underproduction. In Wikipedia, contributors have self-organized around themes of interest to identify and work to close a wide range of participation and content gaps—e.g., WikiProject Women in Red (Tripodi,, 2021), WikiProject Women Scientists (Halfaker,, 2017), and WikiProject Vital Articles (Houtti et al.,, 2022).

However, more work is needed to understand when these interventions are effective. Ford et al., (2018) found that notification of existing contributors, writ large, was insufficient to address content gaps, instead recommending recruiting people with expertise related to the gap. A computational linguistics analysis from Schmahl et al., (2020) found that gendered bias in articles changed modestly, at best, between 2006 and 2020 when many targetted interventions were conducted. Successful interventions in task selection include not only involving diverse participants but also participants with diverse motives (e.g., through course assignments and non-profit outreach activities (as per Konieczny,, 2012; Coelho et al.,, 2018; Gallagher et al.,, 2019; Xing and Vetter,, 2020; Herfort et al.,, 2023; Karczewska,, 2023)).

Our analysis points to promoting contributor retention as a means of countering underproduction. Scholarship in online community and peer production community management has also emphasized retention as a path toward building healthy online communities (e.g., Kraut et al.,, 2012). However, our findings should be read alongside the evidence against requiring account creation and the risks associated with overemphasizing the retention of existing users.

Requiring accounts has been found to carry unintended consequences in terms of diminishing overall community activity levels (e.g., Hill and Shaw,, 2021). Being prompted to create an account increases the barrier to contributing. Maintaining a low barrier to entry may be essential to the success of peer production projects. Beyond this risk, Shaw and Hill, (2014) found that governance of peer production tends to concentrate power in the hands of elite long-standing contributors, emphasizing that longevity risks further empowering elites. Additional analysis of long-term contributors appears in our supplement.

Increased retention may also magnify systemic biases reflected in peer production. For example, previous work has explored geographical biases in OSM (Thebault-Spieker et al.,, 2018), gender inequality in FLOSS participation (Qiu et al.,, 2023), gender bias in how FLOSS projects engage with contributors (Chatterjee et al.,, 2021), gender inequality in Wikipedia contributors (Hill and Shaw,, 2013) and gender bias in Wikipedia content (Adams and Brückner,, 2015). Therefore, while retaining contributors is valuable, it may reinforce inequality in the population of early contributors. Scholarship has shown that communities can make progress in closing participation gaps with interventions that address bias in sociotechnical processes through which people are retained unequally (Karczewska,, 2023; Langrock and González-Bailón,, 2022; Hilderbrand et al.,, 2020; Evans et al.,, 2015).

Along with recruiting and retaining diverse contributors, we believe there is also value in retaining those who contribute without accounts. A proposed update to Wikipedia’s software slated for 2024 describes a change to the way that users without accounts are identified by replacing visible IP addresses with temporary identifiers.⁵⁵5See implementation proposal: https://perma.cc/3H59-YLH8 Although details of this proposal are still in flux, this change will improve the privacy of those contributing without an account by hiding IP addresses that may reveal geographic location. Although the proposed alternative will improve the ability of researchers to track contributions from an individual across locations, it will also make it impossible to do studies of long-term engagement of users contributing without accounts—like ours. Evidence-based management of peer production communities faces multiple critical challenges: countering underproduction, enabling diverse participation, expanding community safety, building long-term commitment, and minimizing power concentration. Some of these goals may force difficult tradeoffs, as interventions intended to address one area may make others more difficult to address.