The Web unpacked: a quantitative analysis of global Web usage

SimilarWeb provides information on how much the monthly visits of a domain increased or decreased in the last month compared to the previous one. This change, represented as a percentage, is called month-over-month variation (MoM). Due to practical data access issues, we could only analyze the MoM distribution of the 8,000 most visited domains. Fig. 1 shows that, despite a moderate variance (68% of the MoMs stay between -5% and 19%), outliers are frequent both at low and high MoMs, strongly affecting non-robust statistical measures like the mean and the standard deviation. The average absolute difference between consecutive monthly visits is 15%, about three times smaller than the average difference between SimilarWeb and NetMonitor measurements (Prantl and Prantl, 2018). This slight variation indicates that methodological differences in traffic measurement are more important sources of error than differences in the time frame.

We also see that the distribution favors positive MoMs – the distribution median is 3,6%, and bootstrapping shows this value is significantly greater than zero. We suggest three explaining factors for this increase in monthly visits over time:

1. (1)

   the total traffic on the Web is increasing, with people spending more time or more people gaining access to it (Union, 2023);

2. (2)

   the top ranking websites are concentrating traffic, at the expense of less visited ones (Society, 2019); and

3. (3)

   there is a seasonal pattern (similar to the ones in (Liu et al., 2017, 2018)) in which traffic typically increases from September to October.

Further quantitative exploration of these factors would require data distributed over a larger time frame.

Fig. 2 shows the MoM distribution as a function of rank position. The dark blue line represents the median computed in a moving window containing 200 domains, and the gray line represents the median absolute deviation (MAD) – multiplied by $-1$ to avoid overlapping with the blue line – computed in the same window. Apart from a statistically significant decrease in the MAD at the top of the rank (meaning that the monthly visits variation is more predictable for the top 2,000 websites), the distribution is reasonably stable in the remaining range.

Since the MoM measures fluctuations in a domain’s visits and its distribution is stable for most of the rank, we adopted a centered (i.e., median-subtracted) version of the MoM distribution shown in Fig. 1 as a model for the statistical error on the monthly visits estimates, assuming the errors are independent and identically distributed. We will use this error model in the following section.

We identified that the traffic (i.e., monthly visits) is distributed among domains in a power-law-like fashion. Previous papers have reported similar findings regarding the number of connections between internet nodes (Faloutsos et al., 1999), number of incoming links to a website (Adamic and Huberman, 2000b), the time spent by users on websites (Crichton et al., 2021), and number of unique visitors to domains (Adamic and Huberman, 2000a; Webster and Lin, 2002). Despite being correlated, these quantities are not the same. Among these studies, (Adamic and Huberman, 2000a) had the most similar metric and data to ours: it used network-centric data from 60,000 AOL users active in December 1997 to rank 120,000 domains concerning the number of unique visitors.

Fig. 3 shows a log-log plot of the monthly visits $V$ vs. rank position $p$ for the domains in our dataset. A power-law $V(p)=V_{0}p^{\beta}$ appears as a straight line in such a plot. We obtain a best-fit model to our data with $V_{0}=4.1\times 10^{11}$ and $\beta=-1.19$. We also represent as pink bands the range covered by a $2\sigma$-equivalent interval (i.e., from the 2nd to the 98th percentile) assuming statistical fluctuations described in Sec. 3.1. The narrowness of the bands demonstrates that the properties of the traffic distribution should be reasonably stable over time (at least for a few months) and that the overall ranking of the domains is not significantly affected by measurement errors.

To calculate the share of the total web traffic captured by each domain in our dataset, we first need an estimate of this total traffic. For that, we extrapolated the best-fit power law to position 354 million, the number of registered domains as of June 2023 (Verisign, 2023), and aggregated the measured traffic along with the estimated traffic beyond position 254,661, resulting in a total of 781 billion visits per month. With 5.35 billion internet users as of January 2024 (Petrosyan, 2024), this amounts to 4.9 domain visits per person per day, a reasonable ballpark figure: previous work has shown that users from a metropolitan area in the United States visited an average of 20.1 websites per day on their computers, while the least active user (out of 257) visited an average of 2.9 (Crichton et al., 2021). Including rural regions and other countries should lead to a lower average. Fig. 4 shows the estimated traffic share for the 254,661 domains in our data.

Extrapolations are risky as there is no guarantee that the data behavior will remain the same beyond the measured range. However, it seems reasonable to assume that the monthly visits will continue to decline more or less like a power law, with an exponent $\beta$ not much different from the observed one. With this assumption, we expect the traffic contribution from the less visited domains to be minor, especially if $\beta<-1$.

To estimate a systematic uncertainty on the shares of total monthly web traffic, we fitted a power law up to position $p_{i}=10(p_{\mathrm{last}}/10)^{i/99}$, with $i=0,\ldots,99$ and $p_{\mathrm{last}}=254,661$, representing several hypothetical cases where we have data only for the first $p_{i}$ domains. The smallest and largest $\beta$ obtained, along with the respective power-law scaling factor $V_{0}=V(p_{\mathrm{last}})/p_{\mathrm{last}}^{\beta}$, were used as alternative extrapolations to estimate a systematic uncertainty interval on the total monthly web traffic. The resulting interval for the traffic shares is shown as a violet band in Fig. 4.

Our analysis reveals a significant concentration of web traffic in a few domains. Our best estimate suggests that 50% of all web traffic is directed to the top 3,000 domains, and 80% is allocated to domains in our dataset. Even our most conservative estimate indicates considerable concentration, with 50% of the traffic directed to approximately 60,000 domains. This scenario corresponds to a lower bound on the Gini coefficient – a measure of inequality commonly used in economic analysis – of 82%, a value comparable to wealth inequalities observed in countries such as El Salvador, India, Kenya, Paraguay, and Turkey (Credit Suisse, 2022).

Table 1 presents values for our best traffic extrapolation and for the systematic uncertainty boundaries with the smallest and largest traffic concentration, denoted as Sys. $-$ and Sys. $+$. These values are: the power-law exponent, $\beta$; the total monthly web traffic, $\sum V$; the average number of domain visits per person per day, $\bar{u}$; the extrapolated traffic for the last domain in the rank (among 354 million), $V_{\mathrm{min}}$; the number of top domains that accumulate 50% of total web traffic, $p_{50\%}$; and the Gini inequality coefficient.

The $V_{\mathrm{min}}$ value for the Sys. $-$ boundary is implausibly large, indicating that the traffic share must decline faster than this rate, especially towards the end of the traffic rank. Since not all 354 million domains have an associated website, traffic must decline even faster to reach a reasonable $V_{\mathrm{min}}$ at an earlier position. Thus, the actual traffic share is likely closer to our best estimate than to its lower boundary. The Gini coefficient associated with our best traffic extrapolation far exceeds the wealth inequality of any country worldwide. The fact that the domains in our dataset likely represent the destination of 80% of all web traffic makes them an excellent sample of the current state of the Web, at least in terms of content access rate.

SimilarWeb categorizes domains into 210 different industry segments. Some of these segments are broader categories that encompass others. By considering only the finer, non-overlapping categories, we are left with 187 industry segments. We aggregated the monthly visits of domains into these 187 categories to estimate the most popular industries on the Web. This computation better represents the popularity of topics than others found in the literature (e.g., (Agarwal and Sastry, 2022)), as it ranks them by monthly visits rather than by the number of websites. Additionally, it utilizes a significantly vaster set of domains compared to previous studies.

Figure 5 illustrates that traffic coalesces in a few industries, with 50% directed to just five segments and 80% directed to 26 segments. Figure 6 depicts the average monthly visits for these sectors. In most cases, despite the power-law distribution of traffic within industries, monthly visits are not monopolized by a single domain or company, and the most popular domains accumulate at most 15% of their industry’s traffic. However, there are exceptions, as noted below.

Figure 6 demonstrates the prevalence of “Search Engines” as the most visited industry. This prevalence is expected due to the lack of a built-in index on the Web, making search engines entry points for most users. This trend is further amplified by the integration of address and search bars in web browsers, leading users to search even for known websites instead of directly typing their URLs (Cannon, 2008). In this industry, google.com dominates with 79% of the traffic (considering the top 10,000 domains in the segment). The following competitors are the Chinese search engine baidu.com, with 4.7%, and the Russian search engine yandex.ru, with 3.0%.

With 40% less traffic than “Search Engines,” the next most popular industry is “News and Media,” which comprises a wide variety of websites, including web portals such as yahoo.com. Following this is ”TV Movies and Streaming,” where youtube.com (owned by Alphabet, like google.com) captures 61% of the industry’s traffic. This segment is succeeded by “Social Networks and Online Communities,” another case where the industry’s traffic predominantly flows to a single company. When combining facebook.com, instagram.com, and whatsapp.com, Meta receives 52% of the segment’s traffic. The last industry in the top five, before a traffic drop of more than 50%, is “Adult.” Despite its popularity on the Web, this segment may be underrepresented in surveys and research where subjects know they are being monitored (Brazilian Network Information Center, 2022; Crichton et al., 2021). Among the remaining 26 industries listed in Figure 6, only “Dictionaries and Encyclopedias” has a domain (wikipedia.org) with more than half (56%) of the total industry’s traffic.

It is essential to observe that domains categorized as “Search Engines,” such as google.com, yandex.ru, and baidu.com, offer more services than just web search. Consequently, a portion of the traffic attributed to the “Search Engines” segment pertains to other industries listed in SimilarWeb’s classification, such as “Email,” “File Sharing and Hosting,” and “Maps.” While some of these services are hosted on subdomains and are tracked independently by SimilarWeb (e.g., news.google.com and cloud.google.com), others are not (e.g., support.google.com, mail.google.com, and calendar.google.com). Nevertheless, it is feasible to disregard traffic from all subdomains (SimilarWeb, 2024a). For google.com, this results in a 15% reduction in traffic, maintaining the integrity of our rankings. Note that services like Google Maps are hosted at www.google.com/maps, so their data remains inseparable from that of google.com.

Considering Google’s widespread popularity and the multitude of services it offers, it is valuable to assess the significance of each service within the overall traffic of google.com and its effect on the traffic attributed to the respective industries. To achieve this, we conducted manual searches on SimilarWeb to gather data on Google’s subdomains that are neither independently tracked nor related to search. Fig. 7 illustrates these findings. By juxtaposing these subdomains’ traffic data with google.com’s average monthly visits of 85.7 billion, it becomes evident that search continues to reign as Google’s most frequented service on the Web.

Nonetheless, Google offers several additional services that would significantly contribute to its industry’s traffic if SimilarWeb had independently monitored them. For instance, in the “Email” segment, mail.google.com received an average of 10.9 billion monthly visits, whereas the top domain in this segment, live.com, received only 2 billion visits. Similarly, with 5.6 billion visits, docs.google.com would lead the “Programming and Developer Software” segment, which includes Software as a Service (SaaS) platforms like office.com, which received 1.5 billion visits. Additionally, drive.google.com would lead the “File Sharing and Hosting” segment with 2.5 billion visits, surpassing the 186 million visits received by the industry leader, mediafire.com. In the “Computers Electronics and Technology - Other” category, domains like calendar.google.com, meet.google.com, and bard.google.com would rank among the top ten most visited domains. Furthermore, translate.google.com would secure the third position in the “Dictionaries and Encyclopedias” segment, surpassing the translation website deepl.com.

SimilarWeb data provides no other information about the domains besides visitation metrics and the domain’s industry. To address this gap, we manually inspected and researched the 116 most visited domains, which collectively capture between 22% and 36% of the total web traffic (with a best estimate of 32%). Our objective was to answer the following questions about them:

1. (1)

   Does the website primarily offer SaaS?

2. (2)

   Does the website produce its content? For marketplaces, does the domain owner sell its own products on the site?

3. (3)

   Does the website function as a platform for user-generated content? For e-commerce platforms, does the site allow third-party sellers to list their products? Comments on original content (such as those found on some news websites) were insufficient to classify the site as a platform. We did not consider news aggregators or similar sites that curate content from the Web as platforms since the content is not user-provided.

4. (4)

   Does the website require users to log in before accessing its main content?

5. (5)

   Does the website charge users for access to its main content and features? We ignored charges for additional features and, for marketplaces, the charges imposed on sellers.

6. (6)

   Is the domain owner’s primary business or activity related to the Web, or is a significant portion of their business conducted offline? We considered news websites primarily web-related, but not e-commerce sites that sell their own products.

7. (7)

   Who is the ultimate owner of the domain? In the case of subsidiaries, we identified the ultimate parent company as the domain’s final owner. For partial ownership, we only considered parent companies that owned at least 50% of the subsidiaries.

8. (8)

   Is the ultimate owner of the domain a for-profit organization?

9. (9)

   In which country is the domain’s ultimate owner located?

We present our annotations in Tables 2 and 3 (see Appendix A), and in our repository.²²2https://github.com/cewebbr/web-unpacked/blob/main/data/cleaned/domains-annotated_v03.csv While the analysis of this subset of domains does not guarantee a comprehensive representation of total Web traffic, the trends observed within it are likely to extend to the following several thousand domains, thus reflecting the characteristics of a significant portion of Web usage.

Among the top 116 listed domains, Microsoft is the largest owner with 11 domains. These include linkedin.com, openai.com³³3While we attribute ownership of OpenAI to Microsoft, the precise nature of their relationship remains a topic of debate., bing.com, github.com, and several others associated with Microsoft itself and its SaaS products like office365.com. Following is Amazon, the owner of twitch.tv, imdb.com, and five Amazon-related domains. Alphabet holds ownership of five domains, including youtube.com and various Google domains, while Meta possesses four (facebook.com, instagram.com, whatsapp.com, and messenger.com). The first non-US company in terms of top domains is the Russian VK, which owns dzen.ru, vk.com, mail.ru, and ok.ru, followed by the Japanese SoftBank Group Corp., which owns three domains.

Suppose we rank the final owners by their aggregated traffic. In that case, Alphabet receives 120 billion visits monthly, accounting for almost 50% of the total traffic of the top 116 domains (see Fig. 8). This ranking also roughly follows a power-law distribution, but its best-fit exponent, $\beta$, is more negative than the best-fit for the disaggregated top 116 domains: -1.22 versus -0.98. It is important to note that this decrease in $\beta$ is not an artifact caused by the aggregation itself. Randomly aggregating domains would typically blend famous and less known domains, evening out the traffic and resulting in a less negative $\beta$. Thus, there must be a socio-economic explanation for why the same companies own highly visited domains. Although advancing such an explanation is beyond the scope of this paper, we conjecture that this phenomenon partially stems from large companies acquiring popular websites (e.g., youtube.com, linkedin.com, dzen.ru, twitch.tv, imdb.com, and github.com were independent websites later purchased by their current owners).

When we aggregate the top domains’ visits by the ultimate owner’s country, the concentration becomes even more pronounced. The best-fit power-law exponent reaches $\beta=-2.00$, with the United States capturing 80% of the total traffic of the top 116 domains. This dominance is followed by China (4.5%), Russia (3.7%), Japan (2.6%), the Czech Republic (2.1%), and South Korea (0.97%).

Fig. 9 summarizes the traffic shares associated with the binary answers to the remaining questions (1 to 6 and 8). Each bar in the plot illustrates how the traffic is distributed among domains subjected to a specific binary classification. Notably, the vast majority (97%) of the traffic to the top 116 domains flows to domains owned by profit-seeking companies. Interestingly, among these 116 domains, only two are not-for-profit: wikipedia.org and archiveofourown.org. This observation is intriguing, especially considering that 96% of the top traffic goes to websites that do not charge for access or their main functionalities. This seeming contradiction underscores emerging business models that rely on charging for additional features (e.g., the Freemium model (Georgieva et al., 2015)), growing the user base for a later sell-out (Lee et al., 2006), and operating two-sided markets where users are subsidized to facilitate data collection, advertising, and sales for the other market side (Hermes et al., 2020b). Additionally, the figure reveals that 94% of the visits stream to web-related domains, indicating businesses with no offline counterpart (e.g., search engines, online advertising, marketplaces, social media, SaaS).

Figure 9 emphasizes that 85% of the total traffic to top domains flows to platforms that do not generate their content, such as social media platforms, marketplaces, news aggregators, and search engines. Conversely, 46% of the traffic goes to websites where users themselves create content. Excluding traffic to search engines, which neither produce content nor allow users to post content, the proportion of traffic allocated to user-generated content platforms rises to 77%, with minimal impact on the distribution across other domain categorizations. This large percentage underscores the central role of user-content platforms in the Web’s contemporary landscape.

Finally, the analysis reveals that SaaS does not emerge as a prominent traffic driver for top domains. This fact suggests that Web visitors prioritize content over services, indicating a lesser emphasis on the Web as a software platform. However, as detailed in Section 3.3, notable domains like google.com, yandex.ru, baidu.com, amazon.com, and yahoo.com offer additional services beyond their primary functions, some of which fall under the SaaS category. For instance, approximately 15% of Google’s traffic is directed to subdomains primarily recognized as SaaS (refer to Fig. 7). A similar scenario unfolds regarding the necessity for user authentication to access the Web: while 76% of the traffic to top domains goes to websites categorized as not requiring login credentials, a considerable portion of this traffic flows to domains with subdomains demanding authentication. Thus, at least 24% of this traffic necessitates user authentication.

We employed clustering techniques to categorize the top 116 domains into four groups. We aimed to minimize the average Hamming distances within each group across a 7-dimensional binary feature space constructed from our questionnaire responses. The domains within each cluster exhibit similar characteristics, allowing us to delineate the clusters using the following archetypes:

1. (1)

   SaaS: This group comprises domains primarily characterized by SaaS offerings, devoid of offline activities and requiring user authentication. While some domains charge users for their core services, many do not. Examples include openai.com, zoom.us, office365.com, and canva.com. Our clustering process assigned 12 domains to this group.

2. (2)

   Open content providers: Domains in this cluster serve as content producers, offering access to their content without imposing a paywall or requiring user authentication. Notably, this is the only group to include businesses with offline operations and closely resembles the ethos of the early internet era, often referred to as “Web 1.0.” Examples encompass samsung.com, walmart.com, bbc.co.uk, weather.com, and yahoo.com. Amazon.com was categorized within this group by our clustering algorithm, along with other 31 domains.

3. (3)

   Platforms: This cluster encompasses domains whose activities are confined to the online realm, where users contribute the entirety of the content, and access to the websites is free. Prominent examples comprise youtube.com, roblox.com, chaturbate.com, tiktok.com, wikipedia.org, and booking.com. Interestingly, search engines and messaging web services such as telegram.org and discord.com were assigned to this group by our clustering method. In total, 67 domains were classified under this archetype.

4. (4)

   Subscription content providers: Domains in this category are online content producers that mandate user authentication and levy charges for accessing their content. Examples include nytimes.com, espn.com, netflix.com, and disneyplus.com. Our clustering process identified five domains within this group.

Figure 10 shows that the cluster designated as “Platforms” is the primary destination for traffic among the top 116 domains, even when excluding search engines from consideration. While some traffic directed toward search engines could be labeled as SaaS due to their supplementary services, the preeminence of the Platforms cluster remains unassailable.