Impact of HTTP Cookie Violations in Web Archives

For a long time we have been observing a strange behavior of various web archives when accessing mementos (Van de Sompel et al., 2013) of Twitter pages, some of the mementos would be replayed in non-English languages. This happens even if those Twitter timelines belong to English-speaking personalities, archived using crawlers in North America, and were not requested in any specific language explicitly as shown in Figure 1. After a thorough investigation we figured it out that it is happening due to the use of HTTP Cookies for content negotiation by Twitter (Alam and Vargas, 2018; Rosenthal, 2018). We found that almost half of the mementos of Barack Obama’s Twitter timeline out of over 9,000 properly archived mementos in five different web archives were in non-English languages, of which, almost half were in Kannada (a regional Indian language) alone, and remaining in 45 other languages (as shown in Figure 2). While language diversity in web archives is generally a good thing, this non-uniform bias is disconcerting when a page is archived in a language not anticipated.

One day we were looking at a Twitter timeline’s memento which should have been in English, but was primarily in Portuguese (for the reason described above), after a while we noticed that a notification appeared in Urdu, suggesting that there were 20 new tweets (as shown in Figure 3). On further inspection found that the page contained a sidebar block in English too. Apparently, we were seeing a defaced composite memento of a page that perhaps never existed on the live web. We knew that live-leakage (also known as Zombies) (Brunelle, 2012) and temporal violations (Ainsworth et al., 2015) can cause such malformed memento reconstruction and we also knew their potential prevention techniques (Lerner et al., 2017; Alam et al., 2017). However, this mixed-language Twitter timeline issue cannot be explained by zombies nor temporal violations. After a thorough investigation we found that Cookies were again the reason behind this replay issue (Alam, 2019; Rosenthal, 2019).

HTTP is stateless, but often applications need to maintain state information between a client and a server. This is often done with the help of Cookies (Barth, 2011). Servers can send one or more Set-Cookie headers containing strings of name-value pairs along with scope (domain and path) and expiration information. Clients store them and send them back with each request in the scope using Cookie header until expired or removed. Cookies are used for session management, personalization, content-negotiation, tracking, and client-side key-value store. The latter is less common now after wide adoption of LocalStorage and other similar techniques in web browsers.

Twitter uses standard method of internationalization in its publicly accessible pages by including alternate links in 47 supported languages and the x-default landing language (as illustrated in Figure 4) to help search engines point users with different locales to the correct language. This technique is utilized by many other multi-lingual sites such as Facebook and Instagram. However, unlike other popular multi-lingual sites, when accessing a language-specific URI (that contains a lang query parameter), Twitter sets a lang Cookie with the corresponding language (as illustrated in Figure 5). This Cookie sticks throughout the session and forces all subsequent pages to be served in that language until another language-specific URI overwrites the Cookie. This essentially means Twitter performs language content negotiation using Cookie header, though it does not acknowledge it in a Vary header.

Some websites insist that certain Cookies are present in a request before they return desired content otherwise they issue redirects and attempt to set those Cookies. Failure to send their desired Cookies in subsequent requests may turn such sites into crawler traps without any useful content. Web archiving crawlers such as Heritrix¹¹1https://github.com/internetarchive/heritrix3 have built-in support for cookies. However, the web surfing pattern of crawlers is generally breadth-first-style and comprehensive (not necessarily how human surf the web) for which they use frontier queue of URIs to be crawled. In case of the Twitter’s example above, when one of the language-specific alternate link is crawled, it impacts all the subsequent non-language-specific URIs due to the lang sticky Cookie. Kannada being the last language in the list (in Figure 4) gets more exposure before it gets overwritten by another language, resulting in the disproportionate language bias.

Popular archival replay systems (such as OpenWayback²²2https://github.com/iipc/openwayback and PyWB³³3https://github.com/webrecorder/pywb) utilize only the canonicalized URI-R and the datetime of the capture to select a memento to replay. Other request headers that might have been used for content negotiation (such as Accept-Language or Geolocation etc.) are ignored at replay. Traditional crawlers did not execute JavaScript, so the likelihood of a custom request header being utilized during crawling was minimal, but it is changing with headless browser-based crawlers. Cookies, however, have been supported even in traditional crawlers that are used by some sites for content negotiation (as is the case with Twitter). Moreover, aggregating private archives (Kelly et al., 2018) containing authenticated resources without isolating them based on session Cookies has some privacy implications.

Based on our assessment we propose that Cookies in crawlers are kept short-lived and pruned frequently to minimize the impact of sticky Cookies. Accommodating Cookies (or other headers that affect the response) at capture/crawl time, but not utilizing them at replay time has this consequence of cookie violations, resulting in defaced composite mementos. On the contrary, blindly utilizing every Cookie as a filter at replay would result in many false negatives. Unfortunately, Cookie names are opaque strings and carry no agreed upon semantics to identify ones that affect the payload.

We identified that certain Cookies on certain sites can be a source of content bias in archival crawls. To address this issue we propose that crawlers store Cookies with short expiration time explicitly, irrespective of the original value. We also identified that Cookie Violations at replay time have the potential to deface composite mementos and reconstruct pages from web archives that never existed on the live web. Archival replay systems need to behave like HTTP proxies or cache servers that accommodate values in the Vary header along with URIs. Not every Cookie is created equal, those impacting the content need to be identified and accounted for at replay. This is a difficult problem which opens up the possibility for a more extensive research to fully address the issue.

This work is supported in part by NSF grant IIS-1526700.