SHARI – An Integration of Tools to Visualize the Story of the Day

Tools such as Google News and Flipboard exist to convey daily news, but what about the news of the past? We have combined StoryGraph¹¹1http://storygraph.cs.odu.edu with tools from the Dark and Stormy Archives Toolkit²²2https://oduwsdl.github.io/dsa/software.html to produce the StoryGraph Hypercane ArchiveNow Raintale Integration (SHARI) process. These tools represent disparate research efforts in news analysis, corpus summarization, web archiving, and visualization. The integration produces a summary of the “biggest story” for a given date. SHARI combines the following components from Old Dominion University’s Web Science and Digital Libraries Research Group³³3https://ws-dl.cs.odu.edu:

• •

  StoryGraph: a platform that downloads RSS feeds and analyzes the linked articles to cluster news stories nwala_365_dots_2020 – http://storygraph.cs.odu.edu/

• •

  Hypercane: a framework for intelligently sampling and analyzing documents from web archive collections jones_hypercane_2020 – https://oduwsdl.github.io/hypercane

• •

  ArchiveNow: a library developed by Aturban et al. aturban_archivenow:_2018 that submits live web URI-Rs to web archives to create URI-Ms – https://github.com/oduwsdl/archivenow

• •

  Raintale: a MementoEmbed jones_preview_2018 client that creates stories from a sample of mementos – https://oduwsdl.github.io/raintale

Nwala et al. nwala_bootstrapping_2018 ; nwala_scraping_2018 have focused on finding seeds within search engine result pages (SERPs), social media stories, and news feeds. As part of this research, Nwala et al. also developed StoryGraph nwala_365_dots_2020 , a service that saves RSS feeds from 17 news sources (Table 1 in Appendix A) every ten minutes. With these RSS feeds, StoryGraph analyzes the lexical connections between articles across feeds to generate JSON output, which drives a graph visualization. Figure 1 displays some of this JSON output for March 23, 2020. StoryGraph then visualizes this output, as shown in Figure 2.

Collections on specific topics exist at various web archives jones_many_2018 . AlNoamany et al. alnoamany_generating_2017 introduced how to use social media storytelling to summarize web archive collections. Klein et al. 10.1145/3201064.3201085 have built collections from web archives by conducting focused crawls. Jones developed Hypercane jones_hypercane_2020 to intelligently sample mementos from larger collections. Jones also developed Raintale jones_raintale_2019 for generating social media stories to summarize groups of mementos, providing visualizations that employ familiar techniques, like cards, that require no training for most users to understand.

The JSON data structure from Figure 1 provides all information gathered but is difficult for humans to understand at a glance. The graph shown in Figure 2 provides an overview of the JSON through favicons and edges, but a user requires some training to fully comprehend what it represents. Figure 3 displays the largest connected component from this graph visualized via the SHARI process. Through images, text snippets, titles, cards, domain names, favicons, and other content, the SHARI output allows the viewer to intuitively understand that the biggest news story for this date consists of different reactions to the growing COVID-19 pandemic.

The StoryGraph Hypercane ArchiveNow Raintale Integration (SHARI) jones_shari_2020 process automatically creates stories summarizing news for a day. Figure 4 details what each tool contributes to the story. Figure 5 shows the steps of the SHARI process.

1. 1.

   With the StoryGraph Toolkit, we query the StoryGraph service for the URI-Rs belonging to the biggest story of the day.

2. 2.

   Hypercane converts these URI-Rs to URI-Ms by first attempting to find a corresponding URI-M by querying the LANL Memento Aggregator⁴⁴4https://timetravel.mementoweb.org via the Memento Protocol van_de_sompel_rfc_2013 . For each URI-M that does not have a memento, Hypercane creates a memento by calling ArchiveNow aturban_archivenow:_2018 (Figure 6).

3. 3.

   Hypercane runs the mementos through spaCy⁵⁵5https://spacy.io/ to generate a list of named entities, sorted by frequency (Figure 7).

4. 4.

   Hypercane runs the mementos through sumgram nwala_sumgram_2019 and generates a list of sumgrams, sorted by frequency (Figure 8).

5. 5.

   Hypercane scores all of the mementos’ embedded images. Images that article authors reference in HTML META tags are favored first, followed by MementoEmbed jones_preview_2018 score, then pixel size, color count, the ratio of width to height, and finally position on the page (Figure 9).

6. 6.

   Hypercane runs the mementos through newspaper3k⁶⁶6https://newspaper.readthedocs.io/en/latest/ to extract each article’s publication date and orders the URI-Ms by that date (Figure 10) .

7. 7.

   Hypercane consolidates the entities, terms, image scores, and ordered URI-Ms into a JSON file containing the structured data for the summary. During this step, Hypercane uses the highest scoring image as the striking image for the summary (Figure 11). In Figure 4, the highest-ranking image is the UK Prime Minister addressing his country about the COVID-19 pandemic.

8. 8.

   Raintale renders the output as Jekyll HTML based on the contents of this JSON file, a template file, and information on each memento provided by MementoEmbed (Figure 11).

9. 9.

   The SHARI script publishes the summary story to GitHub Pages for distribution. Figure 13 shows the output of our dsa_tweeter bot which announces the story after publication through the @StormyArchives Twitter account.

StoryGraph is a valuable resource with additional unrealized potential. We are not only able to create stories for today or yesterday but any date back to August 8, 2017, when Nwala launched StoryGraph. As seen in Figures 14, 15, and 16 we can see how the world has evolved each year on StoryGraph’s launch date. In Figure 14, the biggest news story was that of North Korea threatening other nations with nuclear weapons. One year later, in Figure 15, we see that the biggest news story is the results of several United States Congressional and gubernatorial primaries. Two years after StoryGraph’s launch, Figure 16 shows that the biggest news story is the aftermath of the 2019 shootings in El Paso and Dayton.

SHARI produces a familiar yet novel method of viewing news for a given day. SHARI can create stories for today, yesterday, and back to StoryGraph’s creation on August 8, 2017. It is different from other storytelling services like Wakelet⁷⁷7https://wakelet.com/ because SHARI is entirely automated. The stories produced by SHARI are different from services like Google News⁸⁸8https://news.google.com/ or Flipboard⁹⁹9https://flipboard.com/ because those tools focus on current events and personalized topics. Because StoryGraph samples content from multiple sides of the political spectrum, the SHARI process can provide a visualization of articles not tied to one interest area or even a single side’s terminology. This process works because each component is loosely coupled, has high cohesion, has explicit interfaces, and engages in information hiding. Each command passes data in the expected format to the next.

We are also exploring how to improve striking image selection for stories. One could use this to consider how the same story is told in different venues. For instance, one could ask StoryGraph only to include left-leaning sources and produce a SHARI story. One could then do the same for only the right-leaning sources. With both stories, one could compare the striking images and sumgrams that SHARI produces. We are investigating how to produce and render other news stories for a given day and any given period of time. Finally, we are examining how to best visualize significant events that span substantial periods of time, like the entire COVID-19 news story. Though StoryGraph is an existing service that gathers current news, we also want to apply its algorithm directly to mementos and tell the news stories of past events like the Hurricane Katrina disaster.

This work supported in part by the Institute of Museum and Library Services (LG-71-15-0077-15).