Vertical, Temporal, and Horizontal Scaling of Hierarchical Hypersparse GraphBLAS Matrices

Streaming data plays a critical role in protecting computer networks, tracking the spread of diseases, optimizing financial markets, and ad-placement in social media. The volumes of data in these applications is significant and growing. For example, the global Internet is expected to exceed 100 terabytes per second (TB/s) creating significant performance challenges for the monitoring necessary to improve, maintain, and protect the Internet [1, 2, 3, 4, 5]. As in other applications hypersparse matrices are a natural way to represent network traffic and rapidly constructing these traffic matrix databases is a significant productivity, scalability, representation, and performance challenge [6, 7, 8, 9, 10, 11, 12, 13, 14].

The SuiteSparse GraphBLAS library is an OpenMP accelerated C implementation of the GraphBLAS.org sparse matrix standard [15, 16, 17] and makes widely available the benefits of hypersparse capabilities [18, 19, 20]. Python, Julia, and Matlab/Octave bindings allow the performance benefits of the SuiteSparse GraphBLAS C library to be realized in these highly productive programming environments. Our team has developed a high-productivity scalable platform—the MIT SuperCloud—for providing scientists and engineers the tools they need to analyze large-scale dynamic data [21, 22, 23]. The MIT SuperCloud provides interactive analysis capabilities accessible from high level programming environments (Python, Julia, Matlab/Octave) that scale to thousands of processing nodes. MIT SuperCloud maintains a diverse set of hardware running an identical software stack that allows direct comparison of hardware from different eras.

This paper builds on our prior work [24] with hierarchical hypersparse GraphBLAS matrices by more thoroughly exploring the vertical, temporal, and horizontal scaling performance of hierarchical hypersparse construction. The vertical benchmarking explores how different numbers of GraphBLAS processes and threads perform on different multicore compute nodes. The temporal benchmarking compares the performance of compute nodes from different eras. The horizontal benchmarking examines the aggregate performance achieved by running on thousands of diverse compute nodes.

Streaming updates to a large hypersparse matrix can be accelerated with a hierarchical implementation optimized to the memory hierarchy (see Fig. 1). Rapid updates are performed on the smallest hypersparse matrices in the fastest memory. The strong mathematical properties of the GraphBLAS allow a hierarchical implementation of hypersparse matrices to be implemented via simple addition. All creation and organization of hypersparse row and column indices are handled naturally by the GraphBLAS mathematics. Hierarchical matrices are implemented with a simple heuristic. If the number of nonzero (nnz) entries in the matrix ${\bf A}_{i}$ at layer $i$ exceeds the cut threshold $c_{i}$, then ${\bf A}_{i}$ is added to ${\bf A}_{i+1}$ and ${\bf A}_{i}$ is cleared. The overall usage is as follows

• •

  Initialize $N$-level hierarchical hypersparse matrix with cuts $c_{i}$

• •

  Update by adding new data ${\bf A}$ to lowest layer

  $${\bf A}_{1}={\bf A}_{1}+{\bf A}$$

• •

  If ${\rm nnz}({\bf A}_{1})>c_{1}$, then

  $${\bf A}_{2}={\bf A}_{2}+{\bf A}_{1}$$

  and reset ${\bf A}_{1}$ to an empty hypersparse matrix of appropriate dimensions.

Ai{1} = Ai{1} + A;

for i=1:length(c)

if (GrB.entries(Ai{i}) > c(i))

Ai{i+1} = Ai{i+1} + Ai{i};

Ai{i} = Ai{length(c)+2};

end

end

The GraphBLAS benchmarking code was implemented using the Matlab/Octave bindings with the pMatlab parallel library [26]. A typical run could be launched in a few seconds using the MIT SuperCloud triples-mode hierarchical launching system [23]. The launch parameters were [Nnodes Nprocess Nthread], corresponding to Nnodes nodes, Nprocess Matlab/Octave processes per node, and Nthread OpenMP threads per process. On each node, each of the Nprocess processes and their corresponding Nthread threads were pinned to adjacent cores to minimize interprocess contention and maximize cache locality for the GraphBLAS OpenMP threads [27]. Within each Matlab/Octave process, the underlying GraphBLAS OpenMP parallelism is used. At the end of the processing the results were aggregated using asynchronous file-based messaging [28]. Triples mode makes it easy to explore horizontal scaling across nodes, vertical scaling by examining combinations of processes and threads on a node, and temporal scaling by running on diverse hardware from different eras.

The computing hardware consists of five different types of nodes acquired over a decade (see Table I). The nodes are all multicore x86 compatible with comparable total memory. The MIT SuperCloud maintains the same modern software across all nodes, which allows for direct comparison of hardware performance differences.

The performance of a hierarchical GraphBLAS for any particular problem is determined by the number of layers $N$ and the cut values $c_{i}$. The parameters are tuned to achieve optimal performance for a given problem. These parameters were explored using simulated Graph500.org R-Mat power-law network data containing 100,000,000 connections that are inserted in groups of 100,000. Examples of different sets of cut values with different $c_{1}$ and different ratios between cut values are shown in Figure 2. These sets of cut values allow exploration of the update performance of closely spaced cuts versus widely spaced cuts. For both $N$ and $c_{i}$ there are fairly broad ranges of comparable optimal performance.

Vertical benchmarking explores how different numbers of GraphBLAS processes and threads perform on different multicore compute nodes. This type of benchmarking is generally useful in most projects as it allows the determination of the best combination processing and threads prior to significant computation. The number of processes and threads used for a given benchmark is denoted by Nprocess $\times$ Nthreads whose product is equal to total number threads used in the computation. The GraphBLAS computation described in the previous section was repeated for two sets of parameters: single-process and multi-process. In the single-process case the parameters tested are

In the multi-process case the parameters tested are

Figure 3 shows the single node performance using different numbers of processes and threads for the different servers listed in Table I. In all cases, the multi-process scaling provided greater aggregate performance and the single-process scaling provided a maximum of a 4x speedup over $1{\times}1$ case.

Temporal benchmarking compares the performance of computing hardware from different eras. The MIT SuperCloud provides a unique ability to directly compare hardware from different eras using exactly the same modern software stack. Figure 4 shows the single core, best single-process, and best multi-process performance taken from Figure 3 and plotted versus the year the computing hardware became available. These data show a 2x increase in single-core performance, a 3x increase in single process performance, and a 5x increase in single node performance.

Horizontal benchmarking examines the aggregate performance achieved by running on thousands of diverse compute nodes. The scalability of the hierarchical hypersparse matrices are tested using a power-law graph of 100,000,000 entries divided up into 1,000 sets of 100,000 entries. These data were then simultaneously loaded and updated using a varying number of processes on varying number of nodes on the MIT SuperCloud. The horizontal scaling for each type of hardware is shown in Figure 5. Running on nearly two thousand MIT SuperCloud nodes simultaneously achieved a sustained update rate of over 200,000,000,000 updates per second and is shown in Figure 5 with other technologies ranging such as in-memory index stores (D4M), NoSQL (Accumulo), NewSQL (SciDB, CrateDB), and SQL (Oracle) databases. The achieved update rate of over 200,000,000,000 updates per second is significantly larger than the rate in prior published results [24]. This capability allows the MIT SuperCloud to analyze extremely large streaming network data sets.

A variety of network, health, finance, and social applications can be enabled by hypersparse GraphBLAS matrices. Hierarchical hypersparse GraphBLAS matrices enable rapid streaming updates while preserving algebraic analytic power and convenience. The rate of these updates sets the bounds on performance in many contexts. The GraphBLAS readily enable performance experiments on simultaneous diverse hardware via high-level language bindings to Matlab/Octave, Python, and Julia. A peak single process performance was measured at over 4,000,000 updates per second. The peak single node performance measured was 170,000,000 updates per second. The diverse MIT SuperCloud hardware used spans nearly a decade shows a 2x increase in single-core performance, a 3x increase in single process performance, and a 5x increase in single node performance over this era. Running on nearly two thousand MIT SuperCloud nodes simultaneously achieved a sustained update rate of over 200,000,000,000 updates per second. Hierarchical hypersparse GraphBLAS allows the MIT SuperCloud to analyze extremely large streaming network data sets.

The authors wish to acknowledge the GraphBLAS community and the following individuals for their contributions and support: Bob Bond, Alan Edelman, Nathan Frey, Jeff Gottschalk, Tucker Hamilton, Chris Hill, Hayden Jananthan, Mike Kanaan, Tim Kraska, Charles Leiserson, Dave Martinez, Mimi McClure, Joseph McDonald, Christian Prothmann, John Radovan, Steve Rejto, Daniela Rus, Allan Vanterpool, Matthew Weiss, Marc Zissman.