Towards Aggregated Asynchronous Checkpointing

The first strategy we implement is a POSIX-based aggregation strategy that aims to highlight the bottlenecks during asynchronous flushing. To this end, we leverage the multi-level checkpointing support introduced by VELOC (Nicolae et al., 2021): (1) each process writes its checkpoint in a blocking fashion to a file on node-local storage, e.g. in-memory or to an SSD (denoted the local phase); (2) once files are written locally, the process continues with application computation and a separate asynchronous post-processing engine (denoted the active backend in VELOC) running on each compute node writes the node-local files (generated by processes co-located on the same node) to an offset in a shared file located on the PFS (denoted the flush phase). The offset of each node-local checkpoint in the remote file is calculated using a prefix-sum algorithm over all participating processes. Note that with this strategy, the active backend running on each node can parallelize writing the local checkpoints using a number of I/O threads that can be used to match the desired trade-off between application slowdown and flushing duration. This can be configured for each active backend independently.

Thanks to its simplicity, the POSIX-based aggregation strategy incurs minimal preparation and synchronization overheads, much like VELOC’s default asynchronous strategy that writes each checkpoint to a different file on the PFS (i.e. no intercommunication between processes during the flush phase). Therefore, we can conclude that this strategy is unlikely to cause more interference or slowdown in the presence of an application. However, the flushing duration is significantly higher compared to the default VELOC and GenericIO. This extended flush time is caused by false sharing: to facilitate parallel processing, files are striped on the PFS into chunks that are stored on different I/O servers. If the writes of local checkpoints do not align to the stripe sizes, then the writes will compete for the same stripe, despite using non-overlapping file regions. As a consequence, more advanced POSIX-based aggregation strategies need to address the negative impact of false sharing.

A second strategy we implement on top of VELOC is based on MPI-IO, which is also used by GenericIO and was optimized for synchronous aggregation. Specifically, the same prefix-sum algorithm is applied to determine the offset of each node-local checkpoint in the larger file on the PFS, then the active backend issues a collective MPI-IO write at operation (instead of independent POSIX writes). The underlying collective write implementation is based on two key observations: (1) the number of processes (MPI ranks) is typically much larger than the number of I/O servers, therefore it is suboptimal to issue more concurrent I/O writes than there are I/O servers available to execute them; (2) only one MPI rank is allowed to write to a PFS stripe due to false sharing. Based on these observations, MPI-IO implements optimizations such as gathering the data from multiple MPI ranks on I/O leaders, which are then responsible for concurrently writing files on the PFS. The number of I/O leaders can be adjusted to match the number of I/O servers. Furthermore, each leader can be assigned a set of stripes that is disjoint from the sets of stripes of all other leaders, thereby eliminating false sharing.

While more advanced, an MPI-IO based aggregation strategy has limited applicability in the area of asynchronous checkpointing for several reasons. First, it introduces high communication overheads which is needed to gather the data from multiple processes to the I/O leaders. While this does not cause any interference in synchronous mode (because the application is blocked during checkpointing), it can become a bottleneck in asynchronous mode, because the application may need to perform its own communications during the gathering phase, thereby competing for network bandwidth. Second, since it is based on a collective operation, it assumes all MPI ranks need to participate in the checkpointing process and all data needs to be ready simultaneously in all MPI ranks, introducing synchronization during the flush phase. However, a key feature of multi-level checkpointing strategies is that the MPI ranks write their checkpoints to node-local storage and flush to the PFS independent of the progress or state of other participating processes, then continue executing the application code.

Thus, it is up to the active backends to self-organize in order to optimize the aggregation asynchronously. One potential solution is to perform a collective MPI-IO write on all active backends, where each active backend contributes with its set of node-local checkpoints. However, such a strategy introduces two bottlenecks: (1) the MPI-IO standard does not allow more than a single contiguous memory region to be written to a file, whereas each active backend may need to contribute with multiple such regions (mapped to independent node-local checkpoint files); (2) even if such a MPI-IO collective write were implemented, it places an unnecessary restriction on the active backends to be ready at the same time to facilitate communication and data exchange, which increases the initialization overheads. To address these issues, we propose a multi-phase solution that invokes multiple successive MPI-IO collective write calls, one for each node-local checkpoint.

We evaluate the two strategies introduced in Section 2.1 and Section 2.2 in a series of experiments that involve a micro-benchmark running on an increasing number of MPI ranks and an increasing number of compute nodes. Each MPI rank writes a checkpoint that is 1 GiB large. The testbed used for our experiments is Argonne National Lab’s Theta supercomputer, a Cray XC40 11.60 Petaflops system, equipped with a Lustre parallel file system  (Nicolae et al., 2021, 2019). We compare the two strategies with two additional baseline approaches: the default VELOC multi-level asynchronous strategy (one file per MPI rank) and a synchronous I/O aggregation strategy using GenericIO (GIO) (Habib et al., 2016). The aggregation performed by all approaches is $N\to 1$).

Figure  1 shows the throughput during the local checkpoint phase for the aforementioned checkpoint strategies. Since VELOC is writing to local storage (SSD or memory hierarchy), it is orders of magnitudes faster than GenericIO, which is writing directly to the PFS. Note that all three VELOC strategies exhibit similar throughput, which means the prefix sum employed by the aggregation approaches introduces a negligible overhead during the local checkpointing phase.

Figure 2 shows the throughput for the compared strategies during the flush phase. As can be observed, both the POSIX and MPI-IO based strategies show a significantly lower throughput compared with the original one file per MPI rank strategy, which can be attributed to false sharing in the case of the former, and, respectively, to the fact that we invoke multiple MPI-IO collective calls, which perform suboptimally.

Thus, we can conclude that a simple aggregation strategy based on POSIX or MPI-IO is not sufficient and there is significant room for improvement to reach and potentially even surpass the default, embarrassingly parallel one-file per MPI rank flush strategy.