Async-fork: Mitigating Query Latency Spikes Incurred by the Fork-based Snapshot Mechanism from the OS Level

We first briefly review two operating system concepts, virtual memory and fork that are closely related to this work. As our technique is implemented and deployed in Linux, we introduce these concepts in the context of Linux. Then, we discuss the use cases of fork in databases.

Virtual Memory. As an effective approach to managing hardware memory resources, virtual memory is widely used in modern operating systems. A process has its own virtual memory space, which is organized into a set of virtual memory areas (VMA). Each VMA describes a continuous area in the virtual memory space. The page table is the data structure used to map the virtual memory space to the physical memory. It consists of a collection of page table entries (PTE), each of which maintains the virtual-to-physical address translation information and access permissions. A VMA corresponds to multiple PTEs.

Figure 2 shows an example of the page table. To reduce the memory cost, the page table is stored as a multi-level radix tree in which PTEs locate in leaf nodes (i.e., PTE Level in Figure 2). The part in the area marked with the dashed line corresponds to VMA₀. The tree at most has five levels. From top to bottom, they are the page global directory (PGD) level, the P4D level, the page upper directory (PUD) level, the page middle directory (PMD) level and the PTE level. As P4D is generally disabled, we focus on the other four levels in this paper. Except for the PTE level, an entry stores the physical address of a page while this page is used as the next-level node (table). With the page size setting to 4KB, a table in each level contains 512 entries. Given a VMA, “VMA’s PTEs” refers to the PTEs corresponding to the VMA and “VMA’s PMDs” is the set of PMD entries that are parents of these PTEs in the tree. For example, PMD₀ and PMD₁ belong to VMA₀’s PMDs in Figure 2.

Fork Operation. Fork is a system call that creates a new process by duplicating the calling process (for, [n.d.]). Both processes have separate memory spaces. The new (resp. calling) process is called the child (resp. parent) process. To accelerate the operation, Linux implements fork with the copy-on-write (CoW) strategy. Specifically, while invoking fork, the parent process runs in the kernel mode and copies the metadata (e.g, VMAs, the page table, file descriptors, and signals) to the child process. The PTEs of both parent and child are set to write-protected. After that, the process that first modifies a write-protected page triggers a page fault, which leads to the copy of the page. In a word, benefiting from CoW, fork copies the metadata only.

Use Cases of fork in Database. Fork has a number of database use cases because it can easily and efficiently create a snapshot of in-memory data, the consistency of which is guaranteed by the OS. In general, these cases can be categorized into two classes based on the usage of the snapshot. First, use fork to delegate dedicated tasks, which have expensive IO or computation costs, to a child process without blocking the service of the parent process. MDC (Park et al., 2020) uses fork to record checkpoints for in-memory databases. Redis uses fork to conduct log rewriting (RedisLab, 2021e) that optimizes the Append Only File (AOF). FlurryDB (Mior and de Lara, 2011) proposes to create replica based on fork in distributed environments. Second, use fork to create snapshots to support concurrent transaction processing because fork provides the snapshot isolation between processes. HyPer (Kemper and Neumann, 2011) proposes to evaluate hybrid OLTP and OLAP queries based on snapshots created by fork. AnKer (Sharma et al., 2018) designs a fine-grained snapshot mechanism to support MVCC. In particular, AnKer takes a partial snapshot of in-memory data by co-designing the database engine and the system call fork. Different from our research on accelerating the fork operation from the OS level, AnKer focuses on optimizing which in-memory data should be captured by the snapshot.

All these use cases can potentially benefit from Async-fork because 1) they can encounter the query latency spike problem incurred by the fork operation; and 2) Async-fork can accelerate the snapshot creation. This paper focuses on the scenario that uses fork to take a point-in-time snapshot of in-memory data to persist the data (RedisLab, 2021e; Inc., 2022c). In particular, the storage engine calls fork to create a child process that holds the same data as it. Then, the child process writes the data to the hard drive, while the storage engine can continue to serve users’ queries. Although the storage engine delegates the data dump task to the child process, it will be out-of-service for queries during the invocation of fork because it runs into the kernel mode. We are particularly interested in this scenario because the IMKVS is one of the most important services in public cloud and popular IMKVSes (e.g., Redis and KeyDB) (solid IT, 2022) use this mechanism to persist data. Consequently, these stores encounter serious a query latency spike problem (see Section 3), while they are generally used in mission-critical applications that have a rigid latency constraint. We also evaluate the effectiveness of Async-fork on log rewriting in Redis (RedisLab, 2021e). The experiment results are presented in the technical report (tec, 2022).

Remarks. Instead of developing a general-purpose solution to replace the default fork in the OS, the goal of Async-fork is to provide an efficient fork operation for the scenarios (especially for IMKVSes) where 1) the applications are memory-intensive, and 2) the parent process is latency-sensitive. Our design allows Async-fork and the default fork to run in parallel in the OS. Users can easily choose the fork method used in applications (see Section 5.2).

Consistent snapshot is essential for in-memory databases to support backup and disaster recovery (Zhang et al., 2015; Li et al., 2018; Diaconu et al., 2013; Antirez, [n.d.]). Some consistent snapshot mechanisms have been proposed to trade off throughput, latency, and memory footprint (Bronevetsky et al., 2006; Cao, 2013; Cao et al., 2011; Liedes and Wolski, 2006; Li et al., 2018). Naive snapshot (Bronevetsky et al., 2006) blocked the storage engine until a deep copy of all the in-memory data is created, which is not suitable for IMKVSes in which the latency is critical. There are also some non-blocking snapshot mechanisms. Copy-on-Update (Liedes and Wolski, 2006; Cao, 2013) proposed to create a shadow copy of the in-memory data; the storage engine is free to access any data but create a deep copy when updates it for the first time. Note that, the fork-based snapshot is a Copy-on-Update variant that leverages the operating system. Some other mechanisms used multi-version concurrency control (MVCC) (Bernstein et al., 1987) to keep multiple versions of in-memory data. Zigzag (Cao et al., 2011) maintained another untouched copy of the in-memory data and introduced metadata bits to indicate which copy the store engine should read from or write to. Based on Zigzag, Ping-Pong (Cao et al., 2011) maintained three versions of the data to lower the cost of managing metadata bits. Hourglass and Piggyback (Li et al., 2018) were developed by combining Zigzag and Ping-Pong.

Although fork-based snapshot results in long latency during snapshot process, popular industrial IMKVSes (Redis and KeyDB) still adopts fork-based snapshot for two reasons: 1) fork provides a simple engineering implementation for consistent snapshot, while it requires great efforts to integrate the above approaches into the IMKVS. 2) None of the above approaches completely outperform the fork-based snapshot in write-intensive workloads (Li et al., 2018). For example, Ping-Pong and Hourglass can mitigate the latency spikes, while Ping-Pong incurs 3x memory footprint and Hourglass results in higher latency during normal operation. This work resolves the latency spikes of the fork-based snapshot, while keeping its original superiorities.

Previous work (Liedes and Wolski, 2006; Park et al., 2020) noted that the memory footprint increases during snapshot process due to the CoW strategy. MDC (Park et al., 2020) proposed to release the pages that have been persisted as soon as possible. AnKer (Sharma et al., 2018) introduced a fine-grained version of fork to take partial snapshot when databases do not need to persist all the data. CCoW (Ha and Kim, 2022) optimized the CoW mechanism based on the spatial locality of memory access. It prioritizes the copy for high-locality memory regions to improve the performance on write-intensive workloads. Async-fork is orthogonal and complementary to them.

There is also other approach to persist data in IMKVSes. Redis and KeyDB use Append Only File (AOF) (RedisLab, 2021e; Inc., 2022c) to log every write operation received by the storage engine, that will be played again after the database reboots to reconstruct the original dataset. The snapshot and AOF are complementary, and it is recommended to enable both of them simultaneously in IMKVSes.

Figure 3 presents the execution time of fork and the time of copying the page table in the fork operation. We can see that the execution time grows roughly linearly with the instance size increasing from 1GB to 64GB. The 1GB instance takes less than 10 ms, while the 64GB instance takes more than 600 ms. We can also see that the time of the page table copy dominates the execution time. Particularly, the copy operation takes over 97% percentage of the execution time on all test cases and up to 99.93% percentage on the 64 GB instance. Without loss of generality, we measure the detailed metrics of copying the page table on the 8 GB instance to further study the copy operation.

In the experiment, the page table of the 8GB instance has one PGD entry, eight PUDs, $2^{12}$ PMDs and $2^{21}$ PTEs. Overall, fork copies the table level-by-level from top to bottom along the radix tree. The copy of one PGD/PUD/PMD entry requires to apply for a page to store its children and initialize the page. The operation takes around 500 ns. Thus, the copy of the $2^{12}$ PMDs takes around 2 ms, while the overhead of copying PGDs and PUDs is trivial because there are only a few entries. The rest time (around 70 ms) is spent on copying PTEs. Based on the results, we have the following observation.

Observation 1. For the fork operation, the execution time dramatically grows with the instance size increasing, and the page table copy dominates the cost. For the page table copy, the overhead of copying PGDs and PUDs is trivial, while that of copying PMDs and PTEs is non-negligible.

We reveal the impact of the fork operation on the query latency in this subsection. Before presenting the results, we first introduce existing optimization approaches to the fork operation.

Huge Page (Community, 2021a, b). In the operating system (OS), we can increase the page size to reduce the number of pages used by a process, for example, setting the page size to 2MB instead of 4KB. The large page size can reduce the number of PTEs and accelerate the fork operation. However, previous study found that the page fault latency can increase from 3.6$\mu s$ to 378$\mu s$ after enabling huge page because compacting and zeroing memory in the page fault incurs higher overhead on huge page than that on regular page (Kwon et al., 2016). Moreover, Redis consumes much more memory space after enabling the huge page because applications do not always fully utilize the big page (e.g., in the experiment of (Kwon et al., 2016), the memory consumption of Redis increased from 12.2GB to 20.7GB). Additionally, allocating huge page can lead to many fragments in the physical memory because it requires consecutive physical memory areas to build a huge page. Consequently, the kernel needs to perform a heavy defragment operation which leads to high CPU utilization (Panwar et al., 2018). The experiment of (Panwar et al., 2018) shows that the benchmark milc in SPEC CPU 2006 spends 343 seconds (37% of its overall execution time) in kernel mode to perform the defragment operation when the memory is highly fragmented.

Consequently, huge page is recommended to be disabled in many databases (e.g., Couchbase, MongoDB, KeyDB and Redis (Couchbase, 2021; Mongodb, 2021; Inc., 2022d; RedisLab, 2021d)). In particular, two No-SQL databases, Couchbase and MongoDB, recommend users to disable the technique because huge page performs poorly with random memory accesses in workloads (Couchbase, 2021; Mongodb, 2021). KeyDB and Redis, which are two IMKVSes, recommend users to disable huge page because the technique can incur a big latency penalty and big memory usage (Inc., 2022d; RedisLab, 2021d). Specifically, if we enable the huge page, the parent and child processes share huge pages after calling fork to persist on disk. In a busy instance, a few event loops in either of the two processes will cause to target thousands of pages and trigger the copy operation of a large amount of process memory because of the copy-on-write mechanism in OS. Consequently, this leads to a big latency and a big memory usage. As such, our profiling does not involve this technique.

Shared Page Table (McCracken, 2003; Dong et al., 2016; Zhao et al., 2021). This technique proposed to share the page table between parent and child in a copy-on-write (CoW) manner. Specifically, the fork operation returns immediately after copying the metadata except for the page table. The page table will be copied in a CoW manner. However, we find that the shared page table design introduces the data leakage problem, the working set size estimation problem and the NUMA problem. First, the inconsistency between the shared page table and the translation lookaside buffer (TLB) of the child process can lead to the data leakage problem. Second, we cannot accurately estimate the working set size, which indicates the memory usage of each process and is important for cloud resource management (Zhang et al., 2020). This is because the usage is calculated based on the states in the page table (Gregg, 2018), while the table is shared by multiple processes. Third, the shared page table and the corresponding processes can locate on different NUMA nodes, which increases the TLB miss overhead (Achermann et al., 2020; Panwar et al., 2021). Moreover, the NUMA balance mechanism cannot work as expected due to the shared page table. Due to space limit, we discuss the working set size problem and the NUMA problem in the technical report (tec, 2022). In the following, we use the example in Table 1 to demonstrate the data leakage problem, which is the most serious one among the three problems. We also write a test program²²2 https://doi.org/10.5281/zenodo.7189585, Last accessed on 2022/11/13. to reproduce the example in practice.

TLB is the hardware to accelerate the translation from a virtual address to a physical address by caching recent translation results. Initially, the virtual address “V” is mapped to the physical address “X” in PTE and the mapping is cached in TLB. Note that PTE is shared between parent and child, whereas the two processes have their own TLB entries. Suppose that the memory management mechanism (e.g., memory compaction (Babka, 2016; Torvalds, 2018a), swap (Torvalds, 2018c) and NUMA balance (Torvalds, 2018b)) of OS mitigates the page from “X” to a new page frame “Y” in the parent process. Then, OS sets the mapping from “V-¿X” to “V-¿N” (None Present) to indicate that the mapping is invalid in Step 2 and flushes the parent’s TLB entry in Step 3. For other processes, the OS loops over each of them to check whether its PTEs contain ”V-¿X”; if so then set the value to ”V-¿N” and flush the TLB; otherwise, skip the process. This works well if each process has a private page table. However, ODF uses the shared page table design. When the OS checks the child process, the PTE has been set to ”V-¿N” in the parent process because the PTE is shared between the parent and child processes and therefore the OS cannot find any PTE with the value ”V-¿X” in the child process. Thus, the OS skips the update to the child in Step 4. In Step 5, the parent updates PTE to map “V” to “Y”. Although PTE has the correct mapping, the child’s TLB entry is inconsistent with PTE. Consequently, the future access to “V” in the child can lead to a data leakage problem. Despite that, we study the performance of the shared page table-based fork in our experiments for comparison purposes. As ODF (Zhao et al., 2021) is the latest work and the only one that is publicly available³³3 https://github.com/rssys/on-demand-fork, Last accessed on 2022/11/13. among the methods (McCracken, 2003; Dong et al., 2016; Zhao et al., 2021) adopting the shared page table design, our experiments focus on ODF.

Experiment Results. Figures 4 and 5 present the 99%-ile latency and the maximum latency of normal queries and snapshot queries, respectively. Snapshot-ODF denotes the results of using the On-Demand-Fork (Zhao et al., 2021) to take the snapshot, which is the latest shared page table-based fork method. Snapshot-DEF denotes the results of using the default fork in Linux. As shown in the figures, the latency of normal queries slightly increases with the instance size varying from 1GB to 64GB. In contrast, the value of snapshot queries grows sharply. The shared page table technique dramatically reduces the latency of the default fork, especially for the large instance. For example, on the 64GB instance size, the optimization reduces the 99%-ile latency from 911.95ms to 3.96ms and the maximum latency from 1204.78ms to 59.28ms. The latency of Snapshot-ODF is higher than that of Normal because the CoW of the page table frequently interrupts the engine process. For example, the engine process is interrupted over 7000 times on the 16GB instance, which leads to frequent out-of-service for queries. According to the analysis of existing optimization methods and the experiment results, we have the following observation.

Observation 2. The fork operation has a significant impact on the latency of snapshot queries, and the tail latency of the default function is up to hundreds of milliseconds. Although the shared page table technique reduces the latency of the default fork operation, the overhead incurred by frequent interruptions is non-negligible and the shared page table introduces the potential data leakage problem.

Based on the observations, we find that copying the page table dominates the execution time of the default fork in Linux, especially, for large instances. The overhead of fork results in a long latency (up to hundreds of milliseconds) for queries issued during the invocation of fork. Although several optimization methods (Community, 2021a, b; McCracken, 2003; Dong et al., 2016; Zhao et al., 2021) of the fork operation have been proposed, they either lead to poor performance of IMKVSes, or have a data leakage problem, which potentially generates an inconsistent snapshot. Thus, these optimizations cannot be used in the fork-based snapshot mechanism in IMKVSes. As IMKVSes have a rigid requirement on the latency to serve online scenarios, a high-performance fork is required to reduce the long latency of snapshot queries.

Figure 1(c) shows the general idea of Async-fork. While the parent process is responsible to copy page tables during snapshot in the default fork, as shown in the figure, Async-fork offloads the work of copying page table to the idle child process while keeping other steps in the fork unchanged. In this way, the parent process is able to handle queries while the child process copies the page table from the parent process simultaneously. However, it is non-trivial to achieve the design of Async-fork, as a snapshot may be inconsistent due to the asynchronous operations on the page table.

The inconsistency happens when the parent process modifies a PTE before the child process has copied it. Take Figure 6 as an example. The IMKVS takes a snapshot at time T₀, and the in-memory data is $\{(k_{0},v_{0}),(k_{1},v_{1})\}$. Suppose a user query that sets a new KV pair $(k_{2},v_{2})$ arrives at time T₂ (① in Figure 6), and the child process is copying PTEs from time T₁ to T₃. The parent process handles this query, and PTE₂ is modified to point to a new page (Page₂) that contains $(k_{2},v_{2})$. If the child process has not copied the original PTE₂ before the modification, it would copy the modified PTE₂, and owns the new pair $(k_{2},v_{2})$ in its memory space. In this way, the data $(k_{2},v_{2})$ is persisted, and inconsistency happens (the key-value pair $(k_{2},v_{2})$ does not exist when the snapshot is taken at time $T_{0}$).

Async-fork resolves the above inconsistency problem by using the parent process to proactively synchronize the modified PTEs to the child process. We explain the detailed steps in Section 4.2. Two main challenges have to be resolved in this solution.

Firstly, it is necessary to detect all the PTE modifications. However, besides of the user operations, many inherent memory management operations in the operating system also cause PTE modifications. For instance, the OS periodically migrates pages among NUMA nodes (Torvalds, 2018b), causing the involved PTEs to be modified as inaccessible (② in Figure 6). We describe the method to detect the PTE modifications in Section 4.3.

Secondly, errors may occur during Async-fork. For instance, the child process may fail to copy an entry due to out of memory (③ in Figure 6). In this case, error handling is necessary, as we should restore the process to the state before it calls Async-fork. We present how errors are handled in Section 4.4.

By resolving the above challenges, the time of the parent process used on fork is greatly reduced, and the latency of the snapshot queries can be reduced in consequence. Moreover, after the child process finishes copying the page table, the two processes have their complete private page table. Therefore, the Async-fork does not introduce the data leakage vulnerability.

In the following sections, we use the terminology in x86 Linux to explain our design. We also implement Async-fork on Arm Linux using similar ideas, while similar results are achieved compared with x86. Algorithm 1 describes how the parent process (Line 1 to 14) and the child process (Line 15 to 24) work in Async-fork.

Before introducing the proactive synchronization in detail, we first introduce how we offload the work of copying page table to the child process in Async-fork.

Copying Page Table Asynchronously. In the default fork, the parent process traverses all its VMAs and copies the corresponding parts of the page table to the child process. The page table is copied from top to bottom. In Async-fork, the parent process roughly follows the above process, but only copies PGD, P4D (if exists) and PUD entries to the child process (Lines 1 to 3 in Algorithm 1). After that, the child process starts to run, and the parent process returns to the user mode to handle queries (Lines 7 to 14). The child process then traverses the VMAs, and copies PMD entries and PTEs from the parent process. Figure 7 shows an example of the asynchronous page table copy. In the figure, the PGD/PUD entries have been copied by the parent process, but PMD entries and PTEs have not yet (some PMD entries point to “null”). The child process copies PMD entries and PTEs from the parent process (e.g., PMD₀ and its PTEs).

We offload the work of copying PMD entries and PTEs to the child process because the overhead of copying them is non-negligible, as analyzed in Section 3.1. Meanwhile, we keep that the parent process copies PGD/PUD entries. This is because the overhead of copying PGD/PUD entries is trivial, and it is more robust to minimize the change to the Linux kernel.

Synchronizing Modified PTEs Proactively. When the child process is responsible for copying PMD entries and PTEs, it is possible that the PTEs are modified by the parent process before they are actually copied. Note that only the parent process is aware of the modifications (the way to detect the PTE modification will be introduced in Section 4.3).

In general, there are two ways to copy the to-be-modified PTEs for the consistency. 1) the parent process proactively copies the PTEs to the child process; 2) the parent process notifies the child process to copy the PTEs and waits until the copying is finished. As both ways result in the same interruption in the parent process, we choose the former way (Lines 8 to 12 in Algorithm 1).

More specifically, when a PTE is modified during snapshot, the parent process copies not only this PTE but also all the other PTEs of a same PTE table (512 PTEs in total), as well as the parent PMD entry to the child process proactively. For instance, when PTE₁ in Figure 7 is modified, the parent process proactively copies PMD₂, PTE₀ and PTE₁ to the child process. We choose to copy the entire PTE table because we can quickly detect a range of PTEs that will be modified, but accurately identifying which one will be modified is expensive in practice.

Eliminating Unnecessary Synchronizations. Always letting the parent process copy the modified PTEs is unnecessary, as it is possible that the to-be-modified PTEs have already been copied by the child process. We identify if the PMD entries and PTEs have been copied by the child process, to avoid unnecessary synchronizations. A flag is required to track this status. We reuse the R/W flag of the PMD entry to record the status. Since the R/W flag is only used when the PMD entry points to a huge page in the x86 Linux kernel, the flag tracks the status correctly. We reuse the R/W flag because 1) this design can avoid adding new fields to the kernel data structures; and 2) the popular databases (e.g., Redis, KeyDB, MongoDB, and Couchbase) recommend disabling the huge page to improve the performance (RedisLab, 2021d; Inc., 2022d; Couchbase, 2021; Mongodb, 2021). Additionally, it is unnecessary to use Async-fork if applications use the huge page because the applications with huge page do not require PTEs but a small number of PGD/PUD/PMD entries (i.e., the page table is small).

An alternative approach is to use an unused bit in the struct page as the flag. Specifically, each PMD entry points to a PTE table, while OS maintains a data structure (struct page) for each PTE table. The struct page has bits that are not used by the current Linux kernel. Previous research uses these bits as flags, for example, ODF uses some bits in struct page as a reference counter. However, this approach requires further modification to the kernel to initialize the bit. Therefore, we do not adopt the design using the struct page.

If a PMD entry and its 512 PTEs have not been copied to the child process, the PMD entry will be set as write-protected (e.g., PMD₁ in Figure 7). Note that, it does not break the CoW strategy of fork since it still triggers the page fault when the corresponding page is written on x86 (Guide, 2011). Once the PMD/PTEs have been copied to the child process (e.g., PMD₀), the PMD entry is changed to be writable (the PTEs are changed to be write-protected to maintain the CoW strategy). Since both parent and child processes lock the page of the PTE table with trylock_page() when they are copying PMD entries and PTEs, they will not copy PTEs pointed by the same PMD entry at the same time.

The operations that modify PTEs in the OS can be divided into two categories: 1) VMA-wide modification. Some operations act on specific VMAs, including creating, merging, deleting VMAs and so on. The modification of a VMA may also cause the VMA’s PTEs be modified. For example, the user sends queries to delete lots of KV pairs. The IMKVS (parent process) then reclaims the corresponding virtual memory space by munmap. Some VMAs are hence split or deleted while the VMAs’ PTEs are deleted as well. A VMA is usually large because the operating system always tends to merge adjacent VMAs. It means that the VMA-wide modification usually causes extensive PTE modifications, while many VMA’s PMD entries are involved. 2) PMD-wide modification. Other operations modify the PTE directly. For example, the page of parent process can be reclaimed by the out of memory (OOM) killer. In this case, one PMD entry is involved. Note that swapping or migrating a 4KB page will change the PTE but the data will not be changed, so we will not handle it. Due to limited space, we summarize the locations where operations in the OS modify VMAs/PTEs as checkpoints in the technical report (tec, 2022).

We implement the detection by hooking the checkpoints. Once a checkpoint is reached, the parent process checks whether the involved PMD entries and PTEs have been copied (by checking the R/W flag of the PMD entry). For a VMA-wide modification, all the PMD entries of this VMA are checked, while only one PMD entry is checked for a PMD-wide modification. The uncopied PMD entries and PTEs will be copied to the child process before modifying them.

If a VMA is large, the parent process may take a relatively long time to check all PMD/PTE entries by looping over each of them. We therefore introduce a two-way pointer, which helps the parent process quickly determine whether all entries of a VMA have been copied to the child, to reduce the cost. Each VMA has a two-way pointer, which is initialized by the parent process during the invocation of the Async-fork function. The pointer in the VMA of the parent process (resp. the child process) points to the corresponding VMA of the child process (resp. the parent process). In this way, the two-way pointer maintains a connection between the VMAs of the parent and the child. The connection will be closed after all PMDs/PTEs of the VMA are copied to the child. Specifically, if no VMA-wide modification happens during the copy of PMDs/PTEs of the VMA, then the child closes the connection by setting the pointers in the VMAs of both the parent and child to null after the copy operation. Otherwise, the parent will synchronize the modification (i.e., copying the uncopied PMDs/PTEs to the child), and close the connection by setting the pointers to null after the copy operation. As both parent and child processes can access the two-way pointers, the pointers are protected by locks to keep the state consistent. When a VMA-wide modification occurs, the parent process can quickly determine whether all PMDs/PTEs of a VMA have been copied to the child by checking the pointer’s value, instead of looping over all these PMDs. Besides, the pointer is also used in handling errors (see Section 4.4).

Since copying the page table involves memory allocation, some errors may occur during both the default fork and Async-fork. For instance, a process may fail to initialize a new PTE table due to out of memory. Such error may only happen in the parent process in the default fork and has a standard way to handle the error. However, the copying of page table is offloaded to the child process in Async-fork, such error may happen in the child process, and a method is required to handle such errors. Specifically, we should restore the parent process to the state before it calls Async-fork, to ensure that the parent process will not crash in the future.

As Async-fork may modify the R/W flags of the PMD entries of the parent process, we roll back these entries to be writable when errors occur in Async-fork. Errors may occur 1) when the parent process copies PGD/PUD entries, 2) when the child process copies PMD/PTEs, and 3) during a proactive synchronization. In the first case, the parent process rolls back all the write-protected PMD entries. In the second case, the child process rolls back all the remaining uncopied PMD entries. After that, we send a signal (SIGKILL) to the child process. The child process will be killed when it returns to the user mode and receive the signal. In the third case, the parent process only rolls back the PMD entries of the VMA containing the PMD entry that is being copied. The purpose is to avoid contending for the PMD entry lock with the child process. An error code is then stored into the two-way pointer of the VMA. Before (and after) copying PMDs/PTEs of a VMA, the child process will check the pointer to see whether there are errors. If so, then it stops copying PMD/PTE entries immediately and performs the rollback operations that are already described in the second case.

The parent process is still interrupted when a proactive PTE synchronization is triggered. A straightforward way to reduce the cost of the proactive PTE synchronization is to let the child process to first copy the PTEs potentially modified before other PTEs. However, this method is not practical because the data accessed by user queries are relatively random and the child process cannot determine which PTEs will be modified when the parent invokes Async-fork. We therefore propose to minimize the number of proactive PTE synchronizations by reducing the duration of copying the page table.

As VMAs are independent, the kernel threads can totally perform the copy in parallel and obtain near-linear speedup. Therefore, the child process may launch multiple kernel threads to copy PMD/PTEs in multiple VMAs in parallel, so that the copy completes faster. It can effectively reduce the number of proactive PTE synchronization because the synchronization only happens during the period that the child process is copying PMD/PTEs. The probability of triggering a proactive PTE synchronization gets lower when the period becomes shorter. The experiment in Section 6.3 shows the efficiency of this optimization.

Multiple kernel threads consume CPU cycles. These threads periodically check whether they should be preempted and give up CPU resources by calling cond_resched(), in order to reduce the interference on other normal processes.

Figure 8 shows the implementation of Async-fork with a modular design. We encapsulate the code ingested to the kernel into a hook function, which is instantiated in a kernel module. We can insert/remove the module in/from the kernel as necessary. The hook function is inserted to the call path of default fork and memory subsystem in the Linux kernel (version 4.19). Users can determine the usage of the default fork and Async-fork with the parameter $F$.

The hook function is enhanced from copy_page_range(), which accepts a flag (Fast or Slow) to control the copying of the page table. With “Fast” flag, the function copies PGD/PUD entries and sets PMD entries to be write-protected. With “Slow” flag, the function copies the write-protected PMD entries and PTEs; when the copying finishes, it sets the PMD entries to be writable and sets the PTEs to be write-protected. When Async-fork is called, the parent process copies the page table to the child process using the function with “Fast” flag. At the end of Async-fork’s invocation, the parent process puts the child process into the runqueue of a CPU and returns to the user mode. Before the child process returns to the user mode, it copies the page table using the function with “Slow” flag. The parent process proactively copies the uncopied PMD/PTEs (using the function with “Fast” flag) to the child process before modifying them.

Flexibility. For the IMKVS workload that has small memory footprint, the page table copy is already short. In this case, Async-fork brings small benefit. For these workloads, we provide an interface in memory cgroup to control whether Async-fork is enabled (as well as the number of kernel threads used to speed up copying PMD/PTEs in the child process). Specifically, when users add a process to a memory cgroup, they can pass a parameter to enable/disable Async-fork at run time (i.e., the parameter F in Figure 8). As shown in the figure, if the parameter value is 0, then the process will use the default fork. Otherwise, Async-fork is enabled. As such, users can determine which fork operation the process uses as necessary and use Async-fork without any modification in the source code of applications. The process will use the default fork if no parameter is passed in.

Memory overhead. The only memory overhead of Async-fork comes from the added pointer (8B) in each VMA of a process. In the case, the memory overhead in a process is the number of VMAs times $8B$. Considering a machine with 512GB main memory while 400 processes run simultaneously, there are roughly 760,000 VMAs according to our statistics. In this case, the memory overhead will be 760000$\times$8B $\approx$ 6MB. This overhead is generally negligible.

Support for ARM64. The design of Async-fork can also be implemented on ARM64. Specifically, we use the APTable[1:0] (Limited, [n.d.]) in the table descriptor of the PMD entry as the R/W flag. Async-fork can also be implemented on other architectures that support hierarchical attributes in the page table.

Consecutive Snapshots. It is possible that the parent process starts the next snapshot using Async-fork before the previous child process finishes copying PMD/PTEs, as the parent process returns to the user mode before the child process. In the current implementation, we will not block the next Async-fork call but keep a VMA’s page table be copied by only one child process at any time. When the parent process copies a VMA to the child process, it checks the two-way pointer to identify whether there exists a previous child process copying the page table of this VMA. If exists, it proactively copies the whole page table of this VMA to the previous child process. Async-fork adopts this design because supporting concurrent fork operations need to track all child processes that are copying page tables. This is hard because the kernel does not provide the information and we need to inject new data structures into the kernel to record the states and synchronize these child processes.

As Async-fork cannot support concurrent fork operations in a process (the same as the default fork in the OS), Async-fork cannot support the cases where the parent process needs to conduct the fork operation in an ultra-high frequency (e.g., in milliseconds). In spite of the limitation, Async-fork works well for database use cases because a storage engine does not create many child processes from the parent process simultaneously in practice. Specifically, conducting the fork operation frequently leads the parent to frequently turn into the kernel mode, which hurts the service quality. Moreover, many processes executing in parallel will degrade the performance due to the resource contention (e.g., IO bandwidth and CPU resources). Therefore, IMKVSes (e.g., Redis and KeyDB) do not recommend ultra-frequent data snapshots (generally 60 seconds (RedisLab, 2022b)) and have no cases requiring the parent process to invoke Async-fork in milliseconds to our knowledge. For HyPer, which uses the fork operation to support concurrent transaction processing, Async-fork can work well. This is because the parent process handles OLTP (a single updater) that has a rigid requirement on the latency, whereas the child process executes OLAP which generally has a long execution time and is more tolerant of the latency than OLTP. Moreover, HyPer notices the cost of the fork operation and designs a novel mechanism, which makes multiple OLAP queries to share a snapshot, to improve the performance.

We evaluate Async-fork on a machine with two Intel Xeon Platinum 8163 processors, each of them has 24 physical cores (48 logical cores). The machine has 384GB memory, and 1TB NVMe hard drive. In terms of software setup, the experimental platform runs CentOS 7.9 with Linux 4.19. Except for the experiments in Section 6.4, all the other experiments are conducted within a single machine as described above.

Benchmarks. We use Redis (version 5.0.10) (RedisLab, 2021b) and KeyDB (version 6.2.0) (Inc., 2022a) compiled with gcc 6.5.1 as the representative IMKVS servers, and use Redis benchmark (RedisLab, 2021a) as well as Memtier benchmark (RedisLab, 2022a) to be the workload generators. The benchmarks reveal the scenario where multiple clients send requests to the IMKVS server simultaneously. Similar to prior work (Chen et al., 2019), we enhance Redis benchmark to generate queries in an open-loop mode for measuring the latency accurately (Schroeder et al., 2006; Zhang et al., 2016).

By default, the experiments are conducted with the following settings: 1) 50 clients (default settings) are used in Redis benchmark, while 50 clients are used in Memtier benchmark for consistency. 2) The key range is set to $2\times 10^{8}$, the key size of 8B and the value size of 1024B. 3) Each experiment is repeated by five times and the average results are reported. 4) In Async-fork, the child process launches 7 additional kernel threads (together with the child process itself, there are 8 threads in total) to help it copy PMDs/PTEs faster. Our experiment in Figure 14 shows that Async-fork still outperforms the state-of-the-art solution with a single child process. 5) The KeyDB server is configured with 4 threads.

Metrics. We launch a large number of queries ($5\times 10^{6}$) to the IMKVS and record the latency of the queries that arrive in the snapshot process (start from the parent process calls fork until the child process persists all in-memory data). Specifically, we measure the 99%-ile latency because latency-sensitive services generally provide an SLA on some percentile. Moreover, we report the maximum latency because IMKVS is often used for demanding use cases that have a rigid requirement for the worst-case latency (RedisLab, 2021c). For example, the increase of the maximum latency of Redis can lead to read error on connections (Technology, 2017). Therefore, in the production environment (Gong et al., 2022), the maximum latency is an important indicator of system stability. Due to limited space, we report the experiment results of query processing throughput and total out-of-service time of the parent process in the technical report (tec, 2022). In the report, we also discuss the method of tuning IMKVSes to further improve the performance.

Baselines. We compare Async-fork with On-Demand-Fork (denoted by ODF in short) (Zhao et al., 2021), the state-of-the-art shared page table-based fork. In ODF, each time a shared PTE is modified by a process, not only one PTE but 512 PTEs located on the same PTE table will be copied at the same time. We do not report the results of the default fork in this section since it results in 10X higher latency compared with both Async-fork and ODF in most cases (already presented in Section 3.2). To accurately measure the performance of the IMKVSes in the experiments, we trigger the snapshot operation manually using the BGSAVE command.

We first evaluate Async-fork using write-intensive workloads that require frequent snapshots for data persistence.

Latency Results. In this experiment, we use the Redis benchmark (RedisLab, 2021a) to generate the write-intensive workload by issuing SET queries to the IMKVSes, and configure the clients to send 50,000 such queries in a second. Since the IMKVS is often used in the context of demanding use cases, there are usually strict requirements on both the 99%-ile latency and the worst case latency (RedisLab, 2021c; Inc., 2022b).

Figure 9 shows the 99%-ile latencies of snapshot queries in Redis and KeyDB, with ODF and Async-fork. As observed, Async-fork outperforms ODF in all the cases, and the performance gap increases when the instance size gets larger. For instance, operating on a 64GB IMKVS instance, the 99%-ile latency of the snapshot queries is 3.96ms (Redis) and 3.24ms (KeyDB) with ODF, while the 99%-ile latency reduces to 1.5ms (Redis, 61.9% reduction) and 1.03ms (KeyDB, 68.3% reduction) with Async-fork.

Figure 10 shows the maximum latency of the snapshot queries. As observed, Async-fork greatly reduces the maximum latency of the benchmarks compared with ODF, even if the instance size is small. For a 1GB IMKVS instance, the maximum latencies of the snapshot queries are 13.93ms (Redis) and 10.24ms (KeyDB) respectively with ODF, while the maximum latencies are decreased to 5.43ms (Redis, 60.97% reduction) and 5.64ms (KeyDB, 44.95% reduction) with Async-fork.

Deep Diving. The interruption of Async-fork is caused by the proactive synchronization, whereas the interruption of ODF is caused by the CoW of the page table. We measure the interruptions of the parent process during the snapshot to understand the reason that Async-fork outperforms ODF. Specifically, the parent process turns into kernel mode and is out of service for queries when executing the copy_pmd_range() function. The overhead of copy_pmd_range() dominates the cost of executing in the kernel mode. In order to examine the number of interruptions and the out-of-service time, we use the bcc tool (bcc, 2022) to count the number of copy_pmd_rage() invocations and measure the execution time of each invocation. The result of bcc is a histogram in which the bucket is the time duration and the frequency is the number of invocations whose execution time falls into the bucket. The categories [16$us$, 31$us$] and [32$us$, 63$us$] are two default buckets of bcc. In our experiments, all invocations fall into the two buckets.

Figure 11 shows the frequency of the interruptions within [16us, 31us] and [32us, 63us]. We can see that Async-fork significantly reduces the frequency of interruptions. For example, Async-fork reduces the frequency of interruptions from 7348 to 446 on the 16GB instance. Async-fork greatly reduces the interruptions because the interruptions happen only when the child process is copying PMD/PTEs (the required time is within 600ms as in Figure 15). However, the interruption can happen until all data is persisted by the child process in ODF, while the data persistence operation requires tens of seconds (e.g., persisting 8GB in-memory data takes about 40s). Under the same workload, the parent process is more vulnerable to interruption when using ODF.

Sensitivities to the Read/Write Patterns. Figure 12 shows the results using four workloads with different read-write patterns generated with Memtier (RedisLab, 2022a). In the figure, “1:1 (Uni.)” represents the workload with 1:1 Set:Get Ratio and the uniform random access pattern, while “1:10 (Gau.)” is the workload with 1:10 Set:Get ratio and the Gaussian distribution access pattern.

Observed from Figure 12, Async-fork still outperforms ODF. The benefit is smaller for the workload with more GET queries. This is because the serving (parent) process only copy a small number of PTEs for the GET-intensive workloads. Moreover, the modified memory is smaller in the experiment with the Gaussian distribution compared with the uniform random access pattern. With random pattern, the key-value pairs in the IMKVS have the same probability of being accessed, and parts of key-value pairs may be accessed repeatedly with the Gaussian Distribution access pattern. Since shared PTEs are copied only when they are modified for the first time in ODF, the parent process is interrupted fewer times with the Gaussian distribution access pattern.

In general, Async-fork works better for write-intensive workloads. The larger the modified memory is, the better Async-fork performs. Integrating Async-fork with CCoW (Ha and Kim, 2022) to improve the performance on write-intensive workloads is an interesting research direction because Async-fork can utilize CCoW to copy the PTEs of high-locality memory pages in advance to reduce the number of proactive synchronizations.

The Impact of the Number of Clients. In this experiment, we change the number of clients in Redis-benchmark while keeping sending 50, 000 SET queries every second to an 8GB IMKVS server. Figure 13 shows the results of 99%-ile and maximum latency with 10, 50, 100 and 500 clients. As observed, Async-fork outperforms ODF, while the performance gap increases as the number of clients increases. This is because more requests arrive at the IMKVS at the same time when the number of clients increases. As a result, more PTEs may be modified at the same time, and the duration of one interruption to the parent process may become longer.

The Impact of the Number of Threads In the Child Process. In Async-fork, multiple kernel threads may be used to copy PMD/PTEs in parallel in the child process. Figure 14 shows the 99%-ile and maximum latency of snapshot queries under different Redis instance sizes. Async-fork#$i$ represents the results of using $i$ threads in total to copy the PMDs/PTEs.

Observed from Figure 14, Async-fork#1 (the child process itself) still brings shorter latency than ODF. The maximum latency of the snapshot queries is decreased by 34.3% on average, compared with ODF. We can also find that using more threads (Async-fork#8) can further decrease the maximum latency. This is because the sooner the child process finishes copying PMDs/PTEs, the lower probability the parent process is interrupted to proactively synchronize PTEs.

In more detail, Figure 15(a) shows the time that the child process takes to copy PMDs/PTEs with different numbers of kernel threads, while Figure 15(b) shows the corresponding 99%-ile and maximum latency in an 8GB Redis instance. As we can see, launching more kernel threads effectively reduces the time of copying PMDs/PTEs in the child process. The shorter the time is, the lower the latency becomes.

Async-fork has been deployed in our production Redis Clouds. We rent a Redis database from our public cloud to evaluate Async-fork in the production environment. The rented Redis server has 16GB memory and 80GB SSD. We also rent another virtual machine with 4 vCPU cores and 16GB memory from the same cloud to run the Redis benchmark as the client. The client settings are the same as previous experiments. The network bandwidth between the IMKVS server and the client is 3Gbps.

We compare Async-fork with the default fork in this subsection because ODF is not available in our cloud. Figure 16 shows the 99%-ile latency and maximum latency of the snapshot queries with the default fork and Async-fork. As observed, with an 8GB instance, the 99%-ile latency of the snapshot queries is reduced from 33.29ms to 4.92ms, and the maximum latency is reduced from 169.57ms to 24.63ms. With a 16GB instance, the former is reduced from 155.69ms to 5.02ms, while the latter is reduced from 415.19ms to 40.04ms.

Before Async-fork is deployed on our Redis Clouds, we received a large number of complaints from tenants about the long latency when taking the snapshot. Some tenants even have to disable the snapshot function, and they cannot reboot their Redis servers, otherwise the data lost. After we deploy Async-fork, no more latency complaints have been received.