Advanced Synchronization Techniques for Task-based Runtime Systems

In Nanos6, a task is a sibling of another when they are at the same nesting level. Tasks declared inside another one (nested) are considered child tasks.

The dependencies of a task in Nanos6 are represented as accesses, which are composed of a memory address and an access type, e.g., read or write. Two accesses have a successor relation when they share the same memory address and their tasks have a sibling link. Similarly, two accesses have a child relation if they share the same memory address and their tasks have a child link. Task access relations on OmpSs-2 programs form binary trees between the linked tasks, as shown in Figure 1. Note that on OpenMP users cannot express dependencies crossing nesting levels, and the child relationship is not considered to determine the dependencies between tasks.

During an OmpSs-2 program, several tasks can be created and finished concurrently. They have to propagate dependency information through the data structures to determine if the added tasks are ready and if the recently finished tasks allow any successor to become ready.

The previous implementation of dependencies inside Nanos6 was based on fine-grained locking, but it was very complex to avoid possible deadlocks. Instead of protecting the data structures that hold the information about the dependencies through mutual exclusion, our alternative is to adapt some wait-free programming concepts to create a data structure capable of supporting the concurrency we need. Otherwise, if we had decided to build a data structure based on mutual exclusion, we would have to compromise either with the complexity of fine-grained locking or the performance degradation of coarse-grained locking. The main goal is not to have wait-freedom as a requirement but to provide fast and scalable dependency registration.

In this section, we describe the concept behind the dependency implementation we propose for the Nanos6 runtime. In the following section, we will formalize the approach and prove its wait-freedom property.

When a program creates a task, all the dependencies are registered inside the Nanos6 runtime using the DataAccess structure, shown in Listing 1. The Task structure stores all task-related information, including a pointer to the array of its accesses. Each access has an atomic flags field that stores its current state, indicating if the dependency is currently satisfied (not preventing the task execution) and whether the satisfiability information has propagated to its successor and child accesses. The access also stores a pointer to its successor, which is the next access (belonging to a successor task) to the same address in the current nesting level, and a pointer to the child, which points to the first access to the same address that belongs to a child task.

The flags field represents a Finite State Machine in which the state diagram has no cycles, so there are starting and final states. Since we only modify this data structure with atomic operations, we have named it Atomic State Machine or ASM. Note that there is one instance of this state machine for each access.

The only way for an ASM to transition from one state to another is through receiving a data access message. The structure of a message, shown in Listing 2, contains two flags fields. One field contains the flags to set in the target access. The other has flags that have to be set on the message’s originator as a delivery notification. The ASM’s transitions have to be done as a single atomic operation, optimally through a fetch&or. Based on the values before and after the transition, the ASM may generate additional messages to deliver to its child or successor accesses. All the messages that are still pending to deliver are stored in a simple per-thread queue called MailBox. We illustrate this process in Figure 2.

Figure 2 represents the basic structure of the algorithm. While the MailBox has undelivered messages, we pop one from the container and deliver it to the destination access. Upon receiving the message, the access atomically updates its flags field. The atomic update provides us with the flags’ exact values before and after the message was received. As flags cannot be unset, we know each transition happens only once in the access lifetime. With this information, we decide if it is needed to generate more messages (to propagate information about satisfied accesses, for example). Finally, we atomically update the flags field of the originator access of the message to notify the delivery of the information. We use this last atomic update to determine we can safely delete an access.

In this section, we introduce several relevant definitions and finally prove wait-freedom.

The restrictions on the content of the messages are part of what provides the wait-freedom assurance. Informally, bits in the flag field can only be set, and the field size is limited. Additionally, each message that an access receives has to contain at least one flag, and none of the flags can be already set in the access. By those properties, we can deduce that each access can receive only a limited number of messages, which in the worst case is $|F|$.

A task is unregistered once it finishes its execution, and the message delivered to the access indicates this condition. An unregister operation will thus do $|A_{t}|$ delivery operations. As we have proved that a delivery is wait-free, the unregister operation will be wait-free because it does a finite and known number of delivery operations.

To avoid the stagnation of ready tasks in the scheduler, we have decoupled the actions of adding and scheduling ready tasks. When a task becomes ready, we do not directly add it to the scheduler but a bounded wait-free single-consumer single-producer (SPSC) queue working as a buffer.

The number of SPSC queues can be configured from a single one to one per core. In the first case, we would need a lock to synchronize all task additions, while in the latter, no locking is needed at all. We use the lock to synchronize between producers, but the synchronization between producer and consumers remains wait-free. In our experiments, we use one SPSC queue and lock per NUMA node.

When a worker threads enters the scheduler, it first drains all SPSC queues and inserts the ready tasks into the global ready queue. With this approach, we ensure that any contention generated by many worker threads calling the scheduler does not affect the performance of cores that are creating tasks. We can implement this optimization because the actual addition of tasks can be safely delayed until a core becomes idle and calls the scheduler. Notice that this delegation technique is compatible with any lock implementation.

The scheduler system has to be extensible, and adding new scheduling policies should be easy. We have discarded a wait-free or lock-free scheduler because of its complexity and difficulty to maintain, as each scheduling policy would require a new ad-hoc design and implementation.

Our scheduling system relies on a global lock to protect its internal data structures, making it easy to develop new scheduling policies. Ticket Locks (Reed and Kanodia, 1979) are fair and provide strict FIFO ordering, but they have contention problems under high-load conditions, so they are not suitable for our centralized scheduler. Partitioned Ticket Locks (Dice, 2011) (PTLocks) extend Ticket Locks with a padded array used to do busy-waiting by idle threads. If the size of this array is equal to the number of CPUs, then each core will busy-wait in a different array slot, reducing the cache coherence traffic to the minimum. We use PTLocks as a building block of our optimized lock design presented in the next section.

Listing 3 shows the implementation of a Partitioned Ticket Lock. For the sake of clarity, we have omitted the padding of the fields to prevent false sharing, and the memory order constraints of all atomic operations. The _waitq (line 6) is an array of unsigned 64-bit integers used to implement a circular buffer representing an infinite virtual waiting queue. The _head and _tail fields (lines 4 and 5) are used to index the _waitq array. The _head represents the index of the latest slot in the virtual waiting queue and the _tail is the index of the next slot that will be able to acquire the lock. When the lock is free and no thread is waiting to acquire it, _tail == _head+1.

We initialize the lock such that _waitq[_head%Size] == _head, guaranteeing that the first thread that arrives will be able to acquire it. The lock() operation (line 14) consists of just two calls. The first one is _getTicket() (line 8), which performs an atomic fetch and increment of the _head field to obtain the last ticket (line 9). The second call is _waitTurn (line 11) that receives the ticket as a parameter. The current thread busy-waits on the _waitq[ticket % Size] position until it matches (or exceeds) the ticket value. The unlock operation (line 17) calculates the next slot index that will be able to acquire the lock. Then it increments _tail and writes _tail-1 in the computed slot to release the lock.

PTLocks perform as well as more complex designs such as MCS (Mellor-Crummey and Scott, 1991) or Ticket Locks Augmented with a Waiting Array (TWA) (Dice and Kogan, 2019), however it requires more memory space.

Another well-known technique to improve the performance of data structures that are not amenable to fine-grained locking is delegation (Roghanchi et al., 2017). The main idea behind this method is that protected data structures are accessed only by one privileged thread, which executes all the operations on behalf of the other threads. Delegation relies on a lightweight and optimized communication protocol between the privileged thread and the rest of the threads to be able to delegate operations and forward results back. A drawback of this approach is that it requires a dedicated core for each independent set of data structures that has to be protected, making it unpractical in many situations.

There are variants of the delegation technique that use queues to delegate work to the threads currently inside the lock (Klaftenegger et al., 2018). These are better suited to use in centralized schedulers as they do not require dedicated cores for each lock. In order to build our scalable centralized scheduler, we have developed a novel Delegation Ticket Lock (DTLock) that builds on state of the art delegation techniques and extends our implementation of the PTLock with support for fine-grained and dynamic delegation of operations. The DTLock supports the standard lock, unlock and trylock operations, as well as lockOrDelegate, empty, front, popFront and setItem. The lockOrDelegate operation either acquires the lock if it is free or delegates the operation to the current lock’s owner. However, it is the current owner that will decide if it executes or not the delegated operations. Suppose the current owner releases the lock without performing a pending delegated operation. In that case, the thread that originally delegated that operation will eventually acquire the lock and execute it by itself. The empty, front, popFront and setItem operations can only be called by the thread that owns the lock and are used to manage the threads that are waiting outside the lock.

The main advantages of the DTLock are that it does not require a dedicated core and its additional operations can be freely combined with traditional lock, unlock and trylock operations. Additionally, the DTLock allows for threads to remain inside the critical section of the lock executing delegated operations. This can be leveraged to minimize operation latency when there is not enough work to keep all cores busy.

Listing 4 shows the implementation of a DTLock in C++, which inherits the PTLock’s _head, _tail and _waitq[] members, as well as, lock, unlock and tryLock operations. The DTLock extends the PTLock with two additional arrays. The _logq array (line 3) is used to register waiting threads while the _readyq array (line 4) is used to store the result of delegated operations, i.e. ready tasks in our case.

The first parameter of the lockOrDelegate operation is a unique id that identifies the thread that is calling this function. This id should be in the range 0..Size-1 as it is used to index the _readyq array. Thus, we need to know in advance the maximum number of threads that can call the DTLock. If the lockOrDelegate operation is finally delegated, the second parameter is used to store the result.

The first thing done in lockOrDelegate is to obtain a ticket (line 7). Then, the thread is registered on the _logq (line 8) array with just one store operation that combines the ticket and calling thread’s id. The values written on the _logq array cannot be overrun because it is guaranteed that there will be at most Size threads calling the lockOrDelegate operation, so that each thread will have their own position. Once the thread has been registered, it just busy-waits (line 9) until it acquires the lock (lines 11-12), or the operation has been delegated and the result is stored in the &item parameter (line 14).

The empty operation (line 17) returns true if there is no thread registered in the _logq array and false otherwise. We check the first position of the _logq array, and if the value is smaller than _tail, we know there is no thread waiting yet. Otherwise we would see the value written in line 8. Notice that this operation is intrinsically racy but harmless.

If a call to empty returns false, then the owner of the lock can call front (line 20) to obtain the id of the thread that is waiting. To that end, we only need to do the inverse of the operation done on line 8, subtracting the _tail’s value to obtain the thread id (line 21).

Then, the setItem operation assigns a result T to a registered thread using its id to index the _readyq array. First, it sets the item field (line 27) and then the ticket to the _tail value, marking the entry as valid. We use the ticket field in line 10 to determine if an operation was delegated or not. Finally, the popFront operation wakes up the first thread busy-waiting on the _waitq by executing the unlock operation.

This section presents a synchronized scheduler that leverages the SPSC queues and the DTLock described in sections 3.1 and 3.3, respectively.

Listing 5 shows the implementation of a synchronized scheduler using a DTLock (line 2) and a SPSC wait-free queue (line 5) to synchronize the getReadyTask and addReadyTask operations, respectively. The SyncScheduler is a wrapper of the unsynchronized scheduler (line 3), which implements the actual scheduling policy.

The PTLock (line 4) protects the producer side of the SPSC queue in the addReadyTask function (line 14), while the DTLock protects the consumer side in the processReadyTasks function. In the addReadyTask function, if there is no free space on the SPSC queue, the thread will try to acquire the DTLock with a tryLock operation (line 17). If it succeeds, it will call the processReadyTasks function (line 18), which removes all tasks from the SPSC queue and inserts them to the unsynchronized scheduler (line 8).

We use the lockOrDelegate operation (line 25) of the DTLock to synchronize getReadyTask operations. If the operation is delegated because another thread owns the DTLock, the calling thread will busy-wait until it gets a task (lines 25-26). Otherwise, it will eventually acquire the lock and call the processReadyTasks (line 27) to add the tasks that are waiting on the SPSC queue into the unsynchronized scheduler. Then, it will try to schedule a task for each of the threads that are waiting on the DTLock (lines 28-33) until there are no more waiting threads (line 28) or no ready tasks are left (line 31). At that point, it will try to get a task for him (line 35) and then release the DTLock (line 36). In our simplified design, the thread inside the scheduler leaves as soon as there are no more tasks to schedule. However, it is easy to extend our design in a way that processReadyTasks is called when no tasks are left inside the scheduler.

Figure 3 shows a timeline of five threads creating and scheduling tasks using the SyncScheduler and a simple FIFO scheduling policy. $Th_{0}$ has already created and inserted three tasks ($T_{0}$ - $T_{3}$) that are inside the SPSC queue before creating and inserting four additional tasks ($T_{4}$ - $T_{7}$). $Th_{1}$ to $Th_{4}$ call the getReadyTask function, one after the other, to obtain a ready task. The call to lockOrDelegate of $Th_{1}$ acquires the lock, while the other threads delegate and busy-wait. Once $Th_{1}$ is inside the lock, it calls processReadyTasks and inserts tasks $T_{0}$ to $T_{3}$ into the actual scheduler. Then, $Th_{1}$ schedules one ready task for each of the waiting threads, and finally, it gets a ready task for itself. When $Th_{3}$ finishes the execution of $T_{1}$, it calls again to getReadyTask, and just after that, $Th_{2}$ does the same. $Th_{3}$ acquires the lock and calls processReadyTasks, moving the tasks from the SPSC queue ($T_{4}$ and $T_{5}$) to the actual scheduler. Finally, $Th_{3}$ schedules $T_{4}$ for $Th_{2}$, and then, it executes $T_{5}$.

In microbenchmarks, we found a fourfold speedup on task scheduling using a DTLock compared to a PTLock, and a twelvefold speedup compared to serial task insertion thanks to the SPSC queues.

To evaluate the task-based runtimes and check the capability of scaling to more finely partitioned work, we will use the following benchmarks, running constant problem sizes and varying the task granularity. These are (1) a Dot product between two arrays that uses a task reduction to aggregate the results from each block, (2) an iterative Gauss-Seidel method solving the heat equation of a 2-D matrix in blocks and task reductions to calculate the residual of each time step, (3) a taskified HPCCG with with several kernels using task reductions and multi-dependencies, (4) a taskified version of Lulesh 2.0 (Karlin et al., 2013), (5) a taskified miniAMR that mimics the different patterns of Adaptive Mesh Refinement applications (Sasidharan and Snir, 2016; Sala et al., 2020), (6) a classic parallel blocked Matmul, (7) an NBody benchmark that mimics dynamic particle system simulations, and (8) a blocked Cholesky decomposition that is generally compute bound.

We ran our experiments on various HPC platforms featuring very different architectures: (1) the Intel Xeon with 2x Intel Xeon Platinum 8160 (Sky-lake) for a total of 48 cores, (2) the ARM Graviton2 with 64 ARM Neoverse N1 cores, and (3) the AMD Rome with 2x AMD EPYC 7H12 processors for a total of 128 cores and 256 hardware threads.

For all benchmarks, the parallelization is implemented using tasks with OpenMP and OmpSs-2 versions that feature the same amount of parallelism. However, the kernels used inside each task are sourced from the best available vendor library for each machine, to guarantee competitive performance. In the AMD and Intel machines this library was Intel MKL, and on the ARM Graviton2 we used the ARM Performance Libraries. We ran each benchmark a minimum of five times to extract each measurement.

Following this evaluation, we will compare our optimized Nanos6 runtime with the most relevant OpenMP implementations for each machine, including GOMP 9.2.0 (GNU Project, 2020), LLVM 10 (LLVM Project, 2020), Intel OpenMP and the AMD AOCC depending on availability for each platform. It is worth noting that both the LLVM, AMD AOCC and Intel OpenMP runtime are based on a work-stealing scheduler, which will allow us to determine if our centralized delegation-based implementation can outperform work-stealing runtimes.

Note that some combinations might not be available depending on the platform because of incompatibilities, non-implemented OpenMP features or compiler bugs. For brevity, we only show the four most relevant benchmarks for each machine.

The best way we found to evaluate objectively how each component of the runtime affects the overall performance is to remove the optimizations we have described selectively. To present the results, we use a metric (Slaughter et al., 2020) we will refer to as efficiency. It is calculated by dividing the performance of a specific run of a benchmark by the peak performance obtained across all executions. This efficiency provides a view of how close to peak performance is a specific run while being agnostic to benchmark specific units. Combining this metric with varying task granularity (Gautier et al., 2018; Maroñas et al., 2019) gives a good view of each runtime version’s scalability. The granularity is expressed in instructions executed per task, which gives an approximation of the task’s size. We chose this unit instead of using time or cycles because the scheduling policies used by each of the runtimes can affect the execution time of a task, and thus it could result in unfair comparisons.

The runtime version without the wait-free dependencies uses the previous dependency implementation based on fine-grained locking. The variant without the DTLock has a simple mutual exclusion mechanism (based on the PTLock) protecting the scheduler. Finally, the version without jemalloc uses the standard system allocator for each of the machines.

Figure 4(c) displays our benchmarks running in the Intel Xeon platform. The different versions allow us to explore precisely how each optimization affects the scalability for different benchmarks. The results also confirm that for every benchmark at least one optimization greatly increases the performance for fine-grained tasks.

Figure 5(c) shows a similar picture on the AMD Rome system. This system has a much larger number of CPUs, which can increase the performance degradation caused by heavily contended locks. Illustrating this point, we see that the scheduler optimization is much more relevant than in the Intel Xeon. The clearest example is seen on the miniAMR benchmark, which we analyze with detailed traces in subsection 6.4.

Finally, Figure 6(c) shows the same benchmarks running on an ARM Graviton2 system. Results are similar to our Intel Xeon evaluation, although some benchmarks have different behaviors due to the lack of NUMA effects on this platform.

Overall, we have seen that our optimizations achieve significant performance gains, especially on small task granularities. Depending on the benchmark, the wait-free dependencies or the scheduler are the most important optimizations. However, the scalable memory allocator also delivers some notable performance improvements, especially on the Intel Xeon and AMD Rome machines.

To give context to the results, we compare the optimized Nanos6 runtime to current state-of-the-art OpenMP runtimes on the same three systems. Our baseline for every benchmark will be the GOMP runtime, distributed with the GCC Compiler, and the LLVM OpenMP Runtime. However, on Intel Xeon we use the Intel OpenMP runtime, and on AMD Rome the runtime provided by the AMD AOCC, as long as they implement all the features needed by the benchmark. To ensure a fair comparison, we expressed the same parallelism in the OmpSs-2 and OpenMP versions of the benchmarks. We also used available kernels in Intel MKL or the ARM Performace Libraries, to prevent noise introduced by the compilers to alter the results.

Figures 7(c), 8(c) and 9(c) feature the results of comparing the current OpenMP implementation versus the optimized variant of the Nanos6 runtime. The results are really positive, as in all of the machines, and all of the benchmarks, the best performance in small granularity tasks is provided by the Nanos6 runtime. In some cases, a higher peak performance is also achieved. This happens when the ideal block size for a specific benchmark is small enough that performs better in one runtime than another.

As for the other runtimes, the LLVM OpenMP implementation comes second in most benchmarks in terms of scalability, and ties on AMD Rome with the runtime provided with the AOCC compiler. However, this is expected because the AOCC compiler is based on LLVM 10.

Figure 10 shows two miniAMR traces obtained with the new Nanos6 CTF instrumentation backend comparing the Partitioned Ticket Lock (PTlock) and the combination of wait-free queues with the Delegation Ticket Lock (DTLock). The view displays running tasks (in red), specific runtime subsystems such as task creation (in cyan), or other generic runtime parts (in deep blue) along time (X axis) for a number of cores (Y axis). The total length is 500us and the zoomed areas (black rectangles) are 10us long, approximately. Hint A points to cores running a ”task creation” task. The wait-free version (above) allows created tasks to be queued independently of other cores requesting ready tasks to run. Instead, in the PTLock version (below), adding and getting a ready task requires obtaining a shared lock which leads the ”task creation” task to undergo heavy pressure as other cores requesting a ready task also attempt to acquire the same lock. Consequently, the number of available ready tasks cannot match the task completion rate and most cores starve (in khaki green).

Hint B points to DTLock task serving periods. Yellow arrows depict single tasks being served by the delegation lock owner to waiting cores. When no more ready tasks left, the lock owner moves ready tasks from the wait-free queues (in green) and continues serving tasks.

Figure 11 exemplifies the effect that the operating system noise can incur on the runtime system. The upper trace displays runtime threads (in deep blue) and hardware interruptions (purple). The trace below shows a view similar to Figure 10, where a considerable delay is introduced in the server which causes all cores to stall but the ”task creator” core. Note the yellow lines pattern difference before (irregular) and after (regular) the interrupt. While the serving thread was stalled in the interrupt, a provision of ready tasks was accumulated. The surplus of tasks is enough to feed all cores leading to long periods of red (tasks) without yellow lines. As the extra reserve of tasks lowers, the chances of at least two idle cores requesting a task simultaneously increase, and extra yellow lines appear. In conclusion, combining OS events with runtime events allows us to complete the whole picture and to identify the source of problems better.