Enabling Atomic Durability for Persistent Memory with Transiently Persistent CPU Cache

Pmem.  Pmem embraces both byte-addressability and persistency. Researchers have considered building pmem with various memory technologies, such as phase-change memory [59, 40, 41, 69, 60, 77, 4], spin-transfer torque RAM (STT-RAM) [21, 80, 12, 91, 1, 25], resistive RAM [83, 90, 75, 88, 14], 3D XPoint [28, 31], and DRAM backed by flash [18, 62, 64, 49, 87]. Applications can directly load and store data with pmem [76, 45, 15, 28, 87, 44, 78, 38, 86, 3, 68, 5, 66].

Persistence Domain.  Persistence domain is a region of computer system in which data would not be lost but retrievable when the system crashes or power failures occur [68, 30]. It conventionally includes disk drives at the secondary storage level. The advent of pmem brings it up to the memory level.

ADR.  The ADR further extends the persistence domain to the WPQ of MC [13, 26, 72, 27]. ADR guarantees that data received at the WPQ can be flushed to pmem upon a power outage. Though, the persistence enabled by ADR is transient, as it is the pmem that eventually makes data persistent. Also, CPU cache is still volatile and cache lines would be lost on a crash. Thus, programmers must explicitly flush data staying in cache lines (e.g., clwb) to pmem. Flushing data from CPU cache to pmem is not only synchronized and time-consuming, but is also error-prone and hurts programmability [65, 2].

eADR.  Intel extended ADR as eADR which guarantees to flush all cache lines to pmem in case of a power outage by employing extra power supply [30, 68]. Alshboul et al. [2] proposed BBB that employs a battery-backed persist buffer alongside each core’s L1D cache and achieves an identical effect as eADR with much less cost. They help programmers avoid explicit cache line flushes. More important, they make the multi-level CPU cache hierarchy provide a transient persistence domain in dozens of megabytes on top of pmem.

The atomic durability, or failure-atomic durability, refers to the crash consistency of modifying in-pmem data in case of a crash. The insertion with an in-pmem sorted array mentioned in Section I is a typical transaction programmers would define with their desired semantics. A transaction must be done in an atomic (all-or-nothing) fashion. Otherwise, a half-done change may leave data in ambiguity or uncertainty after a crash.

Hercules uses components distributed in CPU cache, MC, and pmem to jointly control the procedure of transactions and manage the versions of data for each transaction. 3(a) captures the main components of Hercules.

TransTag.  A transaction of Hercules is a contiguous series of data operations covering one or multiple cache lines. In order to manage a transaction’s data, Hercules adds TransTags per cache set. A TransTag has WayNo, TxID, and TxState. The WayNo indicates which actual cache line in the cache set Hercules is using for a transaction that is identified with a unique 21-bit TxID. The reason why 21 bits are used is twofold. Firstly, Hercules demands extra costs such as energy, transistors, and wires to enable the atomic durability. More than 21 bits per TxID may further increase such costs and entail more challenges to achieve systematic efficiency and reliability (see Sections V-D). Secondly, 21 bits support up to $2^{21}$ ($\geq$2 million) transactions, which can satisfy the needs of typical applications at runtime.

The TxState in one bit shows if the transaction is committed or not. We call a cache line being occupied by an uncommitted transaction transactional cache line, of which the TxState is ‘1’. Otherwise, it is non-transactional without a TransTag or with TxState being ‘0’. In short, we configure a logical part of a cache set by filling WayNos for the use of Hercules, without fixing some cache lines for transactions. This brings in spatial efficiency and flexibility. Given a small amount of transactional data, non-transactional data can freely take cache lines with TxStates unset. Also, non-transactional and transactional data can be placed without swaps. For the illustrative example in 3(b), half cache lines of a four-way set (WayNo in two bits) are usable with two TransTags that cost $2\times(21+2+1)=48$ bits per set.

Hercules manages and uses TransTags in a similar form of the directory used to track cache lines for coherence [93, 51, 73, 89, 85]. It places TransTags alongside L1D and L2 caches in each core and has a joint bunch of them for the shared LLC. This organization accelerates filling and changing fields in them. The ratio of TransTag per cache set is defined as the percentage of TransTags installed over all cache ways of a set. As suggested by 2(d), we shall consider a higher TransTags ratio for more usable space to caches that are closer to CPU, as they serve more transactional requests. The ratio of TransTags is thus practically decided by a cache’s proximity to CPU. In practice, we allot TransTags to all cache lines of L1D cache. The ratio can be half or a quarter for L2 and L3 caches. We have a discussion on this ratio in Section V-B.

Registers.  Hercules adds a register GlobalTxID that is monotonically increasing, shared by all cores to compose the next transaction ID. Hercules installs PerCoreTxID and PerCoreTxLen to each CPU core, holding the current running transaction’s ID and length (number of transactional cache lines), respectively. These two are critical for the context switch between threads and stay dormant for threads doing non-transactional operations. A register has 64 bits and Hercules uses the lower bits only for extension. They are kept volatile, not saved in pmem on a crash (see Section IV-C).

eWPQ.  Cache lines are evicted over time. Hercules regularly writes back non-transactional ones but handles transactional ones specifically in order not to harm in-pmem original copies. It adds two registers and an extended WPQ (eWPQ) along WPQ in the MC. LogHead records the address of next available log entry in the in-pmem log zone while LogTail points to the last valid log entry (see Section IV-D). On evicting a transactional cache line, the MC allocates a log entry to put data of the cache line by atomically fetching and increasing LogHead. Hercules retains that cache line’s metadata in an eWPQ entry, including TxID, TxState, and and the mapping from home address to log entry’s address.

Log zone.  3(b) shows three areas of in-pmem log zone. The first area (transaction profiles) is an array of transaction lengths (TxLens) indexed by TxIDs. Hercules keeps the runtime length of a transaction in the corresponding thread’s context while it only allows two legal values for the in-pmem TxLen, i.e., an initial zero and a non-zero eventual length. It uses the atomic change of TxLen to be non-zero to mark the commit of a transaction (see Sections IV-B and IV-C). Using a transaction’s TxID to index and find the transaction’s TxLen is fast and brings about lock-free parallel loads/stores for concurrent transactions. We set a TxLen in 4B, so a transaction can cover up to $2^{32}$ ($\geq$4 billion) cache lines.

In the second area, a log entry keeps a cache line evicted from the eWPQ with its data and metadata including the home address and TxState. The third area of eWPQ extension receives eWPQ entries evicted at runtime. The next area of emergency use is used upon an unexpected crash. We show a contiguous log zone in 3(b) while it can be partitioned to support concurrent accesses with distributed MCs [26].

Primitives. Like prior works [92, 35, 72, 54, 9, 39], Hercules has three primitives for programmers to proceed a transaction, i.e., tx_start, tx_commit, and tx_abort to respectively start, commit, and abort a transaction in applications.

Let us first illustrate how Hercules proceeds a transaction in an optimistic situation, i.e., 1) the transaction manages to commit, 2) all cache lines of the transaction stay in CPU cache until the commit, i.e., with no cache line evicted in the entire course of transaction, 3) Hercules can find a cache line with TransTag available whenever needed, and 4) no crash occurs.

Optimistic procedure.  On a tx_start, Hercules atomically fetches a TxID from the GlobalTxID and increments the register by one for subsequent transactions. Hercules finds its entry with TxID in the array of transaction profiles and initialize the TxLen to be zero ( in 3(b)). Until tx_commit is encountered, Hercules manages and processes cache lines for data that programmers put in the transaction. It adopts the write-allocate caching policy. Modifying data causes a cache miss if the data is not in CPU cache and Hercules allocates a cache line with TransTag before loading the cache line from pmem. It then fills TxID, WayNo, and data and sets the TxState as ‘1’. If data already stays in a clean cache line, Hercules obtains and configures a TransTag. Given a dirty cache line which might be originally non-transactional or belong to a committed transactions, Hercules sends that to the next level in the memory hierarchy, e.g., L1D$\rightarrow$L2, before getting a TransTag, in order not to taint the latest update. When the transactional cache line is evicted to the lower level, Hercules first sends the older dirty non-transactional version to the next lower-level cache or pmem. On a crash, the older version is firstly persisted to the home address. The upper-level transactional cache line, if committed, refills the home address; if not, Hercules writes it to the area of emergency use (to be presented). This rules out any inconsistency and uses lower-level caches for staging to further reduce pmem writes.

Hercules follows generic rules in programming transactions. It disallows nested or overlapped transactions in one thread, so at most one transaction is ongoing within a thread. Inspired by Intel TSX [29], Hercules provides a configurable option to support applications with a fundamental read committed isolation level [55, 7] to defeat against semantic isolation bugs. Programmers may consider concurrency control mechanisms like locks or semaphores between transactions in multi-threading programs. More details can be found in Section IV-D.

When a thread enters a transaction for the first time, the transaction’s TxID and length are used to fill PerCoreTxID and PerCoreTxLen of the running CPU core, respectively. PerCoreTxLen is incremented by one every time Hercules is going to launch a transactional update on an uncovered cache line. If a context switch occurs, the values of PerCoreTxID and PerCoreTxLen are saved as part of the thread’s context for an afterward execution. On committing a transaction, Hercules atomically sets the in-pmem TxLen with PerCoreTxLen and resets the TxState to be ‘0’ for each transactional cache line to make it visible to other threads.

The foregoing procedure shows that Hercules efficiently handles a transaction and commits on-chip. The eADR guarantees committed cache lines would be flushed to their home addresses in case of a crash. Next we present how Hercules handles conditions not covered in the optimistic circumstance.

Premature flush.  The eADR enables Hercules to write back data on cache replacements rather than explicit cache line flushes, so updates to a cache line are coalesced and pmem bandwidths are saved. Hercules handles the write-back of a non-transactional cache line with the MC’s WPQ in the ordinary way. For an evicted cache line recorded in a TransTag, if TxState is ‘0’, i.e., being non-transactional, the MC writes back the cache line to the home address. If TxState is ‘1’, Hercules initiates a premature flush with the MC’s eWPQ ( in 3(b)). As shown in 3(b), the eWPQ is made of eWPQ entries and an eWPQ validity bitmap to track the validity status of each eWPQ entry. The MC finds a free eWPQ entry for the evicted transactional cache line and allocates a log entry by atomically fetching and increasing the LogHead. The MC copies TxID, TxState, home address, and log entry’s address to the eWPQ entry and asynchronously writes back the transactional cache line in the log entry ( in 3(b)).

Hercules employs the eWPQ both for logging uncommitted data and loading proper data. When a thread resumes execution, it may use a cache line that has been prematurely flushed. The cache line may be from any transaction that is already committed or this resumed transaction. Hercules references cache lines regarding their home addresses. On a load request, the MC checks if the target address matches any eWPQ entry and simultaneously tests the TxState. Given a match and ‘1’ TxState, if the TxID is compared to be the same as ongoing TxID, Hercules gets a potential hit. A match with ‘0’ TxState is also likely a hit, because the cache line had been evicted before the relevant transaction committed, but not migrated to the home address yet. No match results in a miss.

Regardless of a hit or miss, once receiving a request, the MC starts loading the cache line from the home address. A hit at the eWPQ fetches the corresponding log entry and halts the load from home address. MC checks the full address stored in the log entry and forwards it to the CPU cache in case of a true match. When CPU cache receives the log entry, MC nullifies the matching eWPQ entry. A miss continues the load of cache line from home address. In addition, a forbidden access from an ongoing transaction may happen to an eWPQ entry with mismatched TxID and ‘1’ TxState. Hercules aborts that transaction with an exception (see Section IV-D for isolation).

The other reason for employing the eWPQ is to commit and migrate cache lines that have been prematurely flushed. A transaction may commit without reusing them. Hercules exploits the eWPQ to deal with them. There are two ways to deal with such cache lines. One is to load them into CPU cache for committing on-chip and store them to home addresses by cache replacements. The other one is to commit them off-chip by resetting the TxStates in corresponding eWPQ entries and migrating them from log entries to home addresses via non-temporal stores. We choose the second way to reduce cache pollution. For efficiency, we periodically scan eWPQ entries for data migrations. The period is configurable, set to be every three million instructions in our tests. A completion of migrating a log entry clears the validity bit for the eWPQ entry ( in 3(b)). A load request that happens before the reset of validity bit still fetches data from the log entry.

Previous works have justified the efficacy and efficiency of using a part of address for cache management [81, 36, 37, 70]. We accordingly devise a compact eWPQ entry that holds one bit of TxState and three fields of 63 bits evenly partitioned for TxID, home address, and log entry’s address. Our evaluation shows that transactions of typical applications are empirically small and generally take few to dozens of cache lines (see Section V-B), so a home address and a TxID in overall 42 bits are sufficient for indexing. If duplicate matches occur, an eWPQ entry leads to a log entry that holds the full home address to rule out ambiguity. We manage the eWPQ like a fully associative cache and set a default size of 4KB for 512 entries, which are ample to serve ordinary workloads found in typical applications (see Section V-B). We believe a larger eWPQ is practically viable [26, 27]. Yet we take into account the very low likelihood of a full eWPQ, wherein Hercules evicts the least-recently-used (LRU) entries to an in-pmem eWPQ extension area that is ten times larger than the eWPQ ( in 3(b)). If a request misses in the eWPQ, Hercules checks the eWPQ extension area with a target home address to properly fetch the corresponding log entry.

Flush on a power-off.  When a power-off occurs, Hercules flushes WPQ, LogHead, LogTail, eWPQ, and then all cache lines to pmem. Hercules writes non-transactional cache lines to home addresses. In case of a crash, it dumps transactional ones to the area of emergency use in the log zone with home addresses and TransTags ( in 3(b)). These metadata and data are useful for recovery (see Section IV-C).

Cache replacement.  Transactional and non-transactional cache lines flexibly share a cache set. Hercules considers an effective algorithm [81, 36, 37, 70, 34] to select a victim for eviction, but entitles a higher priority to transactional ones for staying in cache. A request to place a non-transactional cache line only replaces a non-transactional one in the set. To place a transactional one, Hercules tries to find a free TransTag and may replace a non-transactional victim. If all TransTags are being occupied, Hercules evicts a transactional cache line.

Transaction abortion.  A transaction may abort due to various events like exception, fault, or running out of memory. On an abortion, data recorded in the TransTags and eWPQ entries with TxIDs matched and TxStates being ‘1’s are invalidated and discarded, incurring no harm to original data.

Hercules puts a specific flag in the log zone to mark a normal shutdown or not. The flag is not set if any transactional cache line is saved to the area of emergency use on power-off. Regarding an unset flag, Hercules recovers at the transaction level in order to support applications recovering with semantics. As modifying TxStates of multiple cache lines cannot be atomic, Hercules atomically sets the TxLen to commit a transaction. In recovery, it fetches eWPQ, LogHead, and LogTail into the MC and scans transaction profiles.

Hercules discards transactions with zero TxLens. As to a committed one, Hercules scans the area of emergency use to find out cache lines with TxIDs matched and TxStates being ‘1’s. Hercules moves them to their home addresses. In addition, some cache lines of the transaction might have been prematurely flushed before the commit, being tracked by the eWPQ, but not migrated yet prior to the power-off. That explains why Hercules has saved the entire eWPQ. If an entry with a matching TxID is valid in the eWPQ validity bitmap, Hercules migrates the mapped log entry and then clears the corresponding validity bit. LogTail may be moved after the migration. Once moving all such cache lines is completed, Hercules resets the transaction’s TxLen to be zero. This atomic write rules out ambiguity if a crash takes place in an ongoing recovery. After resetting GlobalTxID and clearing the eWPQ, Hercules is ready to recommence new transactions.

Granularity.  A cache line is the unit transferred between CPU cache and memory, so Hercules chooses it as the unit for transactional operations. Using programmer-defined variables is more fine-grained but must incur higher cost and complexity.

Exclusion of transactional cache lines.  Hercules has no restriction of inclusion or exclusion on non-transactional cache lines. It enforces an exclusive multi-level cache hierarchy to transactional cache lines, which means an evicted transactional cache line, once reloaded into the higher-level cache, will be removed from the lower-level cache. The reason of doing so is twofold. Firstly, an exclusive housing of transactional cache lines substantially saves TransTags and cache space. Secondly, an exclusion management helps to reduce microarchitectural actions. Given a transactional cache line loaded to a higher level, e.g., L2$\rightarrow$L1D, for read purpose, an inclusive cache hierarchy has it at both levels holding the latest version. When the transaction commits, Hercules needs to reset TxStates twice for the same cache line across levels. Given a transactional cache line loaded to a higher level for updating, cache coherence protocols like MESI or MESIF help an inclusive cache hierarchy to invalidate the older version at L2 upon a modification, but that costs a microarchitecture-level coherence state transition ($M\Rightarrow I$). Hercules’ exclusion circumvents these inessential state resets or transitions.

Isolation.  Not all pmem systems supporting transactions provide thread-atomicity (isolation) [6].Like using prior hardware designs [92, 19, 35, 9, 72], with Hercules programmers are responsible for the isolations between threads via concurrency control methods (e.g., locks or semaphores). We consider that an error-prone application may misbehave on isolations and provide a configurable option for Hercules. In brief, following Intel TSX [29], Hercules aborts a transaction when threads contend to modify transactional data. Also, uncommitted data is invisible to other threads. If one thread does a read-only operation on data updated in an ongoing transaction, Hercules loads the data’s original copy with customized non-temporal hint and data-path [16, 67, 61] either from lower-level caches or pmem, depending on where the data’s original copy is (see Section IV-B). Thus, Hercules enables an optional support of the read committed isolation [55, 7] while software concurrency control can promote higher isolation levels.

Coherence. Hercules collaboratively works with cache coherence protocols like MESI or MOESI. TransTags and the directory for cache coherence [93, 51, 73, 85] share similarities in use and we can integrate them to jointly track cache lines. Hercules does not affect the sharing of non-transactional cache lines. As to transactional ones, the application’s concurrency control and architectural TxIDs in TransTags prevent other cores from modifying or fetching uncommitted versions of them. The exclusion and isolation of transactional cache lines also avoid broadcasting invalidation messages to all cores upon committing a transaction. In all, Hercules inflicts no harm but reduces state transitions to maintain cache coherence.

State reset.   Because cache lines of a transaction are likely to be scattered in a multi-level cache/memory hierarchy, all prior designs commit them with concrete efforts for atomic durability [92, 39, 9, 54, 35, 33]. For example, TC forcefully persists data in the side-path cache to pmem [39] while Kiln flushes down cache lines from upper-level volatile caches [92]. HOOP migrates all data staying in its out-of-place (OOP) buffer installed in the MC to its in-pmem OOP region [9]. FWB has to wait for the drain of current log updates [54]. Comparatively, Hercules’ commit is much more efficient and lightweight, as it just sets TxLen and resets TxStates for cache lines that a transaction covers. Since TransTags form a structure similar to the directory for cache coherenceused to track and transit states for cache lines, Hercules employs an auxiliary circuit to select ones with a TxID and clear their TxStates. For data that might be prematurely flushed before the commit, Hercules uses the auxiliary circuit to notify the integrated MC and wait for the completion of resetting TxStates in relevant eWPQ entries.Generally these resets can be swiftly done like state transitions for cache coherence. We preset a uniform state reset latency in which Hercules is supposed to finish, with a discussion presented in Section V-C.

Garbage collection (GC) on log entries.  Concurrent transactions commit at different time and take up discontinuous log entries. Committed log entries become invalid, scattered across the log zone. Figure 4 shows how Hercules cleans them up. LogHead and LogTail frame a window of log entries Hercules is using while the eWPQ tracks all valid ones. When LogTail has not moved for a while, numerous invalid entries might accumulate in the window. If Hercules monitors that the distance between LogTail and LogHead is greater than a threshold, e.g., $2^{20}$, it will initiate a GC ( in Figure 4). Hercules fetches a chunk (e.g., 32) of successive log entries starting at LogTail ( ). It appends valid uncommitted ones to the locations pointed by LogHead and updates corresponding eWPQ entries ( ). Only after updating each eWPQ entry will Hercules move LogHead by one. Then it slides LogTail over the chunk to the next valid entry ( ). Hercules orderly performs these steps to preclude any crash inconsistency. Note that the system may crash in a GC, particularly when a power outage has happened after a movement, which results in a moved log entry existing both at LogHead and LogTail. Such a log entry can be ignored as it belongs to an uncommitted transaction with TxLen being zero, without any loss of Hercules’ crash recoverability.

Log extension.  In a very low likelihood, excessive transactional data might occasionally overfill the entire CPU cache in dozens of megabytes or even flood the log zone. Hercules continues cache placements and replacements to swap in and out data, respectively, to proceed transactions. We can configure a log zone in gigabytes or even larger. In case that such a large space is still to be used up, we extend the log zone by using the end part of default log zone to store indirect indexes to the space in a new log zone allocated on-demand elsewhere. This is like the strategy of indirect blocks used by file systems to manage big files [82]. We also enhance the eWPQ with more entries and extend an eWPQ entry with one more bit to tell the MC if it needs to do indirect references or not to find actual data for a prematurely flushed cache line.

Throughput. We normalize the throughputs (txn/$\mu s$, transactions per microsecond) of all designs against that of OPT. As shown in Figure 6, we include the geometric mean of throughputs over benchmarks for a high-level overview [92, 39, 72, 9, 53]. Hercules achieves comparable performance to OPT. It significantly outperforms prior works with on average 89.2%, 29.2%, 15.2%, 51.3%, 48.0%, and 57.0% higher throughput than SWL_eADR, Kiln, TC, HOOP, Proteus, and FWB, respectively. Hercules leverages CPU cache hierarchy to absorb and coalesce transactional updates and mainly commits them on-chip. It gains superior efficacy in handling continuous transactions with such a spacious transient persistence domain.

The throughputs of Kiln and TC are limited by two factors. One is due to STT-RAM’s longer write/read latencies. The other one is that Kiln and TC have enforced limits in using persistent caches, such as using a small side-path cache [39] or taking a fall-back path to write pmem for backup in case of an almost full request queue [92]. Other hardware designs, such as HOOP, Proteus, and FWB, exploit hardware components like undo/redo log buffers, WPQ, or pmem to compose and persist backup copies. Continuous transactions keep limited log buffers or WPQ entries being fully occupied over time. They are hence inferior to Hercules that leverages the extensive CPU cache hierarchy to absorb and process data.

Figure 6 exhibits different observations across benchmarks, as their transactions are with different semantics and complexities. For example, two insertions with Linked List and RB-Tree differ a lot. Unlike prior designs varying significantly across benchmarks, Hercules shows consistently superior performance. Its strong robustness is mainly accredited to CPU cache it exploits. A multi-level cache hierarchy has been proved to be effectual for various workloads over decades.

Pmem writes.  To minimize pmem writes is another goal of Hercules. Figure 8 captures the quantity of pmem writes caused in running transactions with micro-benchmarks (Y axis in the logarithmic scale). Hercules significantly reduces pmem writes. On average, the data it writes is 29.8%, 37.4%, 5.3%, 1.4%, 1.5%, and 2.1% that of other designs in the foregoing order, respectively. Write-backs of Hercules only happen upon normal cache replacement or power-off, so it performs pmem writes in a passive and lazy way. SWL_eADR incurs double writes. Kiln takes a fall-back path that forcefully sends cache lines to pmem with an overflowing request queue. TC has a similar fall-back path to write pmem when its side-path cache is almost full. Also, whenever a transaction commits, TC issues relevant stores to pmem. Other hardware designs explicitly write backup copies to pmem. HOOP, for example, does address indirection at the MC and migrates data between pmem locations over time for out-of-place updating. Proteus leverages the WPQ for buffering to reduce pmem writes, but the limited capacity of WPQ impedes its efficacy. By using CPU cache in numerous megabytes to absorb and coalesce data updates, Hercules effectively minimizes pmem writes.

Tail latencies. Without loss of generality, we record 99P/99.9P (99- and 99.9-percentile) tail latencies for transactions and show them for three benchmarks in Figure 6 and 8, respectively, with Y axes in the logarithmic scales. These results further justify Hercules’ efficacy as it makes much shorter tail latencies with various workloads. Take B+-Tree for example. Hercules’ 99.9P tail latency is 40.5% and 55.1% shorter than that of Kiln and HOOP, respectively.

A comparison to the approach of checkpointing.  PiCL checkpoints data periodically in pmem for recovery without forming transactions [12, 53]. In Figure 9 we present the number of clock cycles and the quantity of pmem writes Hercules and PiCL have done to complete all operations for each benchmark. PiCL’s checkpointing is like undo-logging all data at the start of every epoch, which, albeit being simplistic, causes on average 78.3% more time and 34.1$\times$ pmem writes than Hercules. Hercules only covers data programmers place in fine-grained transactions and leverages CPU cache to log and buffer them. This is why Hercules costs both much less time and dramatically fewer pmem writes than PiCL.

We summarize TxLens on finishing all transactions for each micro-benchmark in Table III. Modern cache hierarchy has no difficulty in putting few to dozens of cache lines. Hercules hence effectually suits ordinary workloads and almost all transactions can be committed on-chip. Meanwhile, ordinary transactions in small sizes are hardly affected by the change of TransTag ratios and seldom cause premature flushes or GC.

To thoroughly evaluate Hercules, we run unrealistic artificial tests executed in a shrunk cache hierarchy. We scale down L1D/L2/L3 caches by 8$\times$ to be 3840B/32KB/2MB and synthesize test cases in which a huge transaction updates massive cache lines. We set the length of a transaction to be 1k, 10k, 20k to 60k (k: $\times 10^{3}$) and run with unlimited and 512 eWPQ entries, respectively. 10(a) shows the throughputs normalized against that with unlimited eWPQ. For a huge transaction involving tens of thousands of cache lines, CPU cache and eWPQ would be saturated and Hercules must use the in-pmem eWPQ extension. The impact of premature flushes grows up. For example, the throughput with 512 eWPQ entries declines by 32.8% at the 60k case.

We also run unrealistically huge transactions when varying TransTag ratios at L1D, L2, and L3. Without loss of generality, we illustrate with the 30k test case upon changing TransTag ratios at L2 and L3. In 10(b), we normalize the throughputs against that of default 100%/50%/25% ratios. There is an evident uptrend along an increasing ratio at L3. A 2MB L3 cache has 32,768 cache lines. An increased ratio means more space to house the working set of 30k case. Only 100% ratio at L3 manages to fit all 30,000 cache lines of 30k and yields the highest throughput without premature flush.

State reset latency.   We set the default state reset latency per transaction as ten clock cycles to toggle TxStates of aggregated TransTags with auxiliary circuit. We can deploy circuits in different complexities to gain different latencies. Figure 11 shows the throughput (txn/$\mu$s) curves on running micro-benchmarks when we vary the latency duration from the ideal (zero) to 90 cycles. With a longer latency, the throughputs of Linked list, Hash Table, Array Swap, and Binary Heap decrease more severely than those of B+-Tree, RB-Tree, and SDG. The curves in Figure 11 complement TxLens in Table III. The throughput of a benchmark with smaller transactions is surely more sensitive to the increase of state reset latency, since a smaller transaction itself takes less execution time. Take RB-Tree and Linked List for comparison again. An insertion to RB-Tree is generally more complicated and involves 50.6 transactional cache lines per transaction. Yet an insertion to Linked List deals with 12.5 cache lines per transaction on average. As a result, increasing the state reset latency more affects lighter benchmarks like Linked List.

WPQ size. We further test when varying the size of WPQ. Without loss of generality, we choose Linked List and present throughputs of all designs in Figure 13. Hercules’ bottleneck is not on write-backs via the WPQ to pmem, so its performance has no evident fluctuations. This justifies the robustness of Hercules in utilizing extensive CPU cache for atomic durability. A larger WPQ helps prior works like HOOP and Proteus yield performance improvements. Proteus leverages the WPQ of MC for transactional operations while HOOP depends on the MC to do address indirection and data movements. They hence benefit more from an increase of WPQ entries.

Pmem latency. The default write latency of pmem is 100ns. We vary it to emulate different pmem products. Figure 15 captures the throughputs of all designs again on Linked List with three write latencies. By keeping all transactional data in CPU caches without incurring pmem writes, Hercules is unaffected by varying pmem latencies. Notably, the performances of prior works that rely on writing data to pmem for backup badly degrade. In summary, leveraging the capacious transient persistence domain made of CPU caches enables Hercules with high adaptability to various pmem products.

Recovery.  We tailor gem5’s checkpoint function to save all metadata and data into the log zone backed by files and emulate a crash by encountering a simulator magic instruction. Hercules manages to recover properly and resume execution.

Spatial cost. Table IV summarizes the overall spatial cost of Hercules such that removing 2KB at L1D per core is sound to evaluate it. Similar to on-chip buffers used by prior works [2, 39, 33, 53, 54, 79], TransTags incur the main on-chip spatial cost for Hercules. Due to the space limitation, we brief an estimate with a 4-way L1D cache in 30KB. As we use all L1D cache lines, a TransTag needs 22 bits (21-bit TxID and 1-bit TxState) without WayNo. The original metadata per line, such as cache tag and state, takes at most 48 bits for a VIPT cache [94, 57]. TransTags thus cost $3.9$% ($\frac{\textnormal{22}}{48+64\times 8}$) more space. This also explains why we increase the tag latency by 30% on accessing a transactional cache line ($\frac{\textnormal{22}}{48+22}$ $\approx$ 31.4%). Similarly we estimate the proportions of TransTags at L2 and L3 to be $2.2$% and $1.2$%, respectively. In all, the spatial cost of Hercules is insignificant.

Energy cost.  If a crash occurs, besides the eADR’s ordinary flushes, Hercules persists the TransTag and home address that are estimated as at most 10B (e.g., $\geq\frac{22+48}{8}$ at L1D) for a transactional cache line. The energy costs per store from L1D, L2, and L3 caches to pmem on a crash are respectively $11.839nJ/$B, $11.228nJ/$B, and $11.228nJ/$B [2, 56]. The base cost of flushing the entire cache hierarchy is thus 214.831$mJ$ for 8-core CPU. Regarding 3840/ 16384/ 65536 transactional cache lines at L1D/L2/L3 with eight cores, all of them are dirty and uncommitted in the worst case. The cost to flush TransTags and home addresses for them is about 9.653$mJ$. Hercules also needs to identify transactional cache lines by comparing WayNo in each TransTag and all comparisons cost about 84.045$nJ$ with an up-to-date comparator taking $0.98pJ$ per comparison [74]. Overall, Hercules maximally brings about 4.49$\%$ ($\frac{9.653{}+84.045{}\times 10^{-6}}{214.831{}}$) extra energy cost, which is practicable in upgrading the eADR. In addition, a TxID in more bits may increase such extra cost to be beyond 5.0% or even greater, and also impose further challenges on designing and producing chips with efficiency and reliability [8, 10, 52, 58].

We utilize TPC-C and YCSB workloads from WHISPER [17, 33, 27] to evaluate Hercules for two purposes. Firstly, we measure the performance and robustness of Hercules with more sophisticated workloads of realistic applications. Secondly, we run with multi-threads to test the scalability of Hercules in serving concurrent transactions. TPC-C’s New-order follows the de facto semantics. As to YCSB, we vary the value size for a comprehensive test.

As shown in Figure 13, on dealing with more complicated transactions of macro-benchmarks, Hercules still yields a higher throughput than SWL_eADR, Kiln, TC, HOOP, Proteus, and FWB by 39.2%, 20.5%, 19.0%, 31.5%, 41.1%, and 41.7% on average, respectively. Furthermore, with a larger value, the performance gap between Hercules and prior designs generally becomes wider. The root cause is Hercules’ robust design. It leverages the sufficient CPU cache to take in transactions. The eADR renders a value almost persistent in CPU cache. A larger value does not greatly increase the cost of persisting the value, especially on the critical path. Comparatively, the double writes of logging again hinder SWL_eADR from achieving high performance. As to Kiln and TC, a larger value imposes more burdens in using FIFO queues and side-path cache, respectively, and triggers more executions through fall-back paths that severely affect their throughputs. For FWB and Proteus, larger transactions still make immense data updates that continually run out of their log buffer and WPQ entries, respectively. As to HOOP, larger values consume more pmem bandwidths and incur longer time for the MC to wait for the completion of data migrations.

The results with running four threads in Figure 15 justify the scalability of Hercules, which produces 43.7%, 20.1%, 27.8%, 31.6%, 42.9%, and 43.7% higher throughput, respectively, than prior designs on average. A multi-level private/shared cache hierarchy with the set-associativity management implies an innate scalability to support multi-threading with multi-cores. Hercules gains scalability accordingly. Threads may share GlobalTxID register at the start of a transaction and eWPQ outside of the critical path for prematurely flushed cache lines. Though, getting a TxID can be swiftly done in an atomic operation while the probability of massive synchronizations under a spacious cache hierarchy is low. For prior designs, the contention on resources between multi-threads is much fiercer. Take HOOP for illustration again. As it depends on the MC for out-of-place data updates, multiple threads contend flushing data through the MC. This offsets the effect of CPU cache for concurrency and makes the MC a busy synchronization point being shared at runtime, thereby limiting HOOP’s scalability.