A Survey on User-Space Storage and Its Implementations

Figure 3 shows the system structure of a representative operating system (UNIX [48]) from a user process to flash media. As shown in the figure, an I/O service is initialized by the user-space applications. It is then handled by the page cache, file system, multi-queue block layer and NVMe driver residing the kernel space.

The designs of discrete kernel and user spaces can provide the traditional computing system with guaranteed isolation, security and consistency. However, this tight collaboration between the user and kernel spaces incurs frequent user-kernel mode switches, which introduces extra long latency. Specifically, [10] reports that the user-kernel mode switch costs 2$\sim$4 us latency. Considering that the I/O latency of the stale HDD-based storage system is millisecond-scale, the software latency caused by the context switch is relatively minor. However, the I/O latencies of the state-of-the-art storage devices (e.g., SCM and high-performance NVMe SSDs) have decreased to less than 10 us, which in turn exacerbates the software penalty. In addition, multiple prior research [66, 67, 68, 69] reveal that the storage stack, including virtual memory, file system, and storage drivers, imposes huge software overheads to the high-performance storage devices. For example, the storage stack requires multiple address translation and boundary checks, which cost around 10 us latency.

To address the large overheads imposed by the tedious software stack of I/O services, there is continuous research on constructing a direct expressway between user space and the I/O devices. We review the prior works in the past decade aiming to address the software overhead issue in the storage system. In general, most prior works concentrate on three system components: storage driver, virtualization technique, and file system, which are shown in Figure 1. Specifically, from the aspects of storage drivers, researchers propose to enable applications to directly access the underlying storage in user space by either moving drivers from the kernel to the user space or exploiting the unique features of the state-of-the-art storage drivers (e.g., NVMe drivers). Virtualization-centric works map storage devices to user space and expose virtual interfaces to the applications, which are thereby permitted to access storage via such interfaces. Lastly, the research of file systems propose to reassign tasks (e.g., metadata read/write and I/O permission checks) among the user space, the kernel space, and firmware so as to break free from the stale designs of traditional file system and put forward novel mechanisms to achieve user-space direct access. Moreover, for various scenarios that call for fast access to storage (e.g., high performance computing), researchers have proposed several applications that are optimized for the emerging storage devices. Table I lists our classification of the relevant researches based on the core techniques or the application themes of each research.

The traditional storage system suffers from software overheads such as context switch and metadata management. In order to reduce such overheads and also to limit kernel involvement, researchers have paid great attention to the virtualization technique [70]. Based on the specific storage component that is being virtualized in each work, we divide prior works on virtualization into three major groups: device virtualization, storage virtualization and NVM virtualization. We show the general system structures of each virtualization technique in Figure 4.

Peter et al. [14, 15, 16] proposes a specific device model named Arrakis for virtualized I/O, which abstracts the underlying hardware devices as virtual device instances in user space. Specifically, Arrakis presents the storage devices as virtual storage interface cards (VSICs), which function in the same way as their corresponding physical devices from the perspective of users. Arrakis also proposes to handle user-level I/O requests efficiently by splitting the roles of operating systems into data plane and control plane. The operations of data plane are associated with data accessing, such as asynchronous reads and writes. The control plane, on the other hand, manages the core tasks of a system for reliability and security, such as I/O permission checking and hardware resource allocation. For control plane, the device drivers in the kernel manage VSICs and the physical devices via a set of operations such as device creation and destruction. These operations do not interfere with data-plane operations during runtime. For data-plane operations, user-level applications can directly issue I/O requests to VSICs during the execution, which, with the support of hardwares (e.g., DMA controller), directly associates with underlying storage. To this end, thanks to the split of data plane and control plane, user-space application attains direct access to the underlying storage devices without the involvement of the kernel.

Moti et al. [25] designs a user-space file system named Simurgh, which bases its core design on virtualizing NVM. Considering that NVM achieves similar performance to DRAM and is byte-addressable, Simurgh directly maps NVM into the address space of each application without employing DRAM to cache data and metadata from NVM. In this way, neither data copy between NVM and DRAM nor data buffering in DRAM is necessary. Thus, NVM can be accessed directly in user space without DRAM as the medium. Since the metadata is cached in DRAM, it is possible for metadata to be accessed concurrently by independent processes. Moreover, Simurgh adds two additional instructions (i.e., protected jump and return) to the CPU ISA to securely execute user-space functions in privileged mode, which in turn reduces the kernel’s involvement during runtime.

Aside from embedding NVM mapping in file system design (e.g., Simurgh), researchers have also proposed more light-weighted tools (i.e., user-level libraries) for NVM mapping. Eisner et al. [26] proposes Quill, a user-level library specified for accessing NVM block devices. Quill targets at addressing the challenges imposed by the stale memory management mechanism (i.e., paging). Paging is a memory management technique, which retrieves data from secondary storage devices (e.g., SSD, NVM) and stores them to the main memory (e.g., DRAM) for further usage. However, considering that the performance behaviors of NVM (i.e., latency and bandwidth) are comparable to DRAM, data copy from NVM to DRAM cannot benefit from the high speed of DRAM, thus making paging inefficient for NVM. Observing this feature, Quill tries to avoid paging of NVM block devices. To this end, since NVMs can be cached inside CPUs, they are organized as pages in CPUs’ physical address space directly. When a user-level application accesses a file, Quill takes it over and uses the mmap() function to map the physical pages of NVM devices to the virtual address space of user-space applications. The files stored in NVM are permitted to be accessed directly by the application, thus eliminating the necessity of paging.

Rather than solely focusing on the latency and bandwidth properties of NVM, Chou et al. [27, 28] designs a user-level library named vNVML, which explores utilization of other features of NVM. vNVML exploits the byte-addressable feature of NVM by providing a conventional file interface mmaping, which enables applications to access data through byte-level load/store instructions. Moreover, vNVML extends the available physical storage of virtual NVM by integrating DRAM and underlying storage devices (e.g., SSDs) into its design. vNVML leverages DRAM and NVM as the cache to the underlying storage devices. Data read from the underlying storage are cached in DRAM, and NVM functions as both the log buffer and the write cache. When the user-level application issues read or write requests to the virtual NVM, it is actually accessing the cache to underlying storage, which owns much larger physical storage space than NVM. Therefore, the virtual NVM is much larger from the users’ perspective.

While the prior work of the storage driver and the virtualization techniques enable the user-space application directly access data from storage, the kernel still plays a significant role in the entire computing system. Specifically, the kernel needs to control the behaviors of storage drivers and maintain the mapping from storage to user-space address spaces. Since file systems play an essential role in accessing underlying storage, other than simply concentrating on one particular aspect of the storage system, such as the storage driver or the virtualization techniques, researchers have committed to redesigning file systems so as to minimize the kernel’s involvement in data accessing.

General file systems reside in kernel space. User applications can interact with the file systems via customized system calls (i.e., open). However, developing file systems in kernel can be complicated and challenging because kernel codes are deeply coupled and its architecture is complex. Therefore, user-space file systems have attracted researchers attention since they are easier to develop. In this subsection, we mainly review the works focusing on user-space file systems to obtain better performance in accessing storage devices, including the widely used framework called Filesystem in Userspace (FUSE), FUSE-based designs and other user-space file systems, which focus to one particular types of storage media (e.g., NVM and NVMe SSDs).

Specifically, FUSE consists of modules residing in both the user and kernel spaces. The kernel module is registered as a file system driver also named fuse. It manages kernel operations with the virtual file system (VFS), an interface that supports concrete file systems. The user-space modules, including the libfuse library and the fuse daemon, are responsible for setting up the file system in user space and handling file system calls. Using FUSE to implement a new file system requires users to write a handler program, which specifies the behavior of the file system in responding to read/write requests from user space and links such program to the libfuse library. After FUSE is mounted, this handler program is registered by the kernel module (i.e., fuse) for handling runtime data requests.

The process of executing a system call in FUSE can be regarded as a client-server model, where the kernel fuse module is the client and the fuse daemon in user space is the server. When a system call, such as read() or write(), is issued from user space, the VFS first invokes the default handler in the fuse to fetch data in the page cache. The requested data is returned directly to user space if found in page cache. Only one user-kernel context switch is required here. If not found, the system call is thereafter re-directed to the user-space module libfuse. libfuse invokes the user-defined handler in the kernel, which fetches the data according to the specification of the user and returns the fetched data back to user space. In this process, two user-kernel context switches are needed, as shown in Figure 5. The kernel forwards the system call to libfuse, which later invokes the handler in the kernel, and the kernel returns the requested data back to the user space.

By providing the software interface (i.e., libfuse), FUSE allows users to develop customized file systems conveniently. Moreover, FUSE moves parts of code execution from kernel to user space. Such reduction of code execution in kernel would reduce the possibility of kernel crashing. In addition, with libfuse residing in user space and easy to be deployed in different environments, it is effortless to transplant FUSE-based file systems from one environment to another, which in turn makes FUSE achieve compatibility.

Ishiguro et al. in 2012 [29] modifies the kernel module of FUSE to achieve performance improvement. They redesign a mechanism that ports the fuse daemon to the kernel and modify the kernel fuse module to adapt to this mechanism. Thus, all the works that used to be done in the user space are handled by the kernel. This mechanism avoids redundant context switches and memory copies. Zhu et al. [18], on the other hand, focused on different types of FUSE. They describes a FUSE-based framework called Direct-FUSE to support multiple backend file systems without trapping into kernel. Direct-FUSE divides the FUSE framework into three layers including bottom, middle and top layers. The bottom layer (also known as the backend services) provides operations of multiple file systems while the middle layer provides unified file system interfaces for various backend services. Lastly, the top layer receives file system operations and identifies the corresponding backend service for each operation. All the layers of Direct-FUSE are in user space, thereby eliminating non-trivial overheads of user-kernel context switches. Apart from minimizing the heavy overheads imposed by context switches, Huai et al. [58] puts forward a new user-space file system infrastructure called XFUSE. XFUSE targets at improving FUSE performance by adapting FUSE to modern multi-core CPU systems and the emerging storage devices (e.g., NVM). To improve the parallelism, XFUSE enables multiple file system daemon threads handling different requests in parallel, thereby increasing the scalability in multi-core systems. To meet the speed of fast storage devices, it dynamically adjusts the period of busy-waiting according to the I/O requests, thus avoiding the unnecessary waits and improving the I/O throughput.

NVMe protocol has become the dominant storage interface, which can deliver low-latency and highly parallel access to the underlying NVM media (i.e., SSDs). NVMe-based SSDs bring about many advantages, including massive I/O queues and faster speed than traditional SSDs. These advantages have raised new challenges to the design of user-space file systems since they play the major role in accessing NVMe SSDs. Yoshimura et al. [19] and Tu et al. [65] propose EvFS and URFS, respectively. They both are user-level file systems that aim to fully exploit the advantages of NVMe SSDs. EvFS employs SPDK [11] to support various NVMe devices.Specifically, EvFS adopts the event-driven execution model in SPDK for file I/O processing, targeting at the communication between users and the page cache in EvFS. Users submit the I/O operations as events to invoke the user-level NVMe driver in SPDK directly, which highly utilizes the bandwidth of NVM with few user threads and provides low I/O latency. URFS, on the other hand, explores the parallelism provided by NVMe. It utilizes a series of memory sharing and protection mechanisms to share storage accesses among multiple I/O processes. Applications that are executed upon URFS can share multiple NVMe SSDs via a library named fslib, which provides POSIX-like file system APIs for the convenience of developers.

Instead of being constrained to the framework provided by FUSE, researchers redesigned user space file systems from scratch and adopt more aggressive modifications, so that the new file systems fit to new storage devices better. In this subsection, we will review multiple prior file system designs with special focus on the re-organized duties of kernel space and user space. By examining prior file system designs with respect to the emerging storage techniques, we divide these works into two categories according to the roles of kernel. As shown in Figure 6, several designs [17, 30, 21, 22] rely on kernel to maintain metadata integrity, security, consistency and etc. The kernel is also in charge of performing permission checks on each data request sent from the user space. For these designs, kernel is responsible for control-plane operations. Other designs [31, 24, 23, 32], as shown in Figure 7, in contrast, only delegate few auxiliary responsibilities to the kernel. Namely, most frequently used functions (e.g., data read/write) are migrated to user space or device firmware while kernel serves as the auxiliary plane.