Invited Paper: Initial Steps Toward a Compiler for Distributed Programs

An ongoing thread in distributed computing is the development of new programming models and languages built on formal models that can simplify the challenges developers face in building distributed software. This thread includes programming languages like Dedalus (Alvaro et al., 2011b), Bloom (Alvaro et al., 2011a; Conway et al., 2012), LVars (Kuper and Newton, 2013), Lasp (Meiklejohn and Van Roy, 2015), Datafun (Arntzenius and Krishnaswami, 2016) and Gallifrey (Milano et al., 2019), as well as data structures like CRDTs (Shapiro et al., 2011) and their realization in libraries like Automerge (Kleppmann and Beresford, 2018).

This short paper is part of an evolution from language design to a full stack for distributed programming. Is it possible to build a language stack (multiple surface languages with shared compilation, debugging and deployment) that addresses the concerns of developers writing distributed programs? How much work can be done automatically? Of what remains, what is amenable to compiler assistance and human review? Can compiled code compete with hand-written code? Can a compiler discover optimizations that humans do not? This paper is an early snapshot of our work in this domain. It does not claim to answer all of these questions, nor even to answer any one of them definitively. We narrow our focus here to automatic compiler transformations, with much of the discussion driven from simple examples. In short, this paper is intended as a progress report and an opening for community engagement.

Traditional optimizing compilers concern themselves with efficient use of computing resources including the various aspects of CPUs, GPUs, memory and interconnects. All of these concerns exist in distributed systems of course, but are augmented by concerns that are endemic to distributed systems: notably communication, partitioning of work across machines, fault tolerance, concurrency and consistency of data and outcomes, and respect for invariants related to security, privacy and governance. We assume that traditional optimizer toolkits like LLVM (Lattner and Adve, 2004) will continue to serve the purposes of optimizing the code that runs on each machine; we focus on the challenge of providing abstractions and compilers for the unique distributed aspects of modern programs.

We ground our discussion in the context of the Hydro project, a multi-year effort at UC Berkeley and Sutter Hill Ventures. We laid out our vision for Hydro in an earlier paper (Cheung et al., 2021), proposing a compiler stack (Figure 1) with multiple components. As the highest level of input to Hydro, we hope to support a multitude of distributed programming styles, much like LLVM provides a compiler stack for a multitude of sequential programming languages. Also like LLVM, Hydro envisions multiple layers of intermediate representation languages (IRs) that can serve as common ground for program checks and transformations, providing developers with various entry points to work deeper in the stack as they see fit. The top layer of Hydro we call Hydraulic: a system to lift low-level code from legacy interfaces into a higher-level declarative distributed IR we dub Hydrologic. The next layer down is an optimizing compiler we call Hydrolysis, which takes Hydrologic and compiles it to run on multiple instances of a single-threaded, asynchronous dataflow IR called Hydroflow. Work toward Hydraulic and Hydrologic is ongoing, with initial results beginning to emerge (Laddad et al., 2022).

The focus for this paper is on Hydrolysis—specifically we want to explore the feasibility of a compiler that can produce Hydroflow programs optimized for different deployment objectives. We consider the potential for a transformation-based compiler that can correctly modify a Hydroflow program, step-by-step, into one or more alternative Hydroflow programs that offer the same results with different performance or deployment characteristics. Like many dataflow systems, Hydroflow is an extension of the relational algebra. As such it is amenable to the kinds of query optimizations pioneered for database systems, as well as a host of optimizations that are apropos for general-purpose distributed programs. Following the model of the widely-used Cascades query optimizer (Graefe, 1995) and the renewed enthusiasm in the programming languages community for e-graphs (Nelson and Oppen, 1980; Willsey et al., 2021), we view query optimization as a problem of program transformation. Basically, we assume we can translate from Hydrologic to some semantically-equivalent single-node Hydroflow program (“execution plan”) in a naive fashion—essentially via parsing Hydrologic without concern for distributed deployment or performance. The compiler’s job then is to search the space of semantically equivalent dataflow programs running on one or more machines, and choose the configuration that is most desirable according to an objective function on various metrics (e.g. cloud costs, runtime performance, resilience to failures, etc.) with various constraints or invariants on how data and communication are to be managed.

In the style of Cascades and egraphs, we envision optimization via repeated application of simple local program transformations, using compact memoization to keep track of prior states and avoid repeated work. The basic loop is this: the current Hydroflow program is stored in a memoization structure, and a local transformation rule (a kind of peephole optimization) is applied to a small segment of the Hydroflow program to get another semantically equivalent Hydroflow program that we have not seen before. In the presence of a cost model and objective function, pruning is applied to the memo structure and search space to keep only those candidates that may participate in an optimal outcome. This process repeats until all possible semantically equivalent dataflows have been generated or ruled out as suboptimal via pruning.

This paper does not deliver on the full vision of the Hydrolysis optimizer, though we sketch next steps in Section 6. Our discussion here is an early demonstration of manually applied transformation rules that achieve useful optimizations for correctly distributing programs across machines. We do not pretend to offer a comprehensive set of such transformations as of yet. Our goal is to document our growing confidence in the potential for an optimizing compiler—specifically one based in a dataflow model—to meaningfully assist in the development of efficient, correct distributed programs.

Dataflow is a widely-used programming model for composing simple data operators into complex programs represented as directed graphs. In many dataflow systems, the operators have formal semantics, often as a superset of the relational algebra. By contrast, the semantics of “edges” in the dataflow are often implicit in an execution model. Typically, the implicit assumption is that an edge from a producer operator P to consumer operator Q (denoted P -> Q) indicates that P delivers a stream of data to Q. This stream semantics implies an ordering constraint: if P delivers data in a certain order, Q will receive it in that order. In Section 4.1 we discuss how this can be formalized, but for now we can imagine that the producer implicitly assigns a sequence number to each item it sends, and the consumer is guaranteed to receive those items in monotonically increasing order of those sequence numbers.

Dataflow operators can also pass data over a network, a special kind of dataflow edge. A “network edge” of this type downgrades the type of communication from ordered streams, by garbling the ordering, batching, and number of transmissions of each item in the stream. If we again imagine implicit sequence numbers at the producer, the consumer has no guarantee of receiving data over a network edge by monotonically increasing position¹¹1Some networking protocols like TCP offer reliable ordered delivery. These protocols are not a panacaea for many applications, however—it’s quite common for TCP sessions to terminate unpredictably. As a result, many long-running services layer their own solutions to ordering and reliability on top of multiple unreliable TCP sessions (Helland, 2012).. In the absence of these guarantees, it is difficult to reason about the consistency of program outcomes across multiple executions. In particular, if a program with network edges is replicated, the replicas may be inconsistent; alternatively, if such a program crashes and is re-executed (e.g. by a recovery protocol) the second run may not match the prefix of outcomes that happened in the initial run.

To address these concerns, various projects across decades of work have explored the idea that certain operators remain consistent even when run on networked edges and/or replicated. If the operators are inherently Associative, Commutative and Idempotent in their handling of input data, they upgrade the dataflow back to consistency of outcomes across networked executions (see (Garcia-Molina and Salem, 1987; Helland and Campbell, 2009; Shapiro et al., 2011; Kuper and Newton, 2013; Meiklejohn and Van Roy, 2015; Milano et al., 2019; Arntzenius and Krishnaswami, 2016), etc.) Mathematically, an operator with these properties is a join semi-lattice (Shapiro et al., 2011) (henceforth we will just use the term “lattice”), and monotonic. These monotonic lattice operators will produce identical outcomes in the face of data arriving in different batches (associativity), orders (commutativity) or multiplicity (idempotence). These properties are compositional, so a dataflow composed of join semi-lattices is itself a composite join semi-lattice. If we can transform our code—whether it be the entire program, or just a component—to use lattices exclusively, we can rest easy about the distinctions between local and networked edges within that scope. The data types will ensure deterministic outcomes across networks²²2The CALM Theorem (Hellerstein and Alvaro, 2020b) proves that this relationship is bidirectional: programs are consistent across unreliable network edges if and only if they provide monotonicity guarantees of the form guaranteed by lattices. CALM was proven in a formal framework of distributed logic programming rather than distributed algebra, so there are still some technical details to be done to apply this argument to a language like Hydroflow, but the intuition is fairly direct..

Given this background, one of the goals of optimization in Hydro is to transform programs to make liberal use of lattice operators. To ensure consistency—-that is, determinism across runs and replicas—code segments using non-lattice operators either must run on a single sequential core, or, if distributed, must establish consensus on the order of operations using a protocol like Paxos (Lamport, 2019), which often negatively affects latency and availability (Hellerstein and Alvaro, 2020b).

Because lattices lie at the heart of our goals of correctly auto-distributing programs, we focus on foundational lattice-oriented transformations in this paper. There are of course many more transformations that are relevant to distributed program optimization. Two that we have explored extensively in implementing Paxos over Hydroflow are auto-decoupling of subprograms and auto-partitioning (sharding) of code and state (Hellerstein, 2023). However here we stay focused on initial optimizations that demonstrate our ability to safely distribute a simple example in multiple ways.

To illustrate the potential for Hydrolysis, we show how a compiler can take a simple single-node Hydroflow program and transform it step-by-step into a semantically equivalent distributed alternative with a clever twist from the literature. Specifically, we consider the classic problem of implementing a shopping cart, inspired by the Amazon Dynamo paper (DeCandia et al., 2007). Our goal will be to step through a sequence of individual transformations in the Hydrolysis search space, as an example of the transformation paths that Hydrolysis would enumerate. All the code we show below runs correctly in Hydroflow, and is available in full at https://github.com/hydro-project/hydroflow/tree/applied23/hydroflow/examples/shopping.

Given a single-node implementation of a shopping cart system, we partition the program between client and server, replicate the servers for fault tolerance, and introduce an optimization from Conway et al. (Conway et al., 2012) to work with lattices throughout—hence allowing not only shopping but also checkout to proceed without using any distributed coordination.

Our naive Hydroflow code for the shopping carts is shown in Figure 2, along with an auto-generated dataflow diagram of the code. We envision two classes of shopping data, one for basic customers and one for “premium” customers. The type of the shopping data streaming in is of the form Stream<ClientLineItem>—an unbounded list of requests. ClientLineItem is a nested pair (client: usize, (item: String, qty: i16)) representing a request from a client with a non-negative integer ID (Rust’s usize type) to add a quantity of a specific item to their shopping cart, or delete a quantity of an item (via negative qty). In this simple program, a shopping cart is similar to string data, but rather than being an ordered list of characters, it is an ordered list of ClientLineItems³³3It is tempting to assume that shopping requests would be better represented as a set than a stream. The problem is that the quantity of each item needs to be handled carefully. Imagine that a customer orders 2 apples, then orders 2 more apples, then deletes 4 apples. In the end they truly want 0 apples. Two problems arise. One is that sets are idempotent, but counting/summing is not. So the following two sets are equivalent $\{(apple\times 2),(apple\times 2)\}=\{(apple\times 2)\}$, but the following streams are not equivalent $[(apple\times 2),(apple\times 2)]\neq[(apple\times 2)]$. The second problem is that the semantics of deletion and insertion may not be commutative: in some applications, we may ignore ”overdrafts” that go below 0. For example, in some definitions, $[(apple\times-4)],[(apple\times 4)]=[(apple\times 4)]$ because the deletion on an empty cart is ignored. Stream semantics ensure these issues are unambiguous..

We begin in Figure 2 with a single-node Hydroflow implementation that we envision being generated naively from a distribution-agnostic Hydrologic spec. We will walk through this code in some detail; subsequent snippets in the paper use substantially the same operators, just reconfigured via various transformations.

In the Hydroflow language, -> represents a stream of data flowing from a producing operator to a consuming operator on a single node. Hydroflow offers a variety of operators familiar from relational algebra and functional languages like Spark or Pandas, including the ability to embed ”user-defined functions” (i.e. arbitrary single-node sequential code) in operators like map and reduce.

Taking the code one line at a time, we begin in Line 1 with a source_stream operator that takes an unbounded stream of packets as they arrive from an IP port (defined in a variable shopping in a Rust prelude to the Hydroflow program) and passes them to the first input (input [0]) of a subgraph called lookup_class. Line 2 begins with a source_iter operator that iterates once through a iterable Rust collection (defined in the variable client_class) in a Rust prelude to the program) and passes the results to the second input (input [1]) of the lookup_class subgraph.

The remaining four lines specify lookup_class. Line 3 specifies a relational equijoin on (key, value) pairs from the two inputs; inputs that match by key are concatenated in an output tuple of the form (key, (value0, value1)). Line 4 is a map function that takes the output of the join and reformats it to suit the next operator in Line 5. This is a SQL-style group_by that accepts tuples of the form (key, value)—in this case the preceding map generates a key of the form (client, class) and a value li.The group_by partitions the data by key, and per key it aggregates (”folds”) the values using a pair of an initialization function (in this case a Rust Vec::new declaring an empty vector) and an iteration function (in this case Vec::push which pushes each tuple to the end of the vector). The result is a stream of tuples, one per distinct key. In Line 6 we have two operators that together do network transmission. The first is a map function to format tuples of the form (payload, destination), and the second a dest_sink_serde that serializes each payload m via internal Rust libraries and ships each one to the Hydroflow node at the destination out_addr (a Rust variable defined in the prelude).

More intuitively, this flow iterates through customer requests via the source_stream operator. It then does a join with a stored client_class table to look up a unique ClientClass tag for each client via the join operator, and loads the tagged shopping requests into the stateful group_by operator. The group_by operator is initialized with an empty vector (generated by Vec::new) which it accumulates by pushing each LineItem that arrives to the end of the list. The result is tagged via map with a (externally-provided) destination address out_addr and sent over the network in serialized form by dest_sink_serde.

The shopping stream grows monotonically without bound. This means that the group_by operator is never able to assemble a ”final” answer for a group; even if we hacked it to “time out”, at best it could output a “string prefix” (lower bound) of future answers. If we were to allow the group_by to pass values to the network, then the external agent at out_addr could see non-deterministic prefixes of the shopping cart. If we replicated this code to another node in the system, it might make different choices about which group_by values to send out on the network, leading to inconsistency.

For correctness, then, the group_by in this case can only output final results for each client “at the end of time”—i.e. when some operational semantics of the system determines that the stream flowing into the group_by will never produce more data. While we could augment our program with another channel for client “checkout” messages, that would still not help our dataflow system understand the application semantics of when to release the group_by data, because checkouts could race with orders! Instead, we’d like to capture “end-of-stream” explicitly in our type system so the developer can inform the group_by operator how to reason formally about safe release of outputs. We address this issue in the next section.

In this paper, we consider two kinds of transformations: type upgrades (this section) and local graph transformations (next section).

The type upgrades we seek are all bounded join semi-lattice types. As discussed in Section 2, if we can convert to lattice types for our operators, we can override the problems introduced by networks. But we want more than just join semi-lattices; we want bounded join semi-lattices (henceforth “bounded lattices”) that have a well-defined finite ”top” element $\top$. This element at the top of the lattice has the property that once the operator’s output reaches $\top$, it will remain $\top$ in the face of any new input. This allows the operator to output the value $\top$ at any time, without fear of future retraction⁴⁴4Bounded semi-lattices may be the only reasonable data types to transmit across a network. Indeed, many consistency tricks in the distributed systems literature attach lattice metadata to objects in the dataflow; TCP sequence numbers, Lamport clocks and vector clocks are three common examples. In effect this metadata “upgrades” the network to use lattices, but this is not typically reflected in a type system for a compiler or debugger to reason about!.

We now proceed to show how Hydrolysis might transform our program to “upgrade” the types to bounded lattices.

In this section we examing some dataflow graph transformations that, in concert with our lattice-typed shopping carts, allow us to deliver a fully monotonic, lattice-based implementation of our program. This in turn enables graph transformations for safely distributing the program without any coordination.

This paper captures a snapshot of our early explorations of the potential of dataflow optimization as a vehicle for optimizing distributed programs. We are increasingly optimistic that a dataflow kernel like Hydroflow is a useful optimization target for distributed systems concerns. By incorporating the ACI properties of lattices into our type system we can reason about allowing network communication to be introduced safely into programs. As illustrated in Section 4 we are beginning to see the potential for an optimizer to automatically “upgrade” programs to use lattice types without changing semantics. This in turn opens up opportunities for decoupling and replication of program components across networks.

In Section 5.2 we saw that two different choices of program transformations can address different objectives: in that case a tradeoff between fault tolerance on one hand, and governance/privacy on another. This suggests that the objective function and constraints for optimizing modern distributed programs may be quite a bit more varied and nuanced than classical query compilation.

This workshop paper is early, and we are eagerly pursuing a number of directions to go from these concepts and hand-optimizations to a rich, automated reality. The agenda encompasses a range of challenges, including the following.

1. (1)

   We need a language for our own use to formalize our rewrite rules and prove equivalence of rewritten program fragments.

2. (2)

   We need to register a large number of transformation rules. This likely needs to include many classical examples from relational database query optimization, functional programming and stream query processing. In addition, we expect to trip across new optimizations that address issues with cloud deployments.

3. (3)

   We need a way for programmers to define multiple objectives they want to optimize, and to express constraints on the optimization space, e.g. for fault tolerance or governance. In the vision paper for Hydro (Cheung et al., 2021) we highlight fault tolerance as a programming “aspect” that developers should be able to specify independent of their program’s intended semantics. That vision requires further work, but some seeds are apparent even in our simple example here—namely the ability to consistently replicate components. Simultaneously maintaining fault tolerance constraints and performance objectives is an interesting challenge for an optimizer like Hydrolysis.

4. (4)

   We need a transformation-based optimizer that can ingest our rules, objective functions and constraints, and efficiently search the space of equivalent programs to minimize the objective function. We are enthusiastic that recent work on e-graphs could offer an efficient vehicle for our work.

We are optimistic that open-source tools like Egg (Willsey et al., 2021) can make it possible for us to address these challenges relatively quickly. The Hydro stack itself is also open source, and we welcome additional research and development efforts!