Jut: A Framework for Just-in-Time Data Access

In this section, we argue that enabling JIT-DA must start by examining the data processing architecture that typically underlies data-driven apps. Realizing JIT-DA with existing data processing systems is hard for three key reasons.

In this paper we present Jut, a new framework for building JIT-DA apps that addresses the above challenges. Our approach starts with the observation that the abstractions (e.g., devices, rooms, buildings) and information (e.g., when a device is present in a room) needed to address some of our challenges can be found in the application layer. The problem is that these abstractions/information are absent or not well represented in the underlying data processing systems. Moreover, there is no way to systematically and automatically reflect this app-level information in the underlying data processing layer; e.g., when the app detects a new device (described below), this should automatically trigger the app’s data processing pipeline to consider ingesting data from the device’s APIs.

Our framework thus adapts existing data architectures as follows. First, we modularize the data processing layer to reflect the natural modularity found in smart-space apps. Thus, instead of one monolithic data processing pipeline, in Jut, every entity – whether a device data source, a room, or a building – has an independent data processing pipeline that implements an ingest dataflow (i.e., processing input data relevant to that entity), storage, and an egress dataflow (i.e., for exporting data from its store). We refer to an entity modeled in this way as a context.

Next, we extend the data processing architecture to expose a new join(C1, C2) interface (and corresponding leave()) via which an application can inform its underlying data processing system that a context C1 is available to serve as a data source for a context C2. The implementation of join connects C1’s egress pipeline to C2’s ingestion pipeline. Importantly, join does so in a manner that enforces their respective policies (e.g., that $C1$ is from a trusted vendor) and reconciles any mismatch in their heterogeneous data representations (e.g., converting from $C1$’s data format to that required by $C2$’s storage system). Thus a key contribution in Jut is a novel approach that, at runtime, compiles high-level policies into low-level functions for processing newly discovered data source. In effect, join allows applications to "plumb" contexts, to reflect the (dynamically evolving) relationship between JIT data sources and their physical space or context.

Taken together, the above changes address the challenges from §1.1. The modularity of contexts means we can develop and operate context code independently, supporting decentralized ownership and control. In addition, this modularity allows us to efficiently implement “by context” queries, since a context’s data is curated in its datastore (vs. requiring additional steps to first identify data relevant to a context). Finally, the join interface enables data processing pipelines that adapt to reflect the complex interactions between data sources and consumers in the physical world.

Jut thus provides general yet powerful abstractions that simplify building (smart-space) apps. Specifically, an app developer interacts with the Jut framework to create contexts, compose, and query them. Because contexts are an explicit yet modular and composable abstraction, developers can: (i) treat contexts themselves as data sources that can be queried, logged, configured, etc., (ii) can compose contexts into higher level ones (e.g., rooms as data sources in a building context); (iii) reuse contexts to accelerate development, and so forth.

Jut’s changes do not impact the internal techniques of current data processing systems: e.g., their storage formats, query planners, optimizers. Instead, the novelty that Jut brings is a shift in the system architecture of data processing systems: introducing a new modularity (based on contexts) and a new, more systematic and structured, integration between data and app layers (based on joining contexts).

In this paper, we present the design and implementation of the Jut framework that implements the above design approach. We evaluate Jut by using it to implement a range of smart-space scenarios using real physical devices and spaces. Our results show that Jut can implement use-cases that are not easily supported today, or can do so with 3.2-14.8$\times$ less development effort and 3-12$\times$ lower query complexity than current commercial [13] and research [58] systems.

The remainder of this paper is organized as follows: we elaborate on our goals and assumptions in §2, then present Jut’s design and implementation in §3 and §4 respectively. We evaluate Jut in §5 and then conclude.

Contexts impose modularity in the data processing layer, reflecting the modularity commonly found in the app layer.

As mentioned earlier, contexts can be composed which allows data to flow from the egress of one context to the ingress of another: e.g., the egress of a user device might be composed with the ingress of a room context and, simultaneously, the egress of the room context might be composed with the ingress of the building context.

The join (leave) API is how such composition (decomposition) is realized. As described in §2, we assume that two applications can discover one another through some service discovery mechanism.

For illustration, we’ll consider two applications $A_{user}$ and $A_{bldg}$ that have discovered each other: e.g., in our canonical scenario, $A_{user}$ might be an app on a user’s phone that exports sensor readings and other information from the phone, while $A_{bldg}$ is the building app for the building that the user just entered. Let’s say that $C_{user}$ and $C_{bldg}$ are the contexts associated with $A_{u}ser$ and $A_{bldg}$ respectively. As part of the discovery process, we’ll assume that $A_{user}$ and $A_{bldg}$ exchange information about their respective contexts and focus on the actions taken at $A_{bldg}$; processing at $A_{user}$ to incorporate $C_{bldg}$ as a data source in $C_{user}$ follows along the same lines. We assume that $A_{bldg}$ will first perform any application-level security and policy checks deemed necessary and, assuming these are satisfied, $A_{bldg}$ then invokes its underlying Jut layer’s join interface thus letting $C_{bldg}$ know that $C_{user}$ is available as a potential data source.

To implement join, the Jut runtime first checks whether $C_{user}$ is a potential data source for $C_{bldg}$ by comparing $C_{user}$’s name and kind fields against $C_{bldg}$’s sourcing intents. In case of a match, it then checks each of $C_{user}$’s egress schemas against each of $C_{bldg}$’s ingress match:action tables. If the schema for one of $C_{user}$’s egresses, denoted $C_{user}.egress.idx$ matches a rule in the match:action table for $C_{bldg}$’s ingress $C_{bldg}.ingress.idy$ then the corresponding actions are compiled into dataflow operators and the generated dataflow is prepended to the existing dataflow for $C_{bldg}.ingress.idy$ and $C_{user}.egresss.idx$ is added as a data source for $C_{bldg}$. For instance, given a schema <measurement:watt:string>, the first rule from the table above is matched, and a dataflow is added to $ingress$’s dataflow that extracts the field watt from data records matching the schema.

In summary, using the above join (and corresponding leave) interface we can “plumb" data sources to contexts, and contexts to other contexts. Importantly, the use of these interfaces is not limited to JIT sources; rather developers can use them to plumb any data source to the context - whether a JIT phone, a static building sensor, or another known context. Thus, using join/leave gives developers a uniform approach to setting up data processing pipelines that capture the potentially complex and evolving association between data sources and context apps.

Now that we’ve explained how contexts and join work, we briefly discuss the role of naming. Jut does not require that data sources adopt a single naming scheme but it certainly works better if data sources consolidate around a smaller number of approaches. Specifically, in deciding whether and how to ingest data from a source, Jut uses a context’s name and kind fields, together with the data schemas for its egresses. Any of these are optional and developers can always choose whether to accept or reject data with an unknown name, type, or schema. Likewise, the developer of a context can always choose whether to embrace an existing naming scheme or define their own. In practice, we expect that a small number of naming schemes will emerge as defacto standards with widely available libraries to convert between them. This is, in fact, how the world of data works even today with defacto formats such as JSON, Parquet, etc.; protocols such as Matter for unified device communication [32]; Project Haystack [37] and Brick for building ontologies [52], and so forth. Jut’s contribution is in taking these techniques (formats, schema converters, etc.) from the database literature and showing how to automatically and systematically insert them into the processing pipeline as new JIT sources appear/disappear.

More generally, bridging the syntactic/semantic gaps between data representations has historically been achieved by arriving at a middleground between rigid standardization on the one hand, and extreme customization on the other. Jut does not change the fundamentals of these tradeoffs and instead merely provides a flexible framework within which we can arrive at this desired middleground.

{outline}

We now step through the “life of data” in a Jut implementation of our canonical scenario of a user with a mobile phone interacting with a building app.

In addition to any app-level security and access control, Jut supports access control at the context level. We achieve this using standard role-based access control techniques for which each context is associated with role metadata that gets checked at join time. Since our RBAC design is fairly standard, we omit further details.

Beyond JIT data access, we have found that the context abstraction facilitates a number of operational tasks. For example, having a per-context ingress/egress makes it easy to implement policies for tracking provenance, since we can easily log statistics about when and where data was consumed from different sources. Similarly, we can support diverse compliance needs by tailoring how data is stored or replicated for each context; e.g., ensuring different standards of encryption and backup for data collected in a public vs. private building.

Our Jut prototype described in the following section implements RBAC, data provenance, logging, and replication.

As shown in Figure 1, Jut integrates with existing systems that implement data processing and application-layer logic. We briefly describe our system choices for these components.

Fig. 2 illustrates the runtime architecture of Jut, comprising the following components:

We implement a pipelet data sync agent in the runtime that continuously pulls data from a data pool on the Zed lake, processes data with the Zed dataflow, and loads it to another data pool on the Zed lake. For each egress-to-ingress pair and data-store-to-egress pair in context, Jut creates an instance of the pipelet agent to plumb the data flow.

Our Jut prototype consists of 2,393 SLOC of Python and 815 SLOC of Go. The Jut programming library, code generators, join/leave extension, and app are written in Python, while the pipelet data sync agent is written in Go. Besides, we developed a Go-based CLI in $\approx$600 SLOC for Jut that includes APIs for context deployment, configuration, and interaction with Zed and dSpace. To implement device connectors for proxied ingestion, we rely on vendor libraries [12, 18, 17], which account for $\approx$400 SLOC in Python for IoT devices and laptops, and $\approx$550 SLOC in Swift for smartphones [12].

Jut exposes a declarative interface built on Kubernetes [3]. This interface allows context configurations (metadata, ingress, and egress) to be represented as attribute-value pairs, which are stored as API objects on the Kubernetes apiserver [10]. We expose context configurations over this interface so that developers and operators can reuse existing Kubernetes tooling to handle configurations. Developers or context operators can specify the ingresses and egresses on the interface. We present the details of the programming interface in the Appendix.

Developers use the join/leave() API in the Jut programming library to inform the context about availability of new data sources. They use the Zed dataflow language [50, 65] to specify the dataflow operators in each context’s ingress and egress. Zed provides convenient operators such as filtering and cleaning data using type information (§5.2). Users can query the egresses of contexts using the same dataflow language. For example, a building administrator can query the BioHall occupancy by running jut query BioHall.egress.occupancy "avg()" (jut is the CLI). Developers can also use the query API to run a query against the context in the JIT-DA apps; or use the @on.context(egress) (implemented as a Python decorator) to watch and process the data streams continuously, which is useful when implementing data-driven automation. Besides, an app can load data to the $C.store$ by calling context.load(..) with the data records, which can be used for ingesting data from sources that Jut’s ingress doesn’t already support, such as devices that communicate via custom device driver/libraries.

Our overall evaluation approach is as follows. Using a set of real-world physical devices, we construct a series of smart-space scenarios, each designed to highlight a particular aspect of Jut’s design (e.g., querying, implementing policy). We evaluate the complexity of this process using the metrics defined below. In each case, we then attempt to implement the same scenario using two existing systems that we select as our baseline and again quantify the complexity of the experience using the same metrics. In what follows, we introduce our devices, metrics, and baselines.

Unless otherwise mentioned, we run the various software components (Jut runtime, Zed lake, dSpace, and the baselines) using Kubernetes (v1.21.0) in a Thinkcentre M720 machine (Intel Core i5-8400T, 6 cores).

In counting SLOC, we include all (non runtime) code needed to implement the above contexts but leave out the code specific to converting from a native API format to Zed’s format. We omit this code because it is incurred by any solution including our baselines. Moreover, this overhead is somewhat artificial since it depends on the (native and egress) schemas we define, rather than fundamental differences in framework abstractions.

We implemented 7 different smart-space scenarios using the above devices and contexts. Our results showing the SLOC and Qcx for each scenario are summarized in Table 4. Additional detail including code examples can be found in our Appendix.

To set up this experiment, we installed devices in 4 rooms/offices, where each room has one contact sensor, two motion detectors, two lamps, and one appliance; Each room has its own WiFi access point (AP). We reuse this room setup in subsequent scenarios.

We then ran three types of queries over the room contexts: (i) querying an individual room (for its current occupancy or energy usage), (ii) querying across multiple rooms (to rank rooms by their occupancy or energy usage), and (iii) querying across a room’s different egresses (to identify the correlation between a room’s energy usage and occupancy). For example, for (ii), we rank and find the least used room with:

> jut query room1.egress.occupancy "head | sort occupancy"}
Implementing this scenario required 25 SLOC, most of which were to implement the necessary contexts. Composing data sources required few SLOC - calls to \join and, as necessary, configuring context match:action rules.
In terms of query complexity: implementing the above queries with \arch required a total of 18 Qcx.
By contrast, using dSpace with Postgres, we first need to manually create a table that stores the mapping between devices and their room. Given this, we can then run this query but it is still inefficient in Qcx. The same queries required 159 Qcx because dSpace stores input data in per-device Postgres tables. The user thus first has to filter table rows that belong to the room of interest and additionally aggregate those rows over time to derive the room occupancy values (detailed queries are in our Appendix). By contrast, in \arch, the relevant data is already selected and stored in the room context’s store which can be directly queried.
\noindent \underline

S2: Context as a data source Now consider that we want to enable building-level energy and occupancy analysis. A convenient approach is to compose the room contexts to a building context that computes and exposes building-level data from the room-level data. Implementing this in Jut took an additional 19 SLOC (over S1) to implement the building context: we specify the room’s occupancy and energy egresses as the building’s ingresses and the building’s ingress dataflow aggregate input data as it arrives. Querying for building occupancy or energy are then straightforward queries to room1.egress.occupancy or room1.egress.energy, respectively, with a Qcx of 6. Implementing this query with Postgres incurs a higher Qcx for similar reasons as above. S3: Ingestion from JIT sources This scenario focuses on Jut’s ability to opportunistically ingest data from user devices. Here, our smart campus app allows people to identify rooms with both good network connectivity. To achieve this, it consumes network quality data from tenants’ phones and laptops. We now extend the scenario from above to include phones and laptops as potential data sources for a room. Our user phone app is configured to join and leave rooms upon detecting a change in the BSSID for the WiFi AP it is currently connected to. Each time the phone changes APs, it takes a network measurement in the background using the NDT tool [33, 34] and reports the raw data to the phone’s context (exposed by phone.egress.netTest). We add an ingress to the room context to ingest data from phone.egress.netTest, and an egress room.egress.netSpeed that exports the average (across all phones/laptops) download/upload bandwidth data to the building context, and so on. We repeat the same process for the laptop and its context. Implementing the above scenario in Jut was straightforward, because much of the low-level "plumbing" code is encapsulated within the join API. Thus supporting this scenario required creating a context for the phone/laptop with a phone.egress.netTest egress; this took 3 SLOC each. Similarly, adding the room.egress.netSpeed egress to the room context required 5 SLOC and 1 match:action rule to room’s ingress. To find rooms with good connectivity, users simply query the room for their average connectivity quality, which is achieved with a Qcx of 6. By contrast, this scenario cannot be easily supported in AWS-IoT or dSpace. In order to approximate JIT-DA in AWS-IoT we hack together a solution as follows. AWS-IoT allows devices (together with their schemas and other metadata information) to be registered in a device registry. We create an entry for our phone and laptop devices and manually hardcode an associated room identifier (RoomId) for each device (deviceID). We then write a script that, when it receives an app-level signal that a device is in the room, takes the following steps: (i) provision a new table in AWS TimeStream for that device, (ii) updates the AWS message broker with a new topic corresponding to the deviceID in the device’s data stream such that all records from the device are routed to the new table, (iii) set up a rule associated with the topic that allows incoming records to be processed prior to storage in TimeStream; this rule adds a new field to each data record into which we write the RoomId corresponding to deviceID, retrieved from the device registry. (Note that supporting user mobility would require additional techniques to dynamically update the deviceID-to-roomID mapping in the device registry.) Implementing the above took 178 SLOC. A query to compute the average connectivity quality of the room then requires averaging the appropriate rows from each device’s table, for all devices that might have been in the room. Writing such a query involved a Qcx of 43. S4: Improved app-data integration. We focus now on the ease with which applications can leverage insights from JIT data: i.e., for data-driven automation. For this scenario, our goal is to support two tasks. First, we want a room’s brightness level to be automatically set when the door opens, where the level of brightness is computed from the historical data for that room. Second, as a security measure, we want an app that automatically sends an SMS to a home owner when room occupancy exceeds a specified threshold. In Jut, the above automation rules are specific in dSpace but rely on queries made to the underlying Jut data processing system. Writing these queries in Jut is straightforward because the abstraction over which dSpace expresses the above automation logic – i.e., a room – is clearly mirrored in the underlying data storage. Hence, we simply write dSpace’s automation logic to make continuous queries to the relevant room context - this took 14 SLOC (in dSpace) and 11 Qcx to query the room brightness and occupancy. By contrast, implementing the same over Postgres took 3x more Qcx and 3.2x more SLOC for reasons similar to those discussed above (provisioning and querying over multiple per-device tables). S5: Handling heterogeneous data. Our last scenario highlights the issue of heterogeneous data sources. Our goal here is a smart-app that wants to allow users to track their personal carbon footprint: i.e., allowing users to track the energy consumption of (say) lamps, heaters, etc. that they encounter at different locations on campus during their stay. To achieve this, the app must be able to opportunistically consume data from a heterogeneous set of (device, space, building )contexts as they move. To implement this scenario, we configure the match:action rules on the phone context to ingest and aggregate readings from any available data source with a *.egress.energy These data may have heterogeneous and unexpected/unknown schemas, such as data types (e.g., string vs. float), names (energy” vs. power”), or units (e.g., watt” vs. kilowatts”). For example: ⬇ {watt:"80",from:"biolab",event_ts:..,ts:..} {watt:100,from:"office",event_ts:..,ts:..} {power:120.,unit:"watt",from:"lounge",event_ts:..,ts:..} Contexts in Jut are able to ingest these heterogeneous data records. At the phone context’s ingress, we clean and convert the energy readings to the same data type and unit with: ⬇ flow: "rename watt:=power | shape(this, <{watt:float64}>) | cut watt,event_ts,from" Querying for a user’s carbon footprint is then a trivial query to the energy egress of the user’s phone context. We implemented this in Jut with 8 SLOC and a Qcx of 4. In summary, by introducing the modularity of contexts and by systematizing how new sources are added/removed to data pipelines, Jut achieves a significant reduction in both development effort (3.2-14.8x SLOC) and query complexity (3-12x Qcx) relative to AWS-IoT and dSpace.

We aim to answer three performance-related questions for Jut: (1) How does Jut’s context-oriented approach impact query performance compared to device-oriented approaches? (2) What is the overhead of using Jut for data-driven insights in the existing applications? and (3) Can Jut scale to a large number of contexts with our implementation choices?