Transient Typechecks are (Almost) Free [Preliminary Draft 2019-02-18]

Grace is an object-oriented, imperative, educational programming language, with a focus on introductory programming courses, but also intended for more advanced study and research[8, 18]. While Grace’s syntax draws from the so-called “curly bracket” tradition of C, Java, and JavaScript, the structure of the language is in many ways closer to Smalltalk: all computation is via dynamically dispatched “method requests” where the object receiving the request decides which code to run, and returns within lambdas that are “non-local”, returning to the method activation in which the block is instantiated[27]. In other ways, Grace is closer to JavaScript than Smalltalk: Grace objects can be created from object literals, rather than by instantiating classes[9, 33] and objects and classes can be deeply nested within each other[35].

Critically, Grace’s declarations and methods’ arguments and results can be annotated with types, and those types can be checked either statically or dynamically. This means the type system is intrinsically gradual: type annotations should not affect the semantics of a correct program[47], and the type system includes a distinguished “Unknown” type which matches any other type and is the implicit type for untyped program parts.

The static core of Grace’s type system is well described elsewhere[32]; here we explain how these types can be understood dynamically, from the Grace programmer’s point of view. Grace’s types are structural [8], that is, an object implements a type whenever it implements all the methods required by that type, rather than requiring classes or objects to declare types explicitly. Methods match when they have the same name and arity: argument and return types are ignored. A type thus expresses the requests an object can respond to, for example whether a particular accessor is available, rather than a nominal location in an explicit inheritance hierarchy.

Grace then checks the types of values at run-time:

• •

  the values of arguments are checked after a method is requested, but before the body of the method is executed;

• •

  the value returned by a method is checked after its body is executed; and

• •

  the values of variables are checked whenever written or read by user code.

In the spectrum of gradual typing, these semantics are closest to the transient typechecks of Reticulated Python [57, 29]. Reticulated Python inserts transient checks only when a value flows from untyped to typed code, while Grace inserts transient checks only at explict type annotations (but in principle checks every annotation every time).

Our primary motivation for this work is to provide a flexible system to check consistency between an execution of a program and its type annotations. A key part of the design philosophy of Grace is that the language should not force students to annotate programs with types until they are ready, so that teachers can choose whether to introduce types, early, late, or even not at all.

A secondary goal is to have a design that can be implemented with only a small set of changes to facilitate integration in existing systems.

Both of these goals are shared with much of the other work on gradual type systems, but our context leads to some different choices. First, while checking Grace’s type annotations statically may be optional, checking them dynamically should not be: any value that flows into a variable, argument, or result annotated with a type must conform to that type annotation. Second, adding type annotations should not degrade a program’s performance, or rather, programmers should not be encouraged to improve performance by removing type annotations. And third, we allow the programmer to execute a program even when not statically type-correct. Allowing such execution is useful to students, where they can see concrete examples of dynamic type errors. This is possible because Grace’s static type checking is optional, which means that an implementation cannot depend on the correctness or mutual compatibility of a program’s type annotations.

Unfortunately, existing gradual type implementations do not meet these goals, particularly regarding performance; hence the ongoing debate about whether gradual typing is alive, dead, or some state in between[53, 58, 40, 6, 42, 29].

We now illustrate how the gradual type checks work in practice in the context of a simple program to record information about vehicles. Suppose the programmer starts developing this vehicle application by defining an object intended to represent a car (LABEL:lst:car-reg, Line˜1) and writes a method that, given the car object, prints out its registration number (Line˜5).

Next, the programmer adds a check to ensure any object passed to the printRegistration method will respond to the registration request; they define the structural type Vehicle[55] naming just that method (LABEL:ex:vehicle, Line˜1), and annotate the printRegistration method’s argument with that type (LABEL:ex:vehicle, Line˜5). The annotation ensures that a type error will be thrown if an object, passed to the printRegistration method, cannot respond to the registration message. Furthermore, as type errors constituent termination, a crash somewhere in the middle of the implementation of the print method will now be avoided.

In LABEL:ex:complex, the programmer continues development and creates two car objects (Lines˜9 and 17), that conform to an expanded Vehicle type (Line˜1).

Note that each version of the registerTo method declares a different type for its parameter (Lines˜12 and 20). When the programmer executes this program, both personalCar and governmentCar can be assigned to a variable declared as Vehicle because checking that assignment considers only that the vehicle has a registerTo method, but not the required argument type of that method. At Line˜25 the developer attempts to register a government car to a person: only when the method is invoked (Line˜20) will the gradual type test on the argument fail (the object that is passed in is not a Department because it lacks a code method).

One of the goals for our approach to gradual typing was to keep the necessary changes to an existing implementation small, while enabling optimization in highly efficient language runtimes. In an AST interpreter, we can implement this approach by attaching the checks to the relevant AST nodes: the expected types for the argument and return values can be included with the node for requesting a method, and the expected type for a variable can be attached to the nodes for reading from and writing to that variable. In practice, we encapsulate the logic of the check within a new class of AST nodes, specially to support gradual type checking. Moth’s front end was adapted to parse and record type annotations and attach instances of this checking node as children of the existing method, variable read, and variable write nodes.

The check node uses the internal representation of a Grace type (cf. LABEL:ex:type, Line˜13) to test whether an observed object conforms to that type. An object satisfies a type if all members required by the type are provided by that object (Line˜5).

There are two aspects to our implementation that are critical for a minimal-overhead solution:

• •

  specialized executions of the type checking node, along with guards to protect these specialized versions, and

• •

  a matrix to cache sub-typing relationships to eliminate redundant exhaustive subtype tests.

The first performance-critical aspect to our implementation is the optimization of the type checking node. We rely on Truffle and its TruffleDSL[30]. This means we provide a number of special cases, which are selected during execution based on the observed concrete kinds of objects. A sketch of our type checking node using a pseudo-code version of the DSL is given in LABEL:ex:typenode. A simple optimization is for well known types such as numbers (Line˜8) or strings (Line˜12). The methods annotated with @Spec (shorthand for @Specialization) correspond to possible states in a state machine that is generated by the TruffleDSL. Thus, if a check node observes a number or a string, it will check on the first execution only that the expected type, i.e., the one defined by some type annotation, is satisfied by the object by using a static_guard. If this is the case, the DSL will activate this state. For just-in-time compilation, only the activated states and their normal guards are considered. A static_guard is not included in the optimized code. If a check fails, or no specialization matches, the fallback, i.e., check_generic is selected (Line˜24), which may raise a type error.

For generic objects, we rely on the specialization on Line˜20, which checks that the object satisfies the expected type. If that is the case, it reads the shape of the object (cf. Section˜3) at specialization time, and caches it for later comparisons. Thus, during normal execution, we only need to read the shape of the object and then compare it to the cached shape with a simple reference comparison. If the shapes are the same, we can assume the type check passed successfully. Note that shapes are not equivalent to types, however, shapes imply the set of members of an object, and thus, do imply whether an object fulfills one of our structural types.

The other performance-critical aspect to our implementation is the use of a matrix to cache sub-typing relationships. The matrix compares types against types, featuring all known types along the columns and the same types again along the rows. A cell in the table corresponds to a sub-typing relationship: does the type corresponding to the row implement the type corresponding to the column? All cells in the matrix begin as unknown and, as encountered in checks during execution, we populate the table. If a particular relationship has been computed before we can skip the check and instead recall the previously-computed value (Line˜29). Using this table we are able to eliminate the redundancy of evaluating the same type to type relationships across different checks in the program. To reduce redundancy further we also unify types in a similar way to Java’s string interning; during the construction of a type we first check to see if the same set of members is expressed by a previously-created type and, if so, we avoid creating the new instance and provide the existing one instead.

Together the self-specializing type check node and the cache matrix ensure that our implementation eliminates redundancy, and consequently, we are able to minimize the run-time overhead of our system.

To account for the complex warmup behavior of modern systems[4] as well as the non-determinism caused by e.g. garbage collection and cache effects, we run each benchmark for 1000 iterations in the same VM invocation¹¹1 For the Higgs VM, we only use 100 iterations, because of its lower performance. This is sufficient since Higgs’s compilation approach induces less variation and leads to more stable measurements.. Afterwards, we inspected the run-time plots over the iterations and manually determined a cutoff of 350 iterations for warmup, i.e., we discard iterations with signs of compilation. As a result, we use a large number of data points to compute the average, but outliers, caused by e.g. garbage collection, remain visible in the plots. All reported averages use the geometric mean since they aggregate ratios.

All experiments were executed on a machine running Ubuntu Linux 16.04.4, with Kernel 3.13. The machine has two Intel Xeon E5-2620 v3 2.40GHz, with 6 cores each, for a total of 24 hyperthreads. We used ReBench 0.10.1[36], Java 1.8.0_171, Graal 0.33 (a13b888), Node.js 10.4, and Higgs from 9 May 2018 (aa95240). Benchmarks were executed one by one to avoid interference between them. The analysis of the results was done with R 3.4.1, and plots are generated with ggplot 2.2.1 and tikzDevice 0.11. Our experimental setup is available online to enable reproductions.²²2 Removed for double blind review

To establish the performance of Moth, we compare it to Java and JavaScript. For JavaScript we chose two implementations, Node.js with V8 as well as the Higgs VM. The Higgs VM is an interesting point of comparison, because Richards et al. [42] used it in their study. The goal of this comparison is to determine whether our approach could be applicable to industrial strength language implementations without adverse effects on their performance.

We compare across languages based on the Are We Fast Yet benchmarks[38], which are designed to enable a comparison of the effectiveness of compilers across different languages. To this end, they use only a common set of core language elements. While this reduces the performance-relevant differences between languages, the set of core language elements covers only common object-oriented language features with first-class functions. Consequently, these benchmarks are not necessarily a predictor for application performance, but can give a good indication for basic mechanisms such as type checking.

Figure˜1 shows the results. We use Java as baseline since it is the fastest language implementation in this experiment. We see that Node.js (V8) is about 1.8x (min. 0.8x, max. 2.9x) slower than Java. Moth is about 2.3x (min. 0.9x, max. 4.3x) slower than Java. As such, it is on average 32% (min. -18%, max. 2.3x) slower than Node.js. Compared to the Higgs VM, which is on these benchmarks 10.6x (min. 1.5x, max. 169x) slower than Java, Moth reaches the performance of Node.js more closely. With these results, we argue that Moth is a suitable platform to assess the impact of our approach to gradual type checking, because its performance is close enough to state-of-the-art VMs, and run-time overhead is not hidden by slow baseline performance.

The performance overhead of our transient gradual type checking system is assessed based on the Are We Fast Yet benchmarks as well as benchmarks from the gradual-typing literature. The goal was to complement our benchmarks with additional ones that are used for similar experiments and can be ported to Grace. To this end, we surveyed a number of papers[53, 58, 40, 6, 42, 52, 29] and selected benchmarks that have been used by multiple papers. Some of these benchmarks overlapped with the Are We Fast Yet suite, or were available in different versions. While not always behaviorally equivalent, we chose the Are We Fast Yet versions since we already used them to establish the performance baseline. The selected benchmarks as well as the papers in which they were used are shown in Table˜1.

The benchmarks were modified to have complete type information. To ensure correctness and completeness of these experiments, we added an additional check to Moth that reports absent type information to ensure each benchmark is completely typed. To assess the performance overhead of type checking, we compare the execution of Moth with all checks disabled, i.e., the baseline version from Section˜4.2, against an execution that has all checks enabled. We did not measure programs that mix typed and untyped code because with our implementation technique a fully typed program is expected to have the largest overhead.

To characterize the concrete impact of our two optimizations, i.e., the optimized type checking node, which replaces complex type tests with checks for object shapes, and our matrix to cache sub-typing information, we look at the number of type checks performed by the benchmarks, as well as the impact on peak performance.

As discussed in the introduction, in many existing gradually typed systems, one would expect a linear increase of the performance overhead with the increasing number of type annotations.

In this section, we show that this is not necessarily the case on our system. For this purpose, we use two microbenchmarks Check and Nest, which have at their core method calls with 5 parameters. The Check benchmark calls the same method 10 times in a row, i.e., it has 10 call sites. The Nest benchmark has 10 methods with identical signatures, which recurse from the first one to the last one. Thus, there are still 10 method calls, but they are nested in each other. In both benchmarks, each method increments a counter, which is checked at the end of the execution to verify that both do the same number of method activations, and only the shape of the activation stack differs.

Each benchmark exists in 6 variants, going from no type annotations over annotating only the first method parameter to annotating all 5 of the parameters. Furthermore, we present the results for the first iteration as well as the hundredth iteration. The first iteration is executed at least partially in the interpreter, while the hundredth iteration executes compiled.

Figure˜5 shows that such a common scenario of methods being gradually annotated with types does not incur an overhead on peak performance in our system. The plot shows the mean of the run time for each benchmark configuration. Furthermore, it indicates a band with the 95% confidence interval. The green line, Moth (neither), corresponds to our Moth with type checking but without any optimizations. For this case, we see that the performance overhead grows linearly.

For Moth (both) and Moth (untyped) we see for the first iteration that the band of confidence intervals diverges, indicating that the additional type checks have an impact on startup performance. However, for the hundredth iteration, the confidence intervals overlap for the optimized Moth as well as the one that does not perform typechecks. This means that Moth does not suffer from a general linear overhead for adding type checks. Instead, most type checks do not have an impact on peak performance. However, as previously argued for the List benchmark, this is only the case for checks that can be subsumed by shape checks (shape checks are performed whether or not type checks are presents).

Outlined earlier in Section˜3, a secondary goal of our design was to enable the implementation of our approach to be realized with few changes to the underlying interpreter. This helps to ensure that each Grace implementation can provide type checking in a uniform way.

By examining the history of changes maintained by our version control, we estimate that our implementation for Moth required 549 new lines and 59 changes to existing lines. The changes correspond to the implementation of new modules for the type class (179 lines) and the self-specializing type checking node (139 lines), modifications to the front end to extract typing information (115 new lines, 14 lines changes) and finally the new fields and amended constructors for AST nodes (116 new lines, 45 lines changes).

One of the key optimizations for our work and the work of others[42, 6] is the use of object shapes to encode information about types (in our case), or type casts and assumptions (in the case of gradually typed systems).

The general idea is that a VM will already use object shapes for method dispatches, field accesses, and other operations on objects. Thus any further use to also imply type information can often be optimized away when the compiler sees that the same checks are done (and therefore can be combined). This is similar to the elimination of other side-effect-free common subexpressions.

This assumption breaks, however, when checks are introduced that do not correspond to those that exist already. As described in Section˜3, our approach introduces checks for reading and writing to variables. LABEL:ex:pathological-case gives an example of a pathological case. It is a loop traversing a linked list. For this example our approach introduces a check, for the ListElement type, when (1) assigning to and reading from elem and (2) when activating the next method. The checks for reading from elem and activating the method can be combined with the dispatch’s check on object shape. Unfortunately, the compiler cannot remove the check when writing to elem, because it has no information about what value will be returned from next, and so it needs to preserve the check to be able to trigger an error on the assignment. For our List benchmark, this check induces an overhead of 76%.

As a simplification, we currently check variable access on both reads and writes. This approach simplifies the implementation, because we do not need to adapt all built-ins, i.e., all primitive operations provided by the interpreter. One optimization could be to avoid read checks. A type violation can normally only occur when writing to a variable, but not when reading. However, to maintain the semantics, this would require us to adapt many primitives; such as operations that activate blocks to check their arguments, or that write to variables or fields. With our current implementation we get errors as soon as user code accesses fields, which simplifies the implementation.

Another optimization could be to use Truffle’s approach to self-specialization[64] and propagate type information to avoid redundant checks. At the moment, Truffle interpreters typically use self-specialization to specialize the AST to avoid boxing of primitive types. This is done by speculating that some subtree always returns the expected type. If this is not the case, the return value of the subtree is going to be propagated via an exception, which is caught and triggers respecialization. This idea could possibly be used to encode higher-level type information for return values, too. This could be used to remove redundant checks in the interpreter by simply discovering at run time that whole subexpressions conform to the type annotations.

This work is subject to many of the threats to validity common to evaluations of experimental language implementations. Our underlying implementation may contain undetected bugs that affect the semantics or performance of the gradual typing checks, affecting construct validity — we may not have implemented what we think we have. Given that, our benchmarking harness run on the same implementation is subject to the same risks, thus also affecting internal validity — we may not be measuring that implementation correctly. Moth is built on the Truffle and Graal toolchain, so we expect external validity there at least — we expect the results would transfer to other Graal VMs doing similar AST-based optimizations. We have less external validity regarding other kinds of VMs (such as VMs specialized to particular languages, or VMs built via meta-tracing rather than partial evaluation). Nevertheless, we expect our results should be transferable as we rely on common techniques.

Finally, because we are working in Grace, it is less obvious that our results generalize to other gradually typed-languages. We have worked to ensure e.g. our benchmarks do not depend on any features of Grace that are not common in other gradually-typed object-oriented languages, but as Grace lacks a large corpus of programs the benchmarks are necessarily artificial, and it is not clear how the results would transfer to the kinds of programs actually written in practice. The advantage of Grace (and Moth) for this research is that their relative simplicity means we have been able to build an implementation that features competitive performance with significantly less effort than would be required for larger and more complex languages. On the other hand, more effort on optimisations could well lead to even better performance.