Inductive Program Synthesis over Noisy Data

We work with noise models that assume a (hidden) clean data set combined with a noise source that delivers the noisy data set presented to the program synthesis system. Like many inductive program synthesis systems (Gulwani, 2011; Singh and Gulwani, 2016), one target is discrete problems that involve discrete data such as strings, data structures, or tablular data. In contrast to traditional machine learning problems, which often involve continuous noise sources (Bishop, 2006), the noise sources for discrete problems often involve discrete noise — noise that involves a discrete change that affects only part of each output, leaving the remaining parts intact and uncorrupted.

Different loss functions can be appropriate for different noise sources and use cases. The $0/1$ loss function, which counts the number of input-output examples where the data set ${\mathcal{}D}$ and synthesized program $p$ do not agree, is a general loss function that can be appropriate when the focus is to maximize the number of inputs for which the synthesized program $p$ produces the correct output. The Damerau-Levenshtein (DL) loss function (Damerau, 1964), which measures the edit difference under character insertions, deletions, substitutions, and/or transpositions, extracts information present in partially corrupted outputs and can be appropriate for measuring discrete noise in input-output examples involving text strings. The $0/\infty$ loss function, which is $\infty$ unless $p$ agrees with all of the input-output examples in the data set ${\mathcal{}D}$, specializes our technique to the standard program synthesis scenario that requires the synthesized program to agree with all input-output examples.

Because discrete noise sources often leave parts of corrupted outputs intact, exact program synthesis (i.e., synthesizing a program that agrees with all outputs in the hidden clean data set) is often possible even when all outputs in the data set are corrupted. Our experimental results (Section 9) indicate that matching the loss function to the characteristics of the discrete noise source can enable very accurate program synthesis even when 1) there are only a handful of input-output examples in the data and 2) all of the outputs in the data set are corrupted. We attribute this success to the ability of our synthesis technique, working in conjunction with an appropriately designed loss function, to effectively extract information from outputs corrupted by discrete noise sources.

Our technique augments finite tree automata (FTA) to associate accepting states with weights that capture the loss for the output associated with each accepting state. Given a data set ${\mathcal{}D}$, the resulting state-weighted finite tree automata (SFTA) partition the programs $p$ defined by the grammar $G$ into equivalence classes. Each equivalence class consists of all programs with identical input-output behavior on the inputs in ${\mathcal{}D}$. All programs in a given equivalence class therefore have the same loss over ${\mathcal{}D}$. The technique then uses dynamic programming to find the minimum complexity program $p$ in each equivalence class (Gallo et al., 1993). From this set of minimum complexity programs, the technique then finds the program $p$ that minimizes the objective function $U(p,{\mathcal{}D})$.

We have implemented our technique and applied it to various programs in the SyGuS 2018 benchmark set (SyG, 2018). The results indicate that the technique is effective at solving program synthesis problems over strings with modestly sized solutions even in the presence of substantial noise. For discrete noise sources and a loss function that is a good match for the noise source, the technique is typically able to extract enough information left intact in corrupted outputs to synthesize a correct program even when all outputs are corrupted (in this paper we consider a synthesized program to be correct if it agrees with all input-output examples in the original hidden clean data set). Even with the 0/1 loss function, which does not aspire to extract any information from corrupted outputs, the technique is typically able to synthesize a correct program even with only a few correct (uncorrupted) input-output examples in the data set. Overall the results highlight the potential for effective program synthesis even in the presence of substantial noise.

This paper makes the following contributions:

• •

  Technique: It presents an implemented technique for inductive program synthesis over noisy data. The technique uses an extension of finite tree automata, state-weighted finite tree automata, to synthesize programs that minimize an objective function involving the loss over the input data set and the complexity of the synthesized program.

• •

  Use Cases: It presents multiple uses cases including best fit program synthesis for noisy data sets, navigating accuracy vs. complexity tradeoffs, Bayesian program synthesis, identifying and correcting corrupted data, and approximate program synthesis.

• •

  Experimental Results: It presents experimental results from our implemented system on the SyGuS 2018 benchmark set. These results characterize the scalability of the technique and highlight interactions between the DSL, the noise source, the loss function, and the overall effectiveness of the synthesis technique. In particular, they highlight the ability of the technique to, given a close match between the noise source and the loss function, synthesize a correct program $p$ even when 1) there are only a handful of input-output examples in the data set ${\mathcal{}D}$ and 2) all outputs are corrupted.

Finite tree automata are a type of state machine which accept trees rather than strings. They generalize standard finite automata to describe a regular language over trees.

Every symbol $f$ in alphabet $F$ has an associated arity. The set $F_{k}\subseteq F$ is the set of all $k$-arity symbols in $F$. $0$-arity terms $t$ in $F$ are viewed as single node trees (leaves of trees). $t$ is accepted by an FTA if we can rewrite $t$ to some state $q\in Q_{f}$ using rules in $\Delta$. The language of an $FTA$ ${\mathcal{}A}$, denoted by ${\mathcal{}L}({\mathcal{}A})$, corresponds to the set of all ground terms accepted by ${\mathcal{}A}$.

The above tree automaton accepts all propositional logic formulas over $\mathsf{True}$ and $\mathsf{False}$ which evaluate to $\mathsf{True}$. Figure 1 presents the tree for the formula $\mathsf{and}(\mathsf{True},\mathsf{not}(\mathsf{False}))$.

We next define the programs we consider, how inputs to the program are specified, and the program semantics. Without loss of generality, we assume programs $p$ are specified as parse trees in a domain-specific language (DSL) grammar $G$. Internal nodes represent function invocations; leaves are constants/0-arity symbols in the DSL. A program $p$ executes in an input $\sigma$. $\llbracket p\rrbracket\sigma$ denotes the output of $p$ on input $\sigma$ ($\llbracket.\rrbracket$ is defined in Figure 2).

All valid programs (which can be executed) are defined by a DSL grammar $G=(T,N,P,s_{0})$ where:

• •

  $T$ is a set of terminal symbols. These may include constants and symbols which may change value depending on the input $\sigma$.

• •

  $N$ is the set of nonterminals that represent subexpressions in our DSL.

• •

  $P$ is the set of‘ production rules of the form
  $s\rightarrow f(s_{1},\ldots,s_{n})$, where $f$ is a built-in function in the DSL and $s,s_{1},\ldots,s_{n}$ are non-terminals in the grammar.

• •

  $s_{0}\in N$ is the start non-terminal in the grammar.

We assume that we are given a black box implementation of each built-in function $f$ in the DSL. In general, all techniques explored within this paper can be generalized to any DSL which can be specified within the above framework.

We review the approach introduced by (Wang et al., 2017a, b) to use finite tree automata to solve synthesis tasks over a broad class of DSLs. Given a DSL and a set of input-output examples, a Concrete Finite Tree Automaton (CFTA) is a tree automaton which accepts all trees representing DSL programs consistent with the input-output examples and nothing else. The states of the FTA correspond to concrete values and the transitions are obtained using the semantics of the DSL constructs.

Given an input-output example $(\sigma,o)$ and DSL $(G,\llbracket.\rrbracket)$, construct a CFTA using the rules in Figure 3. The alphabet of the CFTA contains built-in functions within the DSL. The states in the CFTA are of the form $q_{s}^{c}$, where $s$ is a symbol (terminal or non-terminal) in $G$ and $c$ is a concrete value. The existence of state $q^{c}_{s}$ implies that there exists a partial program which can map $\sigma$ to concrete value $c$. Similarly, the existence of transition $f(q_{s_{1}}^{c_{1}},q_{s_{2}}^{c_{2}}\ldots q_{s_{k}}^{c_{k}})\rightarrow q_{s}^{c}$ implies $f(c_{1},c_{2}\ldots c_{k})=c$.

The $\mathsf{Term}$ rule states that if we have a terminal $t$ (either a constant in our language or input symbol $x$), execute it with the input $\sigma$ and construct a state $q_{t}^{c}$ (where $c=\llbracket t\rrbracket\sigma$). The $\mathsf{Final}$ rule states that, given start symbol $s_{0}$ and we expect $o$ as the output, if $q^{o}_{s_{0}}$ exists, then we have an accepting state. The $\mathsf{Prod}$ rule states that, if we have a production rule $f(s_{1},s_{2},\ldots s_{k})\rightarrow s\in\Delta$, and there exists states $q_{s_{1}}^{c_{1}},q_{s_{2}}^{c_{2}}\ldots q_{s_{k}}^{c_{k}}\in Q$, then we also have state $q_{s}^{c}$ in the CFTA and a transition $f(q_{s_{1}}^{c_{1}},q_{s_{2}}^{c_{2}},\ldots q_{s_{k}}^{c_{k}})\rightarrow q_{s}^{c}$.

The language of the CFTA constructed from Figure 3 is exactly the set of parse trees of DSL programs that are consistent with our input-output example (i.e., maps input $\sigma$ to output $o$).

In general, the rules in Figure 3 may result in a CFTA which has infinitely many states. To control the size of the resulting CFTA, we do not add a new state within the constructed CFTA if the smallest tree it will accept is larger than a given threshold $d$. This results in a CFTA which accepts all programs which are consistent with the input-output example but are smaller than the given threshold (it may accept some programs which are larger than the given threshold but it will never accept a program which is inconsistent with the input-output example). This is standard practice in the synthesis literature (Wang et al., 2017a; Polozov and Gulwani, 2015).

Given a single input-output example, a CFTA built on that example only accepts programs which are consistent with that example. $/$ intersection is a simple method to prune a SFTA to only contain programs which are consistent with an input-output example.

Given a DSL $G$, a data set ${\mathcal{}D}$, loss function ${\mathcal{}L}$, complexity measure $c$, and positive weight $\lambda$, we wish to find a program $p^{*}$ which minimizes the weighed sum of the loss function and the complexity measure. Formally:

where $G_{d}$ is the set of programs in DSL $G$ with size less than the threshold $d$. By using the objective function $U(l,c)=l+\lambda c$, we can use Algorithm 1 to synthesize program $p^{*}$ which minimizes the objective function given above.

Given a DSL $G$, a data set ${\mathcal{}D}$, loss function ${\mathcal{}L}$, complexity measure $C$ and bound $b$, we wish to find a program $p^{*}$ that minimizes the complexity measure $C$ but has loss less than $b$. Formally: $p^{*}=\mathsf{argmin}_{p\in G_{d}}C(p)~{}\text{ s.t. }{\mathcal{}L}(p,{\mathcal{}D})<b$. Note that this condition can be rewritten as

where ${\mathcal{}A}^{\prime}={\mathcal{}A}({\mathcal{}D},L)\downarrow_{b}$.

By the definition of $\downarrow_{b}$, all accepting states of ${\mathcal{}A}^{\prime}$ have weight less than $b$. Therefore all programs accepted by ${\mathcal{}A}^{\prime}$ have loss less than $b$ (i.e. ${\mathcal{}L}(p,{\mathcal{}D})<b$). Also note that if a program $p$ is not in ${\mathcal{}A}^{\prime}$ then either it has loss greater than $b$ or it is not within the threshold $d$.

Given DSL $G$, a data set ${\mathcal{}D}$, a subset ${\mathcal{}D}^{\prime}\subseteq{\mathcal{}D}$, loss function ${\mathcal{}L}$, complexity measure $C$, and objective function $U$, we wish to find a program $p^{*}$ which minimizes the objective function with an added constraint of bounded loss over data set ${\mathcal{}D}^{\prime}$. Formally:

We first construct a SFTA ${\mathcal{}A}({\mathcal{}D}^{\prime},L)\downarrow_{b}$ which contains all programs consistent with loss less than or equal to $b$ over data set ${\mathcal{}D}^{\prime}$. After constructing ${\mathcal{}A}({\mathcal{}D},L)$ as in Algorithm 1, we modify ${\mathcal{}A}({\mathcal{}D},L)$ by $/$ intersection ${\mathcal{}A}({\mathcal{}D}^{\prime},L)$ (after dropping the weights of the accepting states) with ${\mathcal{}A}({\mathcal{}D},L)$ (i.e. ${\mathcal{}A}({\mathcal{}D},L)\leftarrow{\mathcal{}A}({\mathcal{}D},L)/{\mathcal{}A}({\mathcal{}D}^{\prime},L)$ as in Algorithm 1). By definition of $/$ intersection and ${\mathcal{}A}$, loss of all programs returned by the modified algorithm on ${\mathcal{}D}^{\prime}$ will be less than equal to $b$.