Automata Learning from Preference and Equivalence Queries

Active automata learning has applications from software engineering [37, 1] and verification [29] to interpretable machine learning [44] and learning reward machines [39, 20, 46, 16]. The classical problem formulation involves a teacher with access to a regular language and a learner which asks membership and equivalence queries to infer a finite automaton describing the regular language [5, 22].

Consider an alternative formulation: learning a finite automaton from preference and equivalence queries.¹¹1Shah et al. [38] investigates choosing between membership and preference queries. A preference query resolves the relative position of two sequences in a total preorder available to the teacher. The motivation for learning from preferences stems from leveraging human preferences as a rich source of information. In fact, comparative feedback, such as preferences, is a modality which humans are apt at providing in comparison to giving specific numerical values, as shown by MacGlashan et al. [30] and Christiano et al. [12], likely due to choice overload [26]. But preferences need not be directly obtained from individual humans: preferences can also be obtained from automated systems in a frequency-based sense. For example, consider an obstacle avoidance scenario of a vehicle avoiding large debris in the middle of a roadway. Consider a dataset of a population of driver responses (vehicle trajectories) to such a scenario, procured from dashcam or traffic camera footage. The population preference of a given driving response can be automatically determined by ranking each response by the frequency of occurrence in the dataset. In fact, learning from human preference data has applications in fine-tuning language models [34], learning conditional preference networks [28, 23], inferring reinforcement learning policies [13], and learning Markovian reward functions [11, 36, 10, 27]. Possible applications for learning finite automata from preferences over sequences include inferring sequence classifications (e.g. program executions [21], vehicle maneuvers, human-robot interactions) using ordered classes (e.g. safe, risky, dangerous, fatal) with each automaton state labeled by a class, distilling interpretable preference models [44], and inferring reward machines. However, no method currently exists for learning finite automata from preferences with the termination and correctness guarantees enjoyed by classical automata learning algorithms such as $\textrm{L}^{*}$ [5].

Unfortunately, adapting $\textrm{L}^{*}$ to the preference-based setting is challenging. Preference queries do not directly provide the concrete observations available from membership queries, as required by $\textrm{L}^{*}$, so our solution Remap addresses this challenge through a symbolic approach to $\textrm{L}^{*}$.

Remap²²2Code available at https://eric-hsiung.github.io/remap/, as well as supplementary material. (Figure 1) features termination and correctness guarantees for exact and probably approximately correct [43] (PAC) identification [6] of the desired automaton, with a strong learner capable of symbolic reasoning and constraint-solving to offset the weaker preference-based signals from the teacher. While approximate learners have been proposed to learn from a combination of membership and preference queries [38], Remap is the first exact learner with formal guarantees of minimalism, correctness, and query complexity. By using unification³³3Symbolically obtaining sets of equivalent variables from sets of equations, and substituting variables in the equations with their representatives. See Section 4.2. to navigate the symbolic space of hypotheses, and constraint-solving to construct a concrete automaton from a symbolic hypothesis, Remap identifies, in a polynomial number of queries, the minimal Moore machine isomorphic to one describing the teacher’s total preorder over input sequences when using exact equivalence queries. However, exact equivalence queries may be infeasible if the learner and teacher lack a common representation.⁴⁴4Either a pair of identical representations, or between which a translation exists. This motivates sampling-based equivalence queries under Remap, which achieves PAC–identification.⁵⁵5See Definition 10 for PAC–identification and related Theorems 5.4 and 5.5. Our empirical evaluations apply Remap to learn reward machines for sequential-decision making domains in the reward machine literature. We measure query complexity for the exact and PAC–identification settings, and measure empirical correctness for the PAC–identification setting.

We contribute (a) Remap, a novel $\textrm{L}^{*}$ style algorithm for learning Moore machines using preference and equivalence queries; under exact and PAC-identifica-tion settings we provide (b) theoretical analysis of query complexity, correctness, and minimalism, and (c) supporting empirical results, demonstrating the efficacy of the algorithm’s ability to learn reward machines from preference and equivalence queries.

Prior to introducing Remap, we provide preliminaries on orders and finite automata, followed by a discussion of Angluin’s $\textrm{L}^{*}$ algorithm in order to highlight how Remap differs from $\textrm{L}^{*}$.

Finite Automata  Automata describe sets of sequences. An alphabet $\Sigma$ is a set whose elements can be used to construct sequences; $\Sigma^{*}$ represents the set of sequences of any length created from elements of $\Sigma$. A sequence $s\in\Sigma^{*}$ has integer length $|s|\geq 0$; if $|s|=0$, then $s$ is the empty sequence $\varepsilon$. An element $\sigma\in\Sigma$ has length 1; if $s,t\in\Sigma^{*}$, then $s\cdot t$ represents $s$ concatenated with $t$, with length $|s|+|t|$. Different types of finite automata have different semantics. Deterministic automata feature a deterministic transition function $\delta$ defined over a set of states $Q$ and an input alphabet $\Sigma^{I}$. An output alphabet $\Sigma^{O}$ may be present to label states or transitions using a labeling function $L$. The tuple $\langle Q,q_{0},\Sigma^{I},\Sigma^{O},\delta,L\rangle$ is a Moore machine [33], where $q_{0}\in Q$ is the initial state, $\delta:Q\times\Sigma^{I}\rightarrow Q$ describes transitions, and $L:Q\rightarrow\Sigma^{O}$ associates outputs with states. Extended, $\delta:Q\times(\Sigma^{I})^{*}\rightarrow Q$, where $\delta(q,\varepsilon)=q$ and $\delta(q,\sigma\cdot s)=\delta(\delta(q,\sigma),s)$. In Mealy machines [32], outputs are associated with transitions instead, so $L:Q\times\Sigma^{I}\rightarrow\Sigma^{O}$. Mealy and Moore machines are equivalent [19] and can be converted between one another. Reward machines are an application of Mealy machines in reinforcement learning and are used to express a class of non-Markovian reward functions.

Active Automaton Learning  Consider the problem of actively learning a Moore machine $\langle Q,q_{0},\Sigma^{I},\Sigma^{O},\delta,L\rangle$ to exactly model a function $f:(\Sigma^{I})^{*}\rightarrow\Sigma^{O}$, where $\Sigma^{I}$ and $\Sigma^{O}$ are input and output alphabets of finite size known to both teacher and learner. We desire a learner which learns a model $\hat{f}$ of $f$ exactly; that is for all $s\in(\Sigma^{I})^{*}$, we require $\hat{f}(s)=f(s)$, where $\hat{f}(s)=L(\delta(q_{0},s))$, with the assistance of a teacher $\mathcal{T}$ that can answer questions about $f$.

In Angluin’s seminal active learning algorithm, $\textrm{L}^{*}$ [5], the learner learns $\hat{f}$ as a binary classifier, where $|\Sigma^{O}|=2$, to determine sequence membership of a regular language by querying $\mathcal{T}$ with: (i) membership queries, where the learner asks $\mathcal{T}$ for the value of $f(s)$ for a particular sequence $s$, and (ii) equivalence queries, where the learner asks $\mathcal{T}$ to evaluate whether for all $s\in(\Sigma^{I})^{*}$, $\hat{f}(s)=f(s)$. For the latter query, $\mathcal{T}$ returns True if the statement holds; otherwise a counterexample $c$ for which $\hat{f}(c)\neq f(c)$ is returned. An observation table $\langle S,E,T\rangle$ records the concrete observations acquired by the learner’s queries (see Figure 2). Here, $S$ is a set of prefixes, $E$ is a set of suffixes, and $T$ is the empirical observation function that maps sequences to output values—$T:(S\cup(S\cdot\Sigma^{I}))\cdot E\rightarrow\Sigma^{O}$. The observation table is a two-dimensional array; $s\in(S\cup(S\cdot\Sigma^{I}))$ indexes rows, and $e\in E$ indexes columns, with entries given by $T(s\cdot e)$. Any proposed hypothesis $\hat{f}$ must be consistent with $T$. The algorithm operates by construction: a deterministic transition function must be found exhibiting consistency (deterministic transitions) and operates in a closed manner over the set of states. If the consistency or closure requirements are violated, then membership queries are executed to expand the observation table. Once a suitable transition function is found, a hypothesis $\hat{f}$ can be made and checked via the equivalence query. The algorithm terminates if $\hat{f}(s)=f(s)$ for all $s\in(\Sigma^{I})^{*}$; otherwise $\textrm{L}^{*}$ continues on by adding counterexample $c$ and all its prefixes to the table, and finds another transition function satisfying the consistency and closure requirements.

Consequently, we consider how $\textrm{L}^{*}$-style learning can be used to learn Moore machines from preference queries over sequences, as a foray into understanding how finite automaton structure can be learned from comparison information.

We consider how a Moore machine $\langle Q,q_{0},\Sigma^{I},\Sigma^{O},\delta,L\rangle$ can be actively learned from preference queries over $(\Sigma^{I})^{*}$. We focus on the case of a finite sized $\Sigma^{I}$ and $\Sigma^{O}$ known to both teacher and learner. With a preference query, the learner asks the teacher $\mathcal{T}$ which of two sequences $s_{1}$ and $s_{2}$ is preferred, or if both are equally preferable. This requires a preference model, which we assume represents a total preordering over $(\Sigma^{I})^{*}$; we also assume $\Sigma^{O}$ is totally ordered. Thus, we consider a preference model for $\mathcal{T}$ where $f:(\Sigma^{I})^{*}\rightarrow\Sigma^{O}$ is consistent with both orderings, i.e., $\mathcal{T}$ prefers $s_{1}$ over $s_{2}$ if $f(s_{1})>f(s_{2})$, or otherwise has equal preference if $f(s_{1})=f(s_{2})$.

Several options for evaluating hypothesis equivalence (is $\hat{f}\equiv f$?) can be defined for this problem formulation. We first review the definition of exact equivalence, followed by an alternative notion of equivalence which respects ordering:

Since a hypothesis $\hat{f}$ satisfying exact equivalence is also always order-respect-ing, we focus on exact equivalence queries where $\mathcal{T}$ returns feedback along with counterexample $c$.

The strength of feedback $\phi$ directly impacts how many hypotheses per counterexample can be eliminated by the learner: returning $\phi(c):=\hat{f}(c)\neq f(c)$, which means the value $f(c)$ is not $\hat{f}(c)$, is weak feedback compared to returning $\phi(c):=\hat{f}(c)=f(c)$, which means the value of $\hat{f}(c)$ should be $f(c)$, or returning $\phi(c):=\hat{f}(c)\in X$ (where $X\subset\Sigma^{O}$, which means the value of $\hat{f}(c)$ should be in a subset of $\Sigma^{O}$). Although Remap outputs a concrete Moore machine, Remap navigates symbolic Moore machine⁶⁶6i.e., a Moore machine with symbolic values as outputs. space using solely preference information, while concrete information assists in selecting the concrete hypothesis. Thus, our theoretical analysis considers equivalence queries that provide counterexamples with strong feedback: $\phi(c):=\hat{f}(c)=f(c)$.

Remap is a $\textrm{L}^{*}$-based algorithm employing preference and equivalence queries to gather constraints. By first leveraging unification to navigate the symbolic hypothesis space of Moore machines, solving the constraints yields a concrete Moore machine. In particular, Remap (Algorithm 1) learns a Moore machine $\langle\hat{Q},\hat{q}_{0},\Sigma^{I},\Sigma^{O},\hat{\delta},\hat{L}\rangle$, representing a multiclass classifier $\hat{f}(s)=\hat{L}(\hat{\delta}(\hat{q}_{0},s))$.

Central to Remap are four core components: (1) a new construct called a symbolic observation table (Section 4.1), shown in Figure 3a, 3b, 3d, and 3e; (2) as well as an associated algorithm for unification [35] (Section 4.2) inspired by Martelli and Montanari [31] to contain the fresh variable explosion; both enable the learner to (3) generate symbolic hypotheses (Section 4.3) purely from observed symbolic constraints, along with (4) a constraint solver for obtaining a concrete hypothesis (Section 4.4). We first discuss the four components and how they fit together into Remap (Section 4.5). Afterwards, we illustrate the correctness and termination guarantees.

We now cover the algorithmic guarantees of Remap when $\mathcal{T}$ uses exact equivalence queries, and show how sampling-based equivalence queries achieves PAC–identification. We first detail how Remap guarantees termination and yields a correct, minimal Moore machine that classifies sequence equivalently to $f$. If Remap terminates, then the final hypothesis must be correct, since termination occurs only if no counterexamples exist for the final hypothesis. Therefore, if the hypothesized Moore machine classifies all sequences correctly according to the teacher, it must be correct. Thus, proving termination implies correctness. See Appendix for sketches and proofs. Here, we assume the teacher provides feedback $\hat{f}(c)=f(c)$ with counterexample $c$.

Theorem 1 establishes that the output of Remap will be the smallest Moore machine consistent with all the constraints in $\mathcal{C}$. This is necessary to prove termination.

Lemma 1, Theorem 2, and Corollary 1 imply Remap makes progress towards termination with every hypothesis, and termination occurs when specific conditions are satisfied; therefore its output must be correct. Theorem 3 indicates that REMAP learns the correct minimal automaton isomorphic to the target automaton in polynomial time. Next, we show how Remap achieves PAC–identification when sampling-based equivalence queries are used.

Theorem 4 and Theorem 5 imply Remap achieves PAC–identification for a choice of $\epsilon$ and $d$ as long as the teacher samples sufficient sequences per equivalence query. In particular, Theorem 5 indicates the total quantity of sequences sampled by the teacher to achieve PAC-identification depends on both $n$ the number of states, and $|\Sigma^{O}|$ the number of output classes. This contrasts with the result for PAC-identification of DFAs (Theorem 7 [5]), which depends only on $n$. Finally, since Remap outputs a Moore machine, one can leverage Moore and Mealy machine equivalence in order to convert the final hypothesis into a reward machine, as defined and covered in the next section.

We consider applying Remap to learn reward machines from preferences. Reward machines are Mealy machines with propositional and reward semantics. Equivalence between Mealy and Moore machines allows the output of Remap to be converted to a reward machine. We first review reinforcement learning and reward machine semantics.

Markov Decision Processes  Decision making problems are often modeled by a Markov Decision Process (MDP), which is a tuple $\langle\mathcal{S},\mathcal{A},P,R,\gamma\rangle$ where $\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions, $P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ represents the transition probability from state $s$ to $s^{\prime}$ via action $a$. The reward function $R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ provides the associated scalar reward, and $0\leq\gamma\leq 1$ is a discount factor. In MDPs, the Markovian assumption is that transitions and rewards depend only upon the current state-action pair and the next state. However, not all tasks are expressible using Markovian reward [2].

Non-Markovian Reward  Non-Markovian Reward Decision Processes are identical to MDPs, except that $R:(\mathcal{S}\times\mathcal{A})^{*}\rightarrow\mathbb{R}$ is non-Markovian a reward function that depends on state-action history. This allows reward machines to model a class of non-Markovian reward functions.

Reward Machines  A reward machine (RM) is a Mealy machine where $\Sigma^{I}=2^{\mathcal{P}}$, $\Sigma^{O}$ is a set of reward emitting objects, and $\mathcal{P}$ is a set of propositions describing states and actions. A labeling function $\mathbb{L}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow 2^{\mathcal{P}}$ with $\mathbb{L}(s_{k-1},a_{k},s_{k})=l_{k}$ labels a state-action sequence $s_{0}a_{1}s_{1}a_{2}s_{2}\dots a_{n}s_{n}$ with label sequence $l_{1}l_{2}\dots l_{n}$. Thus, reward machines operate over label sequences.

A single disjunctive normal formula (DNF) labeled transition can summarize multiple transitions with identical $\Sigma^{O}$ labels (connecting a pair of states), since the elements of $2^{\mathcal{P}}$ are sets of propositions. Reward machines map label sequences to reward outputs and can be represented as $f:(2^{\mathcal{P}})^{*}\rightarrow\Sigma^{O}$, so Remap learns a reward machine by converting the output Moore machine to a reward machine.

Sequential Tasks in OfficeWorld and CraftWorld  Icarte et al. [25] and Andreas et al. [4] introduced the OfficeWorld and CraftWorld gridworld domains, respectively, and feature sequential tasks encoded as reward machines. OfficeWorld features 4 sequential tasks across several rooms with various objects available for an agent to interact with. Example tasks include (1) picking up coffee and mail and delivering them to a certain room, or (2) continuously patrolling between a set of rooms. CraftWorld is a 2D version of MineCraft, where the 10 sequential tasks involve the agent collecting materials and constructing tools or objects in a certain order while avoiding hazards.

We evaluate the exact and PAC–identification (PAC-ID) versions of Remap and consider: first, how often is PAC-ID Remap correct? (Exact Remap is guaranteed to be correct). To answer this, we run experiments by applying PAC-ID Remap to learn reward machines (RMs), by converting the Moore machine into a RM. We measure empirical correctness with empirical probability of isomorphism and average regret. Second, how do exact and PAC-ID Remap scale? We measure preference query complexity as a function of the number of unique sequences queried, and present an example phase diagram of algorithm execution. We use Z3 [15] for the constraint solver.

Setup  We investigate these questions on 14 sequential tasks in the OfficeWorld and CraftWorld domains. The Appendix 0.A.1 contains domain specific details. We implement exact equivalence queries (Definition 5) using a variant of the Hopcroft-Karp algorithm [3, 24]. We implement i.i.d. sequence sampling in sampling-based equivalence queries with the following process per sample: sample a length $L$ from a geometric distribution; then, construct an $L$-length sequence by drawing $L$ elements i.i.d. from a uniform distribution over $\Sigma^{I}$. In both the exact and sampling-based equivalence queries, strong feedback is used.

Active approaches for learning automata are variations or improvements of Angluin’s seminal $\textrm{L}^{*}$ algorithm [5], featuring membership and equivalence queries. We consider an alternative formulation: actively learning automata from preference and equivalence queries featuring feedback. We first discuss adaptations of $\textrm{L}^{*}$ for learning variants of finite automata, including reward machine variants.

Learning Finite Automata  Angluin [5] introduced $\textrm{L}^{*}$ to learn deterministic finite automata (DFAs). Remap has similar theoretical guarantees as $\textrm{L}^{*}$, but utilizes a symbolic observation table, rather than an evidence-based one. Other algorithms adopt the evidence-based table of $\textrm{L}^{*}$ to learn: symbolic automata [17, 7], where Boolean predicates summarize state transitions; weighted automata [9, 8] which feature valuation semantics for sequences on non-deterministic automata; probabilistic DFAs [44], a weighted automata that models distributions of sequences. None of these approaches uses preference queries.

However, Shah et al. [38] considers active, cost-based selection between membership and preference queries to learn DFAs, relying on a satisfiability encoding of the problem. They assume a fixed hypothesis space and have probabilistic guarantees for termination and correctness. Remap, through unification, navigates a sequence of hypothesis spaces, each guaranteed to contain a concrete hypothesis satisfying current constraints, and has theoretical guarantees of correctness, minimalism, and termination under exact and PAC–identification settings.

Furthermore, learning finite automata from preference information relates to the novel problem of learning reward machines [25] from preferences. Learning Markovian reward functions from preferences has be studied extensively using neural [11, 36] and interpretable decision tree [10, 27] representations, but approaches for learning reward machines primarily adapt evidence-based finite automata learning approaches.

Reward Machine Variants  Several reward machine (RM) variants have been proposed. Classical RMs [25, 42] have deterministic transitions and rewards; probabilistic RMs [16] model probabilistic transitions and deterministic rewards; and stochastic RMs [14] pair deterministic transitions with stochastic rewards. Symbolic RMs [47] are deterministic like classical RMs, but feature symbolic reward values in place of concrete values. Zhou and Li [47] apply Bayesian inverse reinforcement learning (BIRL) to infer optimal reward values and actualize symbolic RMs into classical RMs, and require a symbolic RM sketch. Remap requires no sketch, since it navigates over a hypothesis space of symbolic RMs and outputs a concrete classical RM upon termination.

Learning Reward Machines  Many RM learning algorithms assume access to explicit reward samples via environment interaction. Given a maximum RM size, Toro Icarte et al. [42] apply discrete optimization to arrive at a perfect classical RM. Xu et al. [45] learn a minimal classical RM by combining regular positive negative inference [18] with Q-learning for RMs [25], and apply constraint solving to ensure each hypothesis RM is consistent with observed reward samples. Corazza et al. [14] extended the method to learn stochastic RMs. Topper et al. [40] extends BIRL to learn classical RMs using simulated annealing, but needs the number of states to be supplied, and requires empirical tuning of hyperparameters. $\textrm{L}^{*}$ based approaches have also been used to learn classical [39, 20, 46] and probabilistic [16] RMs, relying on concrete observation tables. Gaon and Brafman [20] and Xu et al. [46] use a binary observation table, while Tappler et al. [39] and Dohmen et al. [16] record empirical reward distribution table entries.

In contrast, Remap uses a symbolic observation table, and uses preferences information in place of explicit reward values. Remap navigates symbolic hypothesis space, with constraint solving enabling a concrete classical RM.

We introduce the problem of learning Moore machines from preferences and propose Remap, an $\textrm{L}^{*}$ based algorithm, wherein a strong learner with access to a constraint solver is paired with a weak teacher capable of answering preference queries and providing counterexample feedback in the form of a constraint. Unification applied to a symbolic observation table permits symbolic hypothesis space navigation; the constraint solver enables concrete hypotheses. Remap has theoretical guarantees for correctness, termination, and minimalism under both exact and PAC–identification settings, and it has been empirically verified under both settings when applied to learning reward machines. Future work will expound on more realistic preference models, variable strength feedback, and inconsistency.