Lale: Consistent Automated Machine Learning

An individual operator is a data science algorithm (aka. a primitive or a model), which may be a transformer or an estimator such as classifier or a regressor. Individual operators are modular building blocks from libraries such as scikit-learn. Figure 1 Lines 2–6 contain several examples, e.g., import LinearSVC as SVM. Mathematically, Lale views an individual operator as a function of the form

A pipeline is a directed acyclic graph (DAG) of operators and a pipeline is itself also an operator. Since a pipeline contains operators and is an operator, it is highly composable. Furthermore, viewing both individual operators and pipelines as special cases of operators makes the concepts more consistent. An example is make_pipeline(make_union(PCA, RFC), SVM), which is equivalent to ((PCA & RFC) >> ConcatFeatures >> SVM). Here, & is the and combinator and >> is the pipe combinator. Combinators make edges more explicit and code more concise. An expression x & y composes x and y without introducing additional edges. An expression x >> y introduces edges from all sinks of subgraph x to all sources of y. Mathematically, Lale views a pipeline as a function of the form

An operator choice is an exclusive disjunction of operators and is itself also an operator. Operator choice specifies algorithm selection, and by being an operator, addresses problem $P_{1}$ from Section 2. An example is (SVM | RFC), where | is the or combinator. Mathematically, Lale views an operator choice as a function of the form

The combined schema of an operator specifies the valid values along with search guidance for its latent arguments. It addresses problem $P_{4}$ from Section 2, supporting automated search with a pruned search space and early error checking all from the same single source of truth. Consider the pipeline PCA >> (J48 | LR), where PCA and LR are the principal component analysis and logistic regression from scikit-learn and J48 is a decision tree with pruning from Weka (Hall et al., 2009). These operators have many hyperparameters and constraints and Lale handles all of them. For didactic purposes, this section discusses only a representative subset. Figure 7 shows the JSON Schema (Pezoa et al., 2016) specification of that subset. The open-source Lale library includes JSON schemas for many operators, some hand-written and others auto-generated (Baudart et al., 2020). The number of components for PCA is given by $N$, which can be a continuous value in $(0..1)$ or the categorical value mle. The prior distribution helps AutoML tools search faster. J48 has a categorical hyperparameter $R$ to enable reduced error pruning and a continuous confidence threshold $C$ for pruning. Mathematically, we denote allOf as $\wedge$, anyOf as $\vee$, and not as $\neg$. Even though the valid values for $C$ are $(0..1)$, search should only consider values $C\in(0..0.5)$. The constraint in Lines 16–19 encodes a conditional $(R=\textit{true})\Rightarrow(C=0.25)$ by using the equivalent $\neg(R=\textit{true})\vee(C=0.25)$. LR has two categorical hyperparameters $S$ (solver) and $P$ (penalty), with a constraint that solvers sag and lbfgs only support penalty l2, which we already encountered in Figure 6. Going forward, we will denote a JSON Schema object with properties as dict{} and a JSON Schema enum as $[\;]$. Eliding priors, this means Figure 7 becomes:

To auto-configure an operator means to automatically capture $\theta_{\text{hyperparams}}$, which involves jointly selecting algorithms (for operator choices) and tuning hyperparameters (for individual operators). We saw examples for doing this with scikit-learn’s GridSearchCV (Buitinck et al., 2013), hyperopt-sklearn (Komer et al., 2014), and auto-sklearn (Feurer et al., 2015) in Figures 3, 4, and 5. Lale offers a single unified syntax shown in Figure 1 Lines 15–16 to address problem $P_{2}$ from Section 2. Mathematically, let op be either an individual operator or a pipeline or choice whose $\theta_{\text{topology}}$ or $\theta_{\text{steps}}$ are already captured. That means op has the form $\theta_{\text{hyperparams}}\to\mathcal{D}_{\text{fit}}\to\mathcal{D}_{\text{in}}\to\mathcal{D}_{\text{out}}$. Then $\textit{auto\_configure}(\textit{op},\mathcal{D}_{\text{fit}})$ returns a function of the form $\mathcal{D}_{\text{in}}\to\mathcal{D}_{\text{out}}$. Section 4 discusses how to implement auto_configure using schemas.

AutoML users rarely want to completely surrender all decisions to the automation. Instead, users typically want to control certain decisions based on their problem statement and expertise. This section discusses how Lale supports such controlled AutoML.

The lifecycle state of an operator is the number of curried arguments it has already captured that determines the functionality it supports. To enable controlled AutoML, Lale pipelines can mix operators with different states. The visualizations in Figure 1 indicate lifecycle states via colors. A planned operator, shown in dark blue, has captured only $\theta_{\text{topology}}$ or $\theta_{\text{steps}}$, but $\theta_{\text{hyperparams}}$ is still latent. Planned operators support auto_configure but not fit or predict. A trainable operator, shown in light blue, has also captured $\theta_{\text{hyperparams}}$, leaving $\mathcal{D}_{\text{fit}}$ latent. Trainable operators support auto_configure and fit but not predict. A trained operator, shown in white, also captures $\mathcal{D}_{\text{fit}}$. Trained operators support auto_configure, fit, and predict. Later states subsume the functionality of earlier states, enabling user control over where automation is applied. The state of a pipeline is the least upper bound of the states of its steps.

Partially captured hyperparameters of an individual operator treat some of its hyperparameters as given while keeping the remaining ones latent. Specifying some hyperparameters by hand lets users control and hand-prune the search space while still searching other hyperparameters automatically. For example, Figure 1 Line 12 shows partially captured hyperparameters for SVM and RFC. The visualization after Line 13 reflects this in a tooltip for RFC. And the pretty-printed code after Line 20 shows that automation respected the given dual hyperparameter of SVM while capturing the latent C, penalty, and tol. Individual operators with partially captured hyperparameters are trainable: fit uses defaults for latents.

Freezing an operator turns future auto_configure or fit operations into an identity. PCA(n_components=4).freeze_trainable() >> SVM freezes all hyperparameters of PCA (using defaults for its latents), so auto_configure on this pipeline tunes only the hyperparameters of SVM. Similarly, on a trained operator, freeze_trained() freezes its learned coefficients, so any subsequent fit call will ignore its new $\mathcal{D}_{\text{fit}}$. Freezing part of a pipeline speeds up (Auto-)ML.

A custom schema derives a variant of an individual operator that differs only in its schema. Custom schemas can modify ranges or distributions for search. As an extreme case, users can attach custom schemas to non-Lale operators to enable hyperparameter tuning on them—the call to wrap_imported_operators in Figure 1 Line 8 implicitly does that. Figure 8 shows how to explicitly customize a schema. Line 2 imports helper functions for expressing JSON Schema in Python. XGBoost implements a forest of boosted trees, so to get a small forest (a Grove), Line 5 restricts the number of trees to $[2..6]$. Line 6 restricts the booster to a constant, using a singleton enum. Afterwards, Grove is an individual operator like any other. Mathematically, we can view the ability to attach a schema to an individual operator as extending its curried function on the left:

A higher-order operator is an operator that takes another operator as an argument. Scikit-learn includes several higher-order operators including RFE, AdaBoostClassifier, and BaggingClassifier. The nested operator is a hyperparameter of the higher-order operator and the nested operator can have hyperparameters of its own. Finding the best predictive performance requires AutoML to search both outside and inside higher-order operators. Figure 9 shows an example higher-order AdaBoostClassifier with a nested DecisionTreeClassifier. On the outside, AutoML should search the operator choice in Lines 2–3 and the hyperparameters of AdaBoostClassifier such as n_estimators. On the inside, AutoML should tune the hyperparameters of DecisionTreeClassifier. Lale searches both jointly, helping solve problem $P_{3}$ from Section 2. The JSON schema of the base_estimator hyperparameter of AdaBoostClassifier is:

A pipeline grammar is a context-free grammar that describes a possibly unbounded set of pipeline topologies. A grammar describes a search space for the $\theta_{\text{topology}}$ and $\theta_{\text{steps}}$ arguments needed to create planned pipelines and operator choices. Grammars formalize AutoML tools for topology search such as TPOT (Olson et al., 2016), Recipe (de Sá et al., 2017), and AlphaD3M (Drori et al., 2019) that capture $\theta_{\text{topology}}$ and $\theta_{\text{steps}}$ automatically. Lale provides a grammar syntax that is a natural extension of concepts described earlier, helping solve problem $P_{3}$ from Section 2.

Figure 10 shows a Lale grammar inspired by the AlphaD3M paper (Drori et al., 2019). It describes linear pipelines comprising zero or more data cleaning operators, followed by zero or more transformers, followed by exactly one estimator. This is implemented via recursive non-terminals: g.clean on Line 5 is a recursive definition, and so is g.tfm on Line 6, implementing a search space with sub-pipelines of unbounded length. While AlphaD3M uses reinforcement learning to search over this grammar, Figure 10 does something far less sophisticated. Line 13 unfolds the grammar to depth 3, obtaining a bounded planned pipeline, and Line 14 searches that using hyperopt, with no further modifications required.

Figure 11 shows a Lale grammar inspired by TPOT (Olson et al., 2016). It describes possibly non-linear pipelines, which use not only >> but also the & combinator. Recall that (x & y) >> Concat applies both x and y to the same data and then concatenates the features from both. Besides transformers, Line 6 also uses estimators with &, turning their predictions into features for downstream operators. This is not supported in scikit-learn pipelines but is supported in Lale.

Progressive disclosure is a design technique that makes things easier to use by only requiring users to learn new features when and as they need them. The starting point for Lale is manual machine learning, and thus, the scikit-learn code in Figure 2 is also valid Lale code. User needs to learn zero new features if they do not use AutoML. To use algorithm selection, users only need to learn about the | combinator and the auto_configure function. To express pipelines more concisely, users can learn about the >> and & combinators, but those are optional syntactic sugar for make_pipeline and make_union from scikit-learn. To use hyperparameter tuning, users only need to learn about wrap_imported_operators. To exercise more control over the search space, users can learn about freeze and custom schemas. While schemas are a non-trivial concept, Lale expresses them in JSON Schema (Pezoa et al., 2016), which is a widely-adopted and well-documented standard proposal. To use higher-order operators, users need not learn new syntax, as Lale supports scikit-learn syntax for them. Finally, to use grammars, users need to add ‘g.’ in front of their pipeline definitions; however, all the other features, such as the | combinator and the auto_configure function, continue to work the same with or without grammars.

Lale offers two approaches for using a grammar with GridSearchCV, hyperopt, and SMAC: unfolding and sampling. Both approaches produce a planned pipeline, which can be directly used as the input for the compiler in Section 4.2. Unfolding and sampling are merely intended as proof-of-concept baseline implementations. In the future, we will also explore integrating Lale grammars directly with AutoML tools that support them, such as AlphaD3M (Drori et al., 2019).

Unfolding first expands the grammar to a given depth, such as 3 in the example from Figure 10 Line 13. Then, it prunes all disjuncts containing unresolved nonterminals, so that only planned Lale operators (individual, pipeline, or choice) remain.

Sampling traverses the grammar by following each nonterminal, picking a random step in each choice, and unfolding each pipeline. The result is a planned pipeline without any operator choices.

This section sketches how to map a planned pipeline (which includes a topology, steps for operator choices, and schemas for individual operator) to a search space in the format required by GridSearchCV, hyperopt, or SMAC. The running example for this section is the pipeline PCA >> (J48 | LR) with the individual operator schemas in Figure 7.

Lale’s search space generator has two phases: normalizer and backend. The normalizer translates the schemas of individual operators separately. The backend combines the schemas for the entire pipeline and finally generates a tool-specific search space.

The normalizer processes the schema for an individual operator in a bottom-up pass. The desired end result is a search space in Lale’s normal form, which is $\vee(\textrm{dict}\texttt{\{}\textrm{cat}^{*},\textrm{cont}^{*}\texttt{\}}^{*})$. At each level, the normalizer simplifies children and hoists disjunctions up.

The backend starts by first combining the search spaces for all operators in the pipeline. Each pipeline becomes a ‘dict’ over its steps; each operator choice becomes an ‘$\vee$’ over its steps with added discriminants $D$ to track what was chosen; and each individual operator simply comes from the normalizer. This yields an intermediate representation (IR) whose nesting structure reflects the operator nesting of the original pipeline. For our running example, this is:

The remainder of the backend is specific to the targeted AutoML tools, which the following text describes one by one.

The hyperopt backend of Lale is the simplest because hyperopt supports nested search space specifications that are conceptually similar to the Lale IR. For instance, an exclusive disjunction ‘$\vee$’ from the IR can be translated into a hyperopt hp.choice, an example for which occurs in Figure 4 Line 5. Similarly, a ‘dict’ from the IR can be translated into a Python dictionary that hyperopt understands. For working with higher-order operators, Lale adds additional markers that enable it to reconstruct nested operators later.

The SMAC backend has to flatten Lale’s nested IR into a grid of disjuncts with discriminants $D$. To do this, it internally uses a name mangling encoding that extends the __ mangling of scikit-learn, an example for which occurs in Figure 3 Line 7. Each element of the grid needs to be a simple ‘dict’ with no further nesting. For our running example, the result in mathematical notation is:

The GridSearchCV backend starts from the same flattened grid representation that is also used by the SMAC backend. Then, it discretizes each continuous hyperparameter into a categorical by first including the default and then sampling a user-configurable number of additional values from its range and prior distribution (such as uniform in Figure 7 Line 7). The generated search space in mathematical notation is:

To demonstrate the use of Lale, we designed four experiments that specify different search spaces for OpenML classification tasks.

lale-pipe::

The three-step planned lale_openml_pipeline in Figure 12.

lale-ad3m::

The AlphaD3M-inspired grammar of Figure 10 unfolded with a maximal depth of 3.

lale-tpot::

The TPOT-inspired grammar of Figure 11 unfolded with a maximal depth of 3.

lale-adb::

The higher-order operator pipeline of Figure 9.

For comparison, we used auto-sklearn (Feurer et al., 2015) — a popular scikit-learn based AutoML tool that has won two OpenML challenges — with its default setting as a baseline. We chose 15 OpenML datasets for which we could get meaningful results (more than 30 trials) using the default settings of Auto-sklearn. The selected datasets comprise 5 simple classification tasks (test accuracy $>90\%$ in all our experiments) and 10 harder tasks (test accuracy $<90\%$). For each experiment, we used a $66\%-33\%$ validation-test split, and a 5-fold cross validation on the validation split during optimization. Experiments were run on a 32 cores (2.0GHz) virtual machine with 128GB memory, and for each task, the total optimization time was set to 1 hour with a timeout of 6 minutes per trial.

Table 1 presents the results of our experiments. For each experiment, we report the test accuracy of the best pipeline found averaged over 5 runs. Note that for the shuttle dataset, 3 out of 5 runs of auto-sklearn resulted in a MyDummyClassifier being returned as the result. Since we were trying to evaluate the default settings, we did not attempt debugging it, but according to the tool’s issue log, other users have encountered it before. The column askl-adb reports the accuracy of auto-sklearn when the set of classifiers is limited to AdaBoost with only data pre-processing. These results are presented for comparison with the lale-adb experiments, as the data preprocessing operators in Figure 9 were chosen to match those of auto-sklearn as much as possible. Also note that the default setting for auto-sklearn uses meta-learning.

The results show that carefully crafted search spaces (e.g., TPOT-inspired grammar, or pipelines with higher-order operators) and off-the-shelf optimizers such as hyperopt can achieve accuracies that are competitive with state-of-the-art tools. These experiments thus validate Lale’s controlled approach to AutoML as an alternative to black-box solutions. In addition, these experiments illustrate that Lale is modular enough to easily express and compare multiple search spaces for a given task.

While all the examples so far focused on tasks for tabular datasets, the core contribution of Lale is not limited to those. This section demonstrates Lale’s versatility on three datasets from different modalities. Table 2 summarizes the results.

Text.:

We used the Drug Review dataset for predicting a rating given by a patient to a drug. The Drug Review dataset has a text column called review, to which the pipeline applies either TfidfVectorizer from scikit-learn or a pretrained BERT embeddings, which is a text embedding based on neural networks. The dataset also has numeric columns, which are concatenated with the result of the embedding.

⬇

1planned_pipeline = (

2 Project(columns=[’review’]) >> (BERT | TfidfVectorizer)

3 & Project(columns={’type’: ’number’})

4) >> Cat >> (LinearRegression | XGBRegressor)

Image.:

We used the CIFAR-10 computer vision dataset. We picked the ResNet50 deep-learning model, since it has been shown to do well on CIFAR-10. Our experiments kept the architecture of ResNet50 fixed and tuned learning-procedure hyperparameters.

Time-series.:

We used the Epilepsy dataset, a subset of the TUH Seizure Corpus, for classifying seizures by onset location (generalized or focal). We used a three-step pipeline:

⬇

1planned = Window \

2 >> (KNeighborsClassifier| XGBClassifier| LogisticRegression) \

3 >> Voting

We implemented a popular pre-processing method (Schindler et al., 2006) in a Window operator with three hyperparameters $W$, $O$, and $T$. Note that this transformer leads to multiple samples per seizure. Hence, during evaluation, each seizure is classified by taking a vote of the predictions made by each sample generated from it.

Lale’s search space compiler takes rich hyperparameter schemas including side constraints and translates them into semantically equivalent search spaces for different AutoML tools. This raises the question of how important those side constraints are in practice. To explore this, we did an ablation study where we generated not just the constrained search spaces that are default with Lale but also unconstrained search spaces that drop side constraints. With hyperopt on the unconstrained search space, some iterations are unsuccessful due to exceptions, for which we reported np.float.max loss. Figure 14 plots the convergence for the Car dataset on the planned pipeline LR | KNN. Both of these operators have a few side constraints. Whereas the unconstrained search space causes some invalid points early in the search, the two curves more-or-less coincide after about two dozen iterations. The story looks very different in Figure 14 when adding a third operator J48 | LR | KNN. In the unconstrained case, J48 has many more invalid runs, causing hyperopt to see so many np.float.max loss values from J48 that it gives up on it. In the constrained case, on the other hand, J48 has no invalid runs, and hyperopt eventually realizes that it can configure J48 to obtain substantially better performance.

In order to be specific about the exact datasets for reproducibility, Table 3 report the URLs for accessing those.