Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
[Supplemental Material]

Ablation analysis (DBLP:conf/aaai/BiedenkappLEHFH17) selects the features whose changes contribute the most to improve the performance of configurations. We now describe how ablation analysis quantifies the performance change due to a certain feature’s change. Given a default configuration $\bm{\theta}_{default}$ and a target configuration $\bm{\theta}_{target}$ (usually a better one), ablation analysis first computes the feature differences $\Delta(\bm{\theta}_{default},\bm{\theta}_{target})$ between the default and target configurations. Next, an ablation path $\bm{\theta}_{default},\bm{\theta}_{1},\bm{\theta}_{2},\dots,\bm{\theta}_{target}$ is iteratively constructed. In each iteration $i$ with previous ablation path configuration $\bm{\theta}_{i-1}$, we consider all remaining feature changes $\delta\in\Delta(\bm{\theta}_{default},\bm{\theta}_{target})$ and apply the change to the previous ablation path configuration and obtain the candidate $\bm{\theta}_{i-1}[\delta]$. Each parameter change $\delta$ is a modification of one feature from its value in $\bm{\theta}_{i-1}$ to its value in $\bm{\theta}_{target}$, along with any other feature modifications that may be necessary due to conditionally constraints in $\bm{\Theta}$. The next configuration on the ablation path $\bm{\theta}_{i}$ is the candidate $\bm{\theta}_{i-1}[\delta]$ with the best objective performance $f$. The performance evaluation is approximated via surrogate for efficiency reasons. The order that the feature is changed is the importance rank from the ablation analysis. Given a set of observations, we conduct the ablation analysis between the default configuration and the configurations with better performance than the default in the observation set (i.e., target configurations). For each feature, we use the average rank from each ablation path as the final ranking of importance.

SHAP (DBLP:conf/nips/LundbergL17) (SHapley Additive exPlanation) uses Shapley values of a conditional expectation function of the original model. SHAP values attribute to each feature the change in the expected model prediction when conditioning on that feature. They explain how to get from the base value that would be predicted if we did not know any features to the current output. When the model is non-linear or the input features are not independent, the order in which features are added to the expectation matters, and the SHAP values arise from averaging the contributing values across all possible orderings (DBLP:conf/nips/LundbergL17). The exact computation of SHAP values is challenging, which can be estimated by Shapley sampling values method (DBLP:journals/kais/StrumbeljK14) or Kernel SHAP method (DBLP:conf/nips/LundbergL17).

SMAC (DBLP:conf/lion/HutterHL11) constructs a random forest as a set of regression trees, each of which is built on n data points randomly sampled with repetitions from the entire training data set. It computes the random forest’s predictive mean $\hat{\mu}(\bm{\theta})$ and variance $\hat{\sigma}^{2}(\bm{\theta})$ for a new configuration $\bm{\theta}$ as the empirical mean and variance of the Gaussian distribution . SMAC uses the random forest model to select a list of promising parameter configurations. To quantify how promising a configuration $\bm{\theta}$ is, it uses the model’s predictive distribution for $\bm{\theta}$ to compute its expected positive improvement $EI(\bm{\theta})$  (DBLP:journals/jgo/JonesSW98) over the best configuration seen so far. $EI(\bm{\theta})$ is large for configurations $\bm{\theta}$ with high predicted performance and for those with high predicted uncertainty; thereby, it offers an automatic trade-off between exploitation (focusing on known good parts of the space) and exploration (gathering more information in unknown parts of the space). To gather a set of promising configurations with low computational overhead, SMAC performs a simple multi-start local search and considers all resulting configurations with locally maximal $EI$.

RGPE (feurer2018scalable) is a scalable meta-learning framework to accelerate BO-based optimizer. First, for each previous tuning task $T_{i}$, it trains a base Gaussian process (GP) model $M_{i}$ on the corresponding observations from $H_{i}$. Then it builds a surrogate model $M_{meta}$ combine the base GP models, instead of the original surrogate $M_{T}$ fitted on the observations $H_{T}$ of the target task only. The prediction of $M_{meta}$ at point $\bm{\theta}$ is given by:

(1)

$$y\sim N(\sum_{i}{w_{i}\mu_{i}(\bm{\theta})},\sum_{i}{w_{i}\sigma^{2}_{i}(\bm{\theta})}),$$

where $w_{i}$ is the weight of base surrogate $M_{i}$, and $\mu_{i}$ and $\sigma^{2}_{i}$ are the predictive mean and variance of the base surrogate $M_{i}$. The weight $w_{i}$ reflects the similarity between the previous task and the current task. Therefore, $M_{meta}$ utilizes the knowledge on previous tuning tasks, which can greatly accelerate the convergence of the tuning in the target task. We then use the following ranking loss function $L$, i.e., the number of misranked pairs, to measure the similarity between previous tasks and the target task:

(2)

$$L(M_{j},H_{T})\!=\!\sum_{j=1}^{n_{t}}\sum_{k=1}^{n_{t}}\mathbbm{1}\!\Big{(}(M_{i}(\bm{\theta}_{j})\!\leq\!M_{i}(\bm{\theta}_{k}))\!\oplus(y_{j}\leq y_{k})\Big{)},$$

where $\oplus$ is the exclusive-or operator, $n_{t}$ denotes the number of tuning tasks and $M_{i}(\bm{\theta}_{j})$ means the prediction of $M_{i}$ on configuration $\bm{\theta}$. Based on the ranking loss function, the weight $w_{i}$ is set to the probability that $M_{i}$ has the smallest ranking loss on $H_{T}$, that is, $w_{i}=\mathbbm{P}(i=\mathop{\arg\min}_{j}L(M_{j},H_{T}))$. This probability can be estimated using MCMC sampling (DBLP:conf/aistats/MartensTY19).

While we have presented general profile information for workloads in Table  LABEL:wkl in the paper, we detail the implementation and the reason we select those workloads in this section.

Average ranking of optimizers in terms of the best configuration they found. While we have presented the average rankings of optimizers in Table 7 in the paper, we detail the rankings on each workload and configuration space as shown in Table S4. In addition, we present how we calculate the average ranking of optimizers. For each workload and configuration space, we run three tuning sessions for an optimizer and sort the three sessions in terms of the best performance they found within 200 iterations. Then, we rank the optimizers based on the best performance in their best session, and then rank them based on their second session, and lastly the worst session. Finally, we average the three ranks of an optimizer, which corresponds to a row in Table S4.

We have demonstrated the average performance enhancement (i.e., PE), speedup, and absolute performance ranking (i.e., APR) in Table 8 and omit the performance plot in the paper due to space constraints. Figure S2 plots the absolute performance over iteration of each baseline (i.e., the combination of transfer framework and base learner). On TPCC, both RGPE (Mixed-kernel BO) and RGPE (SMAC) find the approximately best performance in 200 iterations, while RGPE (SMAC) has a better speedup. On SYSBENCH, RGPE (SMAC) finds the best performance, though it takes a few more steps. On Twitter, RGPE (Mixed-kernel BO) finds the best performance, and at a fast speed. In general, we find that the combinations of RGPE and base learners have the best absolute performance as well as speedup.