Automatically Reconciling the Trade-off between Prediction Accuracy and Earliness in Prescriptive Business Process Monitoring

Predictive business process monitoring forecasts the future state of an ongoing business process instance (a.k.a. case) by using data produced by the execution of the case so far together with historical data [2]. A broad range of techniques exists that address different prediction tasks, such as predicting the next activity, predicting the remaining execution time of the case, as well as predicting the outcome of the case [3, 4, 5, 6]. For example, predictive process monitoring may predict whether there will be a delay in a transport process, or whether an order-to-cash process will be completed successfully.

If the predicted future state of an ongoing case indicates a deviation from the expected future state, process managers may intervene by proactively adapting the case; e.g., by re-scheduling process activities or by changing the assignment of resources [7, 8, 9]. This can help prevent a deviation or at least mitigate the impact of a deviation [10, 11, 12]. As an example, a delay in the expected delivery time for a freight transport process may incur contractual penalties [13]. If during process execution, a delay is predicted, the execution of faster, alternative transport activities (such as air delivery instead of road delivery) can be proactively scheduled to prevent the delay and thus avoid the contractual penalty.

State-of-the-art predictive process monitoring techniques can continuously generate predictions during the execution of a business process – typically whenever new data from the ongoing case arrives. This means there are multiple points in time, when process managers may decide whether or not to trust the current prediction and act upon it. One key limitation of predictive business process monitoring is that it provides limited support for process managers to decide which prediction to trust and act upon. This decision is important. While sophisticated prediction models (such as deep learning and ensemble models) and higher data volumes lead to impressive improvements in prediction accuracy [3, 4, 5], predictions will never be 100% accurate. This means that some predictions will be wrong. Deciding to adapt an ongoing case based on wrong predictions has negative consequences. On the one hand, false positive predictions may lead to unnecessary process adaptations. On the other hand, false negative predictions may mean that necessary process adaptations are missed.

Prescriptive business process monitoring provides decision support for business process managers, helping them to decide when and how to adapt an ongoing case [14, 15, 16, 17, 18].

We address the specific problem of when to intervene, i.e., whether and when to adapt an ongoing case. Answering this question provides the backbone of a prescriptive process monitoring system [19]. More precisely, we focus on how a prescriptive process monitoring technique can generate alarms [12, 15, 19]. An alarm suggests to a process manager to take action or it may directly feed into an automated decision-making process that adapts the running process instance. By addressing the question “when to intervene?”, we thereby complement related work on prescriptive process monitoring that answers “how to intervene?” (such as presented in [14, 20, 16, 18]).

Generating alarms entails reconciling a fundamental trade-off between prediction accuracy and prediction earliness [21, 12, 4]. On the one hand, alarms should ideally be based on accurate predictions. As motivated above, if decisions are taken based on inaccurate predictions, this may imply unnecessary adaptations or missed adaptations. On the other hand, alarms should ideally be raised early. The later an alarm is raised, the less time and options remain for proactively addressing process deviations [22, 4, 23]. However, earlier predictions typically have a lower prediction accuracy than later predictions, because less information about the ongoing case is available than for later predictions [4, 24].

Different approaches were presented in the BPM literature to reconcile this trade-off between prediction accuracy and earliness. They include:

• 1.

  Using a static prediction point chosen by using the average accuracy of the underlying prediction model [9, 25];

• 2.

  Considering the first prediction with a sufficiently high reliability [26, 21] – typically expressed in terms of a reliability threshold chosen via empirical thresholding [12, 15];

• 3.

  Dynamically deciding which prediction point to consider using online reinforcement learning (Online RL) [27, 28].

So far, these approaches were evaluated using different evaluation setups. The approaches were compared with different baselines; e.g., while the approach in [27] is compared with the approach in [12], the approach in [9] is only compared with the naïve baselines of never or always adapting. Different data sets were used for evaluating the approaches; e.g., [26] used the BPIC11¹¹1BPIC stands for Business Process Intelligence Challenge. and BPIC15 data sets, while [27] used the BPIC12 and BPIC17 data sets. Data sets were pre-processed differently; e.g., in [15], the BPIC17 event log data was subdivided into two sub-processes, while in [21, 28] the whole event log data set was used. Different splits into training and testing data were used; e.g., a 67%-33%-split in  [27, 21], 80%-20%-split in [15], and a 90%-10%-split in [26]. Also, confidential data was sometimes used, such as in [15, 12], which limits the replicability of results. These differences in evaluation setups make it difficult to use evaluation results from the literature to perform a fair comparison of state-of-the-art approaches in terms of their cost savings. In turn, it is difficult to identify the relative strengths and weaknesses of existing approaches and how to choose a concrete approach in practice.

This paper provides the following main contributions:

Overall, our work contributes to the emerging research area of AI-Augmented Business Process Management – ABPM [35]. We demonstrate how the ‘adaptation’ characteristic in ABPM can be automated and empowered by AI (i.e., reinforcement learning) to facilitate real-time adaptation. In addition, by connecting with the work on ECTS, we provide input to machine-learning-based early decision-making research [31].

Section 2 provides relevant fundamentals. Section 3 elaborates the problem and introduces state-of-the-art approaches for reconciling the trade-off between prediction accuracy and earliness. Section 4 describes the enhancements of our Online RL approach. Section 5 reports on the evaluation setup. Section 6 characterizes the data used to compare the approaches. Section 7 presents the results of our comparative evaluation. Section 8 provides initial practical recommendations. Section 9 discusses validity risks and directions for future work. Section 10 analyses related work.

Predictive business process monitoring forecasts how an ongoing business process instance, aka. case, will unfold. A case $k$ is characterized by a finite sequence $\sigma_{k}=\langle e_{1},\ldots,e_{l}\rangle$ of events, with $l$ being the length of the case. An event $e_{i}$ represents the execution of an activity [14, 36] and has at least two attributes: a categorical attribute event type describing the type of activity that was executed and a numeric attribute timestamp recording when the event occurred. Note that timestamps are typically not equidistant in contrast to time series [29]. An event may have further attributes, such as the resources or people that carried out the activity. An event log is a set of $\sigma_{k}$ [14].

Predictive business process monitoring utilizes prediction models trained on event logs [37, 6]. One may, e.g., predict the next activity [38], the remaining time [39], or the outcome of an ongoing case [4]. In this paper we focus on prescriptive business process monitoring approaches for outcome prediction, i.e., we aim to predict the label $\hat{y}$ associated to the complete sequence of events of a case.

Fig. 1 depicts the two main phases and the key steps of predictive business process monitoring. During the training phase, a prediction model is trained using event log data. During the induction phase, the prediction model is used to generate predictions about the ongoing case by using data from the ongoing case as input, typically in the form of a sequence of events executed up to the prediction point. For both training and induction, the process data typically needs to be encoded such as to serve as suitable input for a prediction model (e.g., see [4]).

Various types of prediction models were proposed for predictive business process monitoring. Increasingly, we see the use of sophisticated prediction models, such as random forests (e.g., gradient boosted trees [4, 5]), and deep artificial neural networks (e.g., LSTMs [16]). Compared to simple prediction models, such as decision trees or linear regression, these sophisticated prediction models achieve consistently better prediction accuracy in various types of predictive process monitoring problems [38, 39].

Informally, prediction accuracy characterizes the ability of a prediction model to predict as many true deviations from expected process outcomes as possible, while predicting as few false deviations as possible [40]. Obviously, predictions generated by a predictive business process monitoring system should be accurate in order to be useful [4]. When used for proactive process adaptation, this is especially important, because adaptation decisions are based on these predictions [11, 12, 41, 42].

To elaborate the need for accurate predictions, Table 1 depicts the four cases that can result from using predictions to decide on proactive process adaptation. The two cases in boldface are the critical ones.

Predictions can be made at different prediction points during the execution of a case. Predictions are early if they are made toward the beginning of a case. Ideally, one would like to use early predictions, as this gives more time and options for proactively addressing predicted process deviations [22, 4, 23].

There are different ways to define prediction points. As an example, prediction points can be explicitly defined by determining relevant activities or milestones from a process model. Such a process model may already exist or may be generated using process mining [43]. Such explicitly defined prediction points are often called checkpoints [22, 25] or decision points [4]. Prediction points may also be defined using equidistant points in time, i.e., using prediction windows of a given duration [44].

More typically, predictions are generated after each event $e_{j}$ of an ongoing case $\sigma_{k}$. Such prediction points can be characterized by their prefix length, which gives the number of events that were produced up to the point of the prediction [4, 45]. In this paper we characterize prediction points by their prefix length by giving the index $j$ of event $e_{j}\in\sigma_{k}$ for which a prediction is made.

A proactive adaptation decision entails different costs, which depend on whether an alarm is based on a false positive or a false negative prediction (see Section 2.2), and when the prediction was made during process execution (see Section 2.3). This leads to different costs $C(j)$ depending at which prefix length $j$ an alarm is raised²²2If no alarm is raised, we model this as $j=0$..

We use a cost model based on previous work in BPM [15, 27, 28, 21] and ECTS [31]. This cost model is shown in Table 2 and follows the structure of Table 1. The cost model considers the following parameters:

In general, the later an alarm is raised the less time remains for proactively addressing a deviation via proactive process adaptation [22, 4, 12, 21]. This can be important as adaptations typically have non-negligible latencies, i.e., it may take some time until they become effective [23]. As an example, dispatching additional personnel to mitigate delays in container transports may take several hours. Also, the later a case is adapted, the fewer adaptation options are available. As an example, while at the beginning of a transport process one may be able to transport a container by train instead by ship, once the container is on board the ship such an adaption is no longer possible. Finally, if an adaptation is performed late and turns out to be ineffective (i.e., not preventing the predicted deviation), not much time may remain for any remedial actions or further adaptations. This means one should favor earlier prediction points for raising alarms, thereby leaving sufficient time and options for process adaptation.

However, there is a conflict between generating accurate alarms and generating early alarms [22, 5, 21]. Typically, prediction accuracy increases as the case unfolds, because more information about the ongoing case becomes available. While early predictions tend to exhibit low prediction accuracy, later predictions typically exhibit higher prediction accuracy. This means later predictions have a higher chance of being accurate, thus one should favor later prediction points for raising alarms. However, later alarms leave less time and options for process adaptation.

Considering this trade-off means determining a prediction point $j^{*}$, which balances accuracy and earliness in such a way as to minimize costs $C(j^{*})$. On this conceptual level, the problem matches the problem of ECTS³³3With the exception that ECTS forces a decision, i.e., classification, while in prescriptive process monitoring we may not raise an alarm at all; modeled as $j=0$ as mentioned above. and can be expressed in the below equation [46]:

Obviously, we do not know the resulting costs for future timesteps and thus solving this optimization problem is not straightforward.

Below, we introduce three competitive state-of-the-art approaches and and one simple one to address the above problem. The simple one, we introduce first, may be the straightforward choice of a practitioner, and we include it to assess its limitations.

While this approach is simple to implement and thus may appear attractive from a practical point of view, it ignores the fact that early predictions may not be as accurate as later predictions, and thus may lead to many false alarms.

Similar approaches were proposed for ECTS and categorized as prefix-based early classification, where the idea is to learn a minimum prefix length using training data instances [29, 47] (also see Section 10.2).

Using a fixed prediction point has two main shortcomings. First, no alarms will be raised for cases that have a length that is shorter than $j^{*}_{\mathrm{fix}}$. While one may choose a very early prediction point that captures many or most of the cases, this comes with potentially lower prediction accuracy.

Second, the average accuracy of a prediction model does not provide direct information about the accuracy of an individual prediction for a concrete case [21]. In particular, for a given prediction model, the accuracy of individual predictions may differ across prediction points and cases [48]. These differences in prediction accuracy are not taken into account when choosing a fixed, static prediction point.

As a straightforward approach to leverage reliability estimates, one may use the earliest prediction with sufficiently high reliability to raise an alarm [4, 21, 50, 51, 12]. To determine whether the reliability is sufficiently high, one can set a concrete reliability threshold; e.g., 95%. Depending on how this threshold is set, one can trade earliness against accuracy [21, 12]. If earlier predictions are preferred, a lower threshold may be chosen, which raises alarms more speculatively at the risk that predictions are not very accurate. If accurate predictions are required, a higher threshold may be chosen, which raises alarms more conservatively and thus poses the risk that alarms may be raised too late as to be effective or that no alarm may be raised.

Similar approaches were proposed for ECTS and categorized as model-based approaches using discriminative classifiers, i.e., classifiers that provide a probability together with the actual prediction [29, 30, 52] (also see Section 10.2).

It can be difficult for a process manager to define a threshold that is optimal for a given situation, because, for instance, the optimal threshold depends on the actual type of business process and costs entailed in process execution. A recent BPM approach to address the problem of determining a threshold is Empirical Thresholding [12, 15]. In Empirical Thresholding, a dedicated training process – involving a dedicated training data set – is used to determine a suitable threshold. In the basic variant of Empirical Thresholding, a cost model, which defines adaptation, compensation and penalty costs, is used together with a dedicated training data set to compute the optimal threshold.

In [15], two variants of this basic Empirical Thresholding approach are suggested. First, it is suggested to introduce a so-called firing delay $p$. This means that an alarm is only raised if the reliability was above the threshold for $p$ consecutive prefix lengths. Second, it is suggested to train different thresholds for different (groups of) prefix lengths. While the rationale behind these variants is convincing, experimental results in [15] indicate that they only provide limited improvements. Both the firing delay approach and the multiple threshold approach only provided additional cost savings in around 8% of the experimental situations. We thus focus on the basic variant of Empirical Thresholding in the remainder of this paper.

One general concern of Empirical Thresholding is that while the threshold is optimal for the training data, it may not remain optimal over time. Concept drifts of the process environment and data [53, 54] may impact on prediction model accuracy and thus affect the reliability of individual predictions.

Similar RL-based approaches were proposed for ECTS [29, 31, 32] (also see Section 10.2).

There is one critical assumption encoded in earlier Online RL approaches, such as presented in our previous work [27, 28] and in ECTS research [31, 32]. The assumption is hat one can assess whether an adaptation was correct by determining the alternative process outcome if that adaptation were not executed. Yet, knowing such alternative process outcome once the process has been adapted may not be feasible in general, as it would require an accurate and reliable what-if business process analysis [33]. This posed an important limitation for the practical application of our earlier approach.

Here, we eliminate this assumption by introducing the concept of artificial curiosity into the reinforcement learning process as explained below.

In general, RL aims at finding an optimal action selection policy $\pi$ for a given environment by interacting with this environment. A policy $\pi$ defines a mapping from environment states to actions. Upon executing an action $a$ in a state $s$, the environment transitions to the next state $s^{\prime}$ and awards a specific numeric reward based on a reward function $r(s,a)$. An optimal policy $\pi$ is a policy that optimizes the cumulative rewards received [55].

Policy-based RL offers several benefits. First, it can cope with multi-dimensional, continuous state spaces, which we face due to the input for the Online RL approach including the continuous variables $\rho_{j}$ and $\delta_{j}$ [27]. Second, it can generalize over unseen neighboring states. Third, and most importantly, it can readily cope with non-stationarity via the aforementioned stochastic action selection and thereby can address concept drifts of the underlying prediction model.

Deep neural networks are also used in another class of deep RL, which is value-based deep RL [55]. Here, the neural network is used to represent the action-value-function $Q(s,a)$, which gives the expected reward when taking action $a$ in state $s$. Value-based RL requires explicitly implementing a policy function to determine whether – in given state – the agent should exploit the current policy (i.e., choose the best action based on $Q$) or to explore new actions (e.g., by randomly selecting an action). As we discussed in our earlier work, this imposes the additional engineering challenge on determining how to balance exploitation and exploration, in particular in the presence of concept drifts [28, 27].

The successful application of RL depends on how well the learning problem, and in particular the reward function, is defined [58]. By defining a reward function, one expresses the learning goal in a declarative fashion. As mentioned above, the goal of RL is to maximize cumulative rewards. Therefore, designing a suitable reward function is key to successful learning.

Note that the definition of the reward function is an inherent and fixed element of our Online RL approach, and there is no need to fine-tune it for specific data sets.

When designing a suitable reward function for reconciling the trade-off between prediction accuracy and earliness, the particular challenge is that we need to assess whether raising an alarm was the right decision. This requires determining the alternative process outcome if such an alarm were not raised and consequently no process adaptation were performed. As motivated in Sections 1 and 3, knowing such alternative process outcome once the process has been adapted is not feasible in general. Not knowing the alternative process outcome after an adaptation means that we lack an essential element of the reward function that would punish false alarms; e.g., by awarding negative reward values for false alarms.

We address this lack of an essential element of the reward function by leveraging the concept of artificial curiosity [34]. The principal idea of artificial curiosity is that instead of only using extrinsic rewards, we also use intrinsic rewards in the definition of the reward function. While an extrinsic reward has to be provided externally by the environment (i.e., computed from environment states), an intrinsic reward can be created from information available internally to the RL algorithm. In the reward function for Online RL, we include intrinsic rewards that positively reward transitioning to previously unexplored states. This enables the RL system to explore its environment even in the absence of extrinsic rewards.

As shown in Fig. 3, we differentiate between three kinds of end states that result from taking an adaptation decision and thus should be explored.

The parameter $d\in[0,1]$ is the rate of adaptations among the last seen 30 cases. As it has the negative coefficient $-2$, $d$ punishes high adaptation rates and thus fosters exploring not raising an alarm. We choose to average over 30 cases, as this provides a working compromise between the parameter changing quickly enough, yet not too erratically. When the RL policy update leads to a lower adaptation rate, the parameter $d$ changes quickly enough to prevent the policy update from further lowering the adaptation rate. On the other hand, the parameter does not change too quickly to prevent the policy update from effectively lowering the adaptation rate.

The parameters $b\in[\frac{1}{2},1]$ and $c\in[0,3]$ foster exploring different degrees of the earliness of alarms. The parameter $b$ decreases linearly with the prefix length $j$, being equal to $1$ on the first prediction point and $\frac{1}{2}$ on the last. Thereby, $b$ facilitates that early alarms should be preferred to late alarms.

The parameter $c$ is defined to be bi-linearly dependent on $d$ and $v$ as follows:

The parameter $v$ is the negative predictive value computed for the last $100$ non-adapted cases. We use the negative predictive value $v$ as an estimate for the accuracy of raising alarms and consider the last 100 cases to get a stable enough accuracy estimate. The negative predictive value $v$ is computed as follows:

The reward function parameter $c$ thereby encodes two assumptions we make about the learning process. First, when the current RL policy leads to a high negative predictive value $v$, indicating high accuracy in raising alarms, there is no longer a need to explore raising alarms later. Concretely, once $v$ nears or exceeds 70%, we assume that intrinsic rewards are no longer necessary, which results in $c=0$.

Second, when the adaptation rate $d$ is small, the extrinsic rewards will suffice to facilitate learning. Thereby, the parameter $c$ is gradually reduced to $0$, effectively leaving only the extrinsic rewards. In the overall reward function, if $c$ is equal or close to $3$, this reinforces late adaptations, if $c$ is equal to $0$ or at least smaller than 1, this reinforces early adaptations.

Figure 4 plots the curiosity modifier $c$ over the negative predictive value $v$ with regard to several adaptation rates $d$. It demonstrates visually the relationship between the three variables as defined by Equation 2: only for low value of $v$ and high values of $d$ is the curiosity modifier $c$ higher than the threshold of 1.

The Online RL approach requires the following inputs that form the state space for the RL algorithm.

Typically, reliability estimates computed from ensembles of prediction models (a typical example are random forests) can provide good estimates of the probability that an individual prediction is correct [61]. Ensemble prediction is a meta-prediction technique where the predictions of $m$ base prediction models are combined. While the main aim of ensemble prediction is to increase prediction accuracy, ensemble prediction also allows computing reliability estimates [9, 21].

Assuming that for a prediction point $j$ we are given the predictions of each base prediction model $i=1,\ldots,m$ of the ensemble as $\delta_{j,i}$. One straightforward yet effective way to determine the reliability estimate $\rho_{j}$ is to compute the fraction of the predictions of the individual prediction models $i$ as follows [9, 21]:

This way of computing reliability estimates facilitates the practical application of Online RL. First, many ensemble prediction models directly provide such an estimate (e.g, in form of class probabilities of random forests). Second, compared to other reliability approaches, it does not require additional tuning steps. For example, ”conformal prediction” requires defining a suitable non-conformity measure and calibration [62, 19], while ”corrected variance” requires hyper-parameter tuning [63].

As we are dealing with an episodic RL problem (see Figure 3), $l$ is known at the end of the episode and can be used to compute the reward signal.

We experiment with two alternative ways of realizing the prediction models. On the one hand, we use ensembles of random forests (concretely regression forests), as a representative of rule-based learning [64]. On the other hand, we use ensembles of deep artificial neural networks, concretely LSTMs, as a representative of perceptron-based learning [64]. Both random forests and LSTMs are widely used for predictive process monitoring [4, 65].

These two realizations exhibit different advantages and shortcomings. On the one hand, LSTMs can directly handle arbitrary-length sequences of events, while random forests require the encoding of such event sequences into fixed-length input vectors [4, 66, 6]. On the other hand, random forests can be trained rather efficiently [4], while LSTMs require significant time and resources for training [21].

While research in predictive process monitoring used individual LSTM models [16], in our earlier work we introduced ensembles of LSTMs for the sake of generating reliability estimates [21, 24], which we also use for the experiments reported below. We use bagging as a concrete ensemble technique.

We executed the following steps for both types of prediction models: To train a prediction model, we used 67% of the event log as training data, while we used the remaining 33% as ”test” data, i.e., as data used as input to the approaches being compared.

For the sake of generalizability, we used a naïve approach to select the input features from the respective event logs. This means we used all input features that were available (provided an input feature did not reveal the process outcome) and did not perform any additional feature engineering or selection. We used one-hot encoding to treat categorical (i.e., non-numeric) features such as event labels.

For the size of the ensemble, we chose an ensemble size of $m=100$. In earlier experiments, we varied ensemble sizes from 2 to 100. The size of the ensemble did not lead to different principal findings [9]. There was a trend that larger ensembles generally delivered predictions with higher accuracy. More importantly, larger ensembles delivered more fine-grained reliability estimates.

As for bootstrap size (i.e., the size of the bags), we used 60%. Earlier experiments did not show a clear trend that larger bootstrap sizes would perform better than smaller ones and different bootstrap sizes did not impact the general shape of the experimental results [9].

As a principle, we used the standard hyper-parameters as provided by the respective implementations of the machine learning models. One main reason was that we were not interested in finding a prediction model with the highest accuracy, but wanted to evaluate how the approaches work for predictions even if they are of lower accuracy. Also, ensembles in general provide good prediction accuracy even if composed of weak individual prediction models [71]. This reduces the relevance of finding optimal hyper-parameter settings. Finally, hyper-parameter tuning for LSTMs can become prohibitively expensive, because the training of an individual LSTM model takes considerable resources and time. This is exacerbated when using $n$-fold cross-validation, a typical approach to reliably estimate the model performance for a given set of hyper-parameters.

As mentioned in Section 5.1, random forests require the encoding of sequential event sequences into fixed-length inputs. To facilitate as much comparability with the LSTM realization as possible, we encoded the event sequences in such a way as to retain the same amount of information as given to the LSTM models. Also, we chose an encoding that does not require tuning of additional parameters, such as cluster size or aggregation functions⁶⁶6Note that experimental results suggest that choosing an optimal encoding is challenging and may require choosing a different encoding and parameterization for each data set [72]..

Following [4], we performed the following two-step encoding: First, event sequences were divided into buckets, a strategy called “trace bucketing”. We used prefix-length bucketing, which means that each bucket $p$ contains the (partial) event sequences of length $p$. For each of the buckets, a random forest model was trained.

Second, we transformed the event sequences within each of the buckets into fixed-length input vectors, a strategy called “sequence encoding”. We use index encoding, which generates one feature per event and class attribute. Each feature that encodes an event or class attribute includes an index $(1,\ldots,p)$ that specifies where the event occurred in the case. Index encoding retains all the information in the event sequence and we thereby use the same information as for the LSTM models. Index encoding requires that all event sequences in a bucket have the same length, thus the decision to use prefix-length bucketing in the first step.

We realize the Online RL approach described previously in the same way we did in our earlier work [28, 27]. An exception is the reward function, which is realized as explained in Section 4.2. As a concrete deep RL algorithm, we use PPO (proximal policy optimization [75]), a state-of-the-art policy-based actor-critic algorithm. One main advantage of PPO is that it avoids too large policy updates by using a so-called clipping function. A too-large policy update could mean that Online RL misses the global optimum and remains stuck in a local optimum, or even that the policy is destroyed.

Also, compared with other RL algorithms, the PPO algorithm is rather robust for what concerns selecting hyper-parameter settings that facilitate stable learning. Thereby, we avoid extensive hyper-parameter tuning compared to similar algorithms and basically can use standard hyper-parameter settings. One exception is the discount factor $\gamma$, which defines the relevance of future rewards. Given the way we define the learning episodes (see Section 4.1), we set the discount factor to $\gamma=1$ in order not to discount the reward received for the end state of each case.

To represent the actor and the critic parts of PPO, we use two multi-layer perceptron networks with two hidden layers of 64 neurons each. The input layers of both networks consist of five neurons representing the three state variables ($\delta_{j}$, $\rho_{j}$, $\tau_{j}$), from which extrinsic rewards are computed, as well as the two variables adaptation rate $d$ and the negative predictive value $v$ (see Section 4.2), from which intrinsic rewards are computed. The output layer of the critic consists of one neuron representing the estimated value of a state, while the two neurons of the actor’s output layer represent the probability distribution for each of the two available actions ”alarm” and ”no alarm”. The concrete action is chosen via sampling from these probability distributions.

For our experiment, we parametrize the cost model introduced in Section 2.4 as follows. As we are interested in comparing the relative cost savings of the approaches, it is sufficient to work with normalized cost model parameters for determining $C(j)$. As a basis, we use normalized penalty costs of $C_{\textrm{p}}=100$. Like in [15], we express $C_{\mathrm{a}}$ and $C_{\mathrm{c}}$ relative to $C_{\textrm{p}}$. Concretely, we model this as:

By varying $\lambda$ and $\kappa$, we can reflect different situations that may be faced in practice concerning how costly a process adaptation and compensation may be.

We also vary $\alpha$ in our experiments such that it linearly decreases from $\alpha_{\mathrm{max}}=1$ for the first prediction point to $\alpha_{\mathrm{min}}$ for the last prediction point. We vary $\alpha_{\mathrm{min}}$ between two extreme settings to cover different possible circumstances: $\alpha_{\mathrm{min}}=1$ models that late adaptations are always feasible, while $\alpha_{\mathrm{min}}=0$ models that late adaptations are never feasible.

We vary the cost model parameters for the experiment as follows: $\lambda,\kappa,\alpha_{\mathrm{min}}\in\{0,0.25,0.75,1.0\}$ and consider all combinations of these values, leading to a total of 64 combinations.

Of the approaches, only Empirical Thresholding requires a cost model as input to determine the optimal threshold. As such a cost model may not be precisely known in practice or a-priori, we model the uncertainty of knowing the actual cost model parameters using the parameter $\xi\in\{0.025,0.1,0.175,0.25\}$. The parameter $\xi$ gives an envelope around the actual cost model parameters from which the actual cost model parameters that are used as input for Empirical Thresholding are uniformly sampled. As an example, if $\lambda_{\mathrm{actual}}$ is the actual value for the adaptation costs, then we sample from $[\lambda_{\mathrm{actual}}-\xi,\lambda_{\mathrm{actual}}+\xi]$.

The data sets used exhibit different characteristics as shown in Table 4. They cover different application domains: finance (BPIC12 and BPIC17), government (Traffic), and transport (Cargo), and also exhibit different kinds of deviations⁷⁷7Note that while Cargo has a real-valued deviation (i.e., delay in terms of hours), BPIC12, BPIC17, and Traffic have categorical deviations (violation or non-violation). To provide the numeric reward signal (relative predicted deviation) for our RL agent, we map a non-violation to 0.0 and a violation to 1.0, and train regression prediction models on these numeric outcome labels. To compute $\delta_{j}$, we set $A=0.5$ in Equation 4.. They differ in terms of the rate of actual deviation as well as in the size of the data set (note that we report the size of the ”test” data set, i.e., 67% of the actual data set as explained above). Also, they differ in terms of complexity and length of the process instances⁸⁸8Like in [4], we only consider prediction points up to a certain prefix length in order not to bias the results toward extremely long cases. Concretely, we consider the 99% quantile of all prefix lengths of the respective data set..

For each of the four data sets, we trained an LSTM ensemble and an RF prediction model, considering the successful and unsuccessful process outcomes as indicated in Table 4. Below, we provide a characterization of the prediction models in terms of their prediction accuracy and analyze potential concept drifts in terms of the prediction models’ accuracy. We provide this characterization as relevant context for explaining the evaluation results in Section 7.

For each of the data sets, Fig. 5 shows the average prediction accuracy for each possible prediction point (i.e., prefix length) as well as the average accuracy ($\varnothing$) across all prediction points. We measure the accuracy of the prediction models using the ”test” data (i.e., 33% of the overall data of the respective data set). Also, the figure shows the relative number of cases that reach the respective prefix length (grey bars underlying the curves).

We use contingency table (binary) metrics, because this conveys a better impression of the performance of the prediction model. The numeric prediction error of the regression model does not directly indicate whether the process outcome may violate the expected process outcome or not [42]. As contingency table metric, we use the Matthews Correlation Coefficient (MCC), which is robust against class imbalances [77]. Also, MCC is a more challenging metric to score high on and therefore provides more realistic estimates of real-world model performance. Using the four prediction contingencies (see Section 2.2), MCC $\in[-1,1]$ is defined as follows (1 indicating a perfect predictor; $-1$ indicating the opposite):

As one can observe in Fig. 5, in most data sets there is a tendency that prediction accuracy continuously increases as the cases unfold. There are exceptions though as discussed below.

Figure 6 shows the prediction accuracy per each case in the ”test” set. This shows how prediction accuracy may fluctuate over longer periods, thereby indicating concepts drifts. One reason for such fluctuations may be that the prediction models are presented with unseen and out-of-sample process monitoring data. We measure the prediction accuracy for each case in terms of the Mean Absolute Error (MAE) computed for all predictions $\hat{y}_{j}$ of a case; $j=1,\ldots,l$, where $l$ is the case length, and $y$ the actual process outcome:

Figure 7 provides an overview of the experimental results. Considering only situations in which proactive adaptation offers cost benefits, the figure provides (a) the relative number of situations in which an approach performs best, and (b) the average, relative cost savings per case in such situations.

The results in Figure 7(a) show that the more recent and advanced techniques – Empirical Thresholding and Online RL – outperform the more simplistic approaches. Comparing Empirical Thresholding and Online RL, we can observe that both tend to work in many situations, albeit with some exceptions that we discuss as part of our detailed analysis in Section 7.2 as well as in Section 8.

Figure 7(b) quantifies the extent of the cost savings for the two best-performing approaches, i.e., Empirical Thresholding and Online RL. To this end, the average, relative cost savings $c_{\mathrm{rel}}$ of the respective approach $x$ are computed using the costs of the approach $c_{x}$ and the costs of never adapting $c_{\mathrm{never}}$:

We can observe that both approaches consistently deliver cost savings. On overage, Empirical Thresholding delivers cost savings of 34% and Online RL delivers cost savings of 27%. The reasons for the smaller cost savings of Online RL are discussed as part of our detailed analysis below.

As a basis for a detailed analysis of the experimental results, Figures 8 and 9 show the average process execution costs per case for the different approaches and data sets (remember these are normalized costs as explained in Section 5.4). The figure shows the costs clustered by cost model parameters $\lambda$ (adaptation costs) and $\kappa$ (compensation costs), averaged over $\alpha$ (adaptation effectiveness). In addition, given the stochastic nature of Empirical and Online RL, we also report the standard deviation. Below, we refer to the different lines in the tables of Figure 8 as $(\lambda,\kappa)$.

In addition to the four state-of-the-art approaches (as introduced in Section 3), we also report the results when never adapting the case, i.e., the costs entailed if process execution would commence without intervention. This provides us an important baseline to understand under which situations proactive adaptation may not lead to cost savings. As can be observed, in situations where adaptation and/or compensation costs are high, proactive adaptation does not pay off corroborating the results of our earlier research [9, 21]. For example, proactive adaptation does not pay off for BPIC12-LSTM beginning with cost model parameters $(0.5,0.75)$, for BPIC17-RF beginning with $(0.75,0.25)$, and for Traffic-RF beginning with $(0.75,1)$.

We can observe that no single approach works best for all data sets and all cost model configurations. As a general observation, Empirical Thresholding tends to work best for cost model parameters that are in the middle range of values. For example, Empirical Thresholding works best for BPIC12-RF between $(0.5,0)$ and $(0.75,1)$, and for Cargo-LSTM between $(0.25,0.25)$ and $(0.5,1)$. Similarly, Online RL tends to work best for smaller cost model configurations. For example, Online RL works best for BPIC12-LSTM between $(0,0)$ and $(0.25,0.75)$, and in BPIC17-LSTM between $(0,0.9)$ and $(0.25,1)$. However, we can also see several exceptions to this general observation. We thus discuss the results for each data set individually, considering the data set and prediction model characteristics identified in Sections 6.

For RF, the change in prediction accuracy (as analyzed in Section 6.3) is much smaller. For Online RL, the changes in the rate and earliness of alarms are so small as to fall within the variance of these metrics. Again, Empirical Thresholding keeps the rates of alarms and earliness roughly the same. Taking into account the variance (standard deviation) of the approaches, Empirical Thresholding and Online RL perform roughly the same with a difference of 5% as shown in Figure 7(a). Empirical Thresholding works quite well even though average accuracy drops visible after around prefix length 35. As discussed in Section 6.2, such negligible impact was expected, because only 7% of cases reach prefix length 35 or higher.

The BPIC17-LSTM and BPIC17-RF basically exhibit no change in prediction accuracy (see Section 6.3) and thus the rate of alarms and earliness remains roughly the same. There is an abrupt change in accuracy at the very end, i.e., after around case # 10,000. Yet, the remaining cases are too few to observe how Online RL may respond to it. As shown in Figure 7(a), Empirical Thresholding and Online RL perform similarly for BPIC17-LSTM with a difference of 6%.

For BPIC17-RF, Online RL outperforms Empirical Thresholding with 69% against 19% (see Figure 7(a)). We attribute this to the shape of the average accuracy curve shown in Figure 5. In contrast to LSTM, the accuracy radically drops after around prefix length 40. As around 32% of all cases have a length of 40 or longer, this means that an empirically determined threshold (which is computed considering all cases) may not be optimal for this relatively high number of remaining cases.

The reason that Online RL outperforms Empirical Thresholding for Traffic-LSTM lies in the high level of concept drift. The data set shows visible changes in prediction accuracy, which happen between around case # 14,000 and # 16,000, between around case # 31,000 and # 33,000 and again after case # 45,000.

To illustrate what happens for this specific data set, Figure 10 visualizes the behavior of the two approaches¹⁰¹⁰10We have averaged the values for each of these metrics over the last 100 cases for reasons of visual stability and charts thus start at case # 100.. For Empirical Thresholding, we chose a cost model configuration by using the average value for $\alpha_{\mathrm{min}}$, i.e., $0.5$. We then use the first $\lambda$ and $\kappa$ where Empirical Thresholding outperforms the other approaches. For Online RL, we chose the run (out of the 10 runs) that led to the highest rate of correct adaptation decisions.

As can be seen, Online RL responds to this concept drift by changing the rate of alarms and earliness. In contrast, Empirical Thresholding responds differently. For the first two changes in prediction accuracy, it shows a much smaller response, basically keeping earliness roughly the same, but changing slightly the rate of alarms. For the last change in prediction accuracy, the earliness shows major fluctuations, while Online RL can keep earliness stable. As a result, Online RL outperforms Empirical Thresholding with 84% to 0%.

The reason why Empirical Thresholding outperforms Online RL for Traffic-RF – or rather why Online RL fails – lies in the shape of the accuracy curve. Here, after prediction point 1, the accuracy significantly decreases and only increases again after prediction point 3. The way we formulate artificial curiosity in Online RL (see Section 4) is that we start with the curious exploration at the later prediction points. In this specific data set it means, that the curious exploration will be stuck in a local maximum at prefix lengths 4 and greater and will not be able to overcome the trench to reach prediction point 1. As a result, the earliness of Online RL is low, which negatively impacts process costs.

When analyzing the actual thresholds computed by Empirical Thresholding for LSTM-RF, it turns out that for 90% of the cost model parameters the threshold is equal to $0.5$. This means that effectively there is no threshold¹¹¹¹11Due to the way that reliability estimates are computed (see Section 4.3), all estimates are $>0.5$. and that Empirical Thresholding raises an alarm for the first positive prediction in an ongoing case. The costs for the first positive prediction approach are accordingly close to the ones of Empirical Thresholding for Traffic-RF.

We took great care to ensure we measure the right things. For example, we used accuracy metrics which are robust against class imbalances. We also made sure that our experimental results are not random. To this end, we repeated the cost measurements for approaches that include stochastic elements (i.e., Empirical Thresholding and Online RL) $10$ times. This number of repetitions provided a good balance between experimental effort and variance of results.

Still, the realism of our experiments and thus generalizability is limited. First, due to the limitations of the real-world data sets, we could only measure the effect of triggering at most one process adaptation per each ongoing case (like in  [76]). In reality, several adaptations may be possible during the execution of a case.

Second, we only considered a single type of alarm, i.e., we only prescribed whether to perform a proactive process adaptation or not. In practice, one may raise different kinds of alarms that trigger different types of adaptations [15].

Third, to quantify cost savings, we used a cost model from the BPM and ECTS literature. The benefit of this cost model is that it keeps the number of independent variables in the experiments manageable. Yet, this cost model represents an approximation of the cost situations which may be faced in practice [80]. For example, the adaptation costs may depend on the extent of the deviation, or they may depend on where in the process an adaptation is performed. Also, we modeled adaptation effectiveness to linearly decrease as the case unfolds. In reality, there may be different shapes; e.g., it may be that after a certain point adaptations are no longer possible [60].

Comprehensive overviews of predictive process monitoring approaches are given in [3, 4, 5, 37, 11, 6]. Here we discuss the approaches that explicitly consider the trade-off between prediction accuracy and earliness.

One group of works uses prediction earliness as a dependent variable in the analysis of prediction accuracy. This means one evaluates the prediction models separately for each prefix length. The prediction model is applied to a subset of prefixes of exactly the given length. The improvement of prediction accuracy as the prefix length increases provides an implicit notion of earliness [4]. As an example, Leontjeva et al. [45] exploit the data payload of process events to increase prediction earliness. Teinemaa et al. [4], and we in our earlier work [66] measured the accuracy of different prediction techniques for the different prediction points along process execution. Results presented in the aforementioned works clearly show the trade-off between prediction earliness and accuracy. However, it was left open how to resolve this trade-off and how to use the results to facilitate prescriptive process monitoring.

Another group of works uses reliability estimates to filter among more or less reliable predictions. This means one keeps on monitoring each case until the prediction model gives a prediction with sufficiently high reliability. Earliness is then measured as the average prefix length when such a prediction was made [4]. Maggi et al. [85] use decision tree learning for predictive process monitoring. They use class probabilities of decision trees and analyze how selecting predictions using class probabilities impacts average prediction accuracy and earliness. Di Francescomarino et al. [26] employ random forests for prediction. Similar to Maggi et al. they analyze how selecting predictions using class probabilities (of random forests) impacts average prediction accuracy. They observe that using class probabilities may improve average prediction accuracy, but at the expedient of loosing predictions that are below a given probability threshold. Yet, they do not analyze to what extent using these class probabilities may facilitate prescriptive business process monitoring. Teinemaa et al. investigate whether unstructured data may increase prediction earliness and accuracy [51]. Francescomarino et al. investigate in how far hyper-parameter optimization [26] and clustering [86] can improve earliness.

As explained in Section 1, a similar trade-off between accuracy and earliness is investigated for time series classification. Gupta et al. provide a comprehensive review of the literature on Early Classification of Time Series (ECTS).

One principal difference between ECTS and process prediction is the type of the underlying data. While time series data represent values of single or multiple variables at different points in time (such as temperatures or stock prices), process monitoring data represent sequences of process events. A process event is characterized by a timestamp, which indicates the occurrence of the event, such as the completion of a process activity. Like in time series, the events and thus timestamps form an ordered sequence. However, in contrast to time series events are not typically happening at equal-space time intervals [29]. In addition to a timestamp, an event includes an event type (uniquely identifying the process step) and may include additional attributes of the event [6, 51]. As such, there is an important semantic difference between a data point in a time series and a process event.

Recent work indicates how to generalize from ECTS to a more data-type-agnostic approach for early decision making using machine learning [31]. While data streams are mentioned (which could be considered somewhat similar to event log data in BPM), the specifics of ECTS for data streams are not elaborated. Our paper can be considered to provides such additional specifics.

Independent from these principal differences, approaches for ECTS and prescriptive process monitoring share many similarities. We introduced some of them already in Section 3, but add further ones following taxonomy provided in [29] below.

One particular sub-class are so-called non-myopic approaches [46, 87]. They work by estimating the best timestep $\tau*$ for a decision and then taking a decision when this timestep is reached. Non-myopic approaches do so by predicting the continuation of the time series and using this continuation to estimate $\tau*$. As discussed in [15], non-myopic approaches assume a-priori knowledge of the length of the sequence, which is not given in BPM. Thereby, these approaches come with the risk that the running process will end before $\tau*$ is reached.

As introduced in Section 1, existing research addresses two closely related, complementary aspects of prescriptive process monitoring. One aspect is concerned with answering the question ”how to intervene?”. Recent work along this dimension includes [76, 14, 16, 17, 18]. The other aspect is concerned with answering the question ”when to intervene?”, thereby providing the backbone for answering the first questions [76]. Our contribution focuses on the question of ”when”, which we thus discuss below.

Teinemaa et al. were among the first to introduce the concept of alarm-based prescriptive process monitoring [12]. They use class probabilities generated by random forests as reliability estimates to determine whether to raise an alarm to trigger a proactive adaptation. They use Empirical Thresholding to determine reliability thresholds above which alarms are raised. Follow-up work by Fahrenkrog-Petersen et al. extends this initial work in particular with the capability of raising multiple types of alarms [15]. The benefit of Empirical Thresholding pursued by these papers is that it ensures that the threshold is optimal for the specific training data used and the given cost model. Yet, as we have seen above, the threshold may not remain optimal over time due to concept drift.

Shoush and Dumas propose a prescriptive process monitoring approach that explicitly considers whether resources are available for performing an adaptation as well as whether it may be beneficial to delay the adaptation [76]. Their approach prioritizes the adaptations across a set of ongoing cases, thereby addressing the ”infinite capacity” assumption concerning available resources. As such it sketches an interesting path to enhance the approaches we analyzed in this paper. To overcome the problem that the alternative process outcome after an adaptation is not known in general (see Section 1), they use causal inference to predict the process outcome for a given adaptation as an additional input for their approach. Thereby, they provide an alternative solution to the problem, for which we applied artificial curiosity. In follow-up work, Shoush and Dumas introduce the use of conformal prediction to give confidence guarantees instead of only providing estimates of the prediction reliability [76]. While their approach is intrinsically explainable, it requires defining a suitable non-conformity measure and calibration (as we discussed in 4.3).

In our earlier work, we used reliability estimates computed from ensembles of multi-layer perceptrons to decide on proactive adaptation [9, 60]. If the reliability of a given prediction is equal to or greater than a predefined threshold, the prediction is used to trigger a proactive adaptation. In [9] we focused on ensembles of classification models. In [60] we extended this work to ensembles of regression models, thereby also including the extent of a predicted deviation in the adaptation decision. Yet, in this earlier work we used a static prediction point (the 50% mark of process execution), and thus did not consider the aspect of prediction earliness.

In [24, 21], we use reliability estimates computed from ensembles of LSTM models to dynamically decide on proactive adaptation. We determine during an ongoing case the earliest prediction with sufficiently high reliability and use this prediction to trigger a proactive adaptation. However, this previous approach required the explicit definition of a reliability threshold.

In the literature, RL approaches in the context of BPM were proposed for different main purposes. Huang et al. employ RL for the dynamic optimization of resource allocation in operational business processes [88]. However, they do not consider proactive adaptation of processes at run time. Also, they use Q-Learning as a classical RL algorithm, and thus assume the environment can be represented by a finite, discrete set of states. As mentioned above, Online RL does not have this limitation, as it can directly handle large and continuous environments, as well as can deal with the non-stationarity of these environments and thereby concept drifts affecting the machine learning models.

Satyal et al. propose solving the problem of deciding how new instances are assigned to a specific version of a process by modeling it as a multi-armed bandit problem [59]. A multi-armed bandit problem can be considered a simple variant of RL, which only takes one-shot decisions and not sequential decisions. They observe that reward engineering needs to be done carefully, and especially that the reward function must provide a strong enough signal for effective learning. We similarly have elaborated on the reward engineering concern for Online RL.

Silvander proposes using Q-Learning with function approximation via a deep neural network (DQN) for the optimization of business processes [89]. In contrast to policy-based RL used in Online RL, Q-Learning faces the exploration-exploitation dilemma [55, 28]. To optimize rewards, RL should exploit actions that have shown to be effective. However, to discover such actions in the first place, actions that were not selected before should be explored. One typical solution to the exploration-exploitation dilemma is the $\epsilon$-greedy mechanism. During learning, $\epsilon$-greedy chooses a random action with probability $\epsilon$. Resolving the dilemma means finding a balance between exploitation and exploration to facilitate the convergence of the learning process. To this end, Silvander suggests defining an $\epsilon$ decay rate, to reduce the amount of exploration over time. However, he does not consider using RL at run time, and thus does not take into account how to increase the rate of exploration in the presence of concept drifts.

Branchi et al. propose using RL for prescriptive business process monitoring [90]. As an RL algorithm they use policy iteration with Monte Carlo methods to determine the best next action to execute for optimizing process KPIs, such as revenue or costs. The chosen RL algorithm belongs to the class of model-based algorithms, which require an explicit model of the environment. The authors approximate such an environment model by mining it from event log data. As they focus on which action to execute next, they complement our work on when to perform an adaptation.

In our previous work, we proposed a generic framework and implementation for using policy-based RL for self-adaptive information systems [28]. In follow-up work, we customized this framework specifically for the problem of generating alarms [27]. In particular, we integrated this framework with our work on predictive process monitoring and presented initial promising results for typical BPM benchmark data sets [21]. Here, we relax the fundamental assumption about knowing the alternative process outcome after an adaptation (e.g., see discussion in Sections 1 and 4).

Recent work by Bozorgi [14] enhances our previous work on Online RL  [27] along three main directions: First, they use causal effect estimations to determine the effectiveness of an adaptation and only raise an alarm if an adaptation may indeed lead to an effective outcome. Second, they pre-train the RL agent in an offline setting using simulated data, thereby addressing the problem of low initial performance of online RL (see discussion in Section 9.2). Third, to further speed up convergence of the learning process, they introduce conformal prediction (see Section 10.3), such that the RL agent is able to avoid cases that most likely will end up in a successful outcome anyways.