XNAP: Making LSTM-based Next Activity Predictions Explainable by Using LRP

LRP is a technique to explain predictions of DNNs in terms of input variables [3]. For a given input sequence $\sigma=\langle\mathbf{x}^{(1)},\mathbf{x}^{(2)},\mathbf{x}^{(3)}\rangle$, a trained DNN model $\mathcal{M}_{c}$ and a calculated prediction $\mathbf{o}=\mathcal{M}_{c}(\sigma)$, LRP reverse-propagates the prediction $\mathbf{o}$ through the DNN model $\mathcal{M}_{c}$ to assign a relevance value to each input variable of $\sigma$ [1]. A relevance value indicates to which extent an input variable contributes to the prediction. Note $\mathcal{M}_{c}$ is a DNN model, and $c$ is a target class for which we want to perform LRP. In this paper, $\mathcal{M}_{c}$ is an LSTM model, i.e. a DNN model with an LSTM [11] layer as a hidden layer. The architecture of the “vanilla” LSTM (layer) is common in the PBPM literature for the task of predicting next activities [25]. For instance, an explanation of it can be found in the work of Evermann et al. [9].

To calculate the relevance values of the input variables, LRP performs two computational steps. First, it sets the relevance of an output layer neuron corresponding to the target class of interest $c$ to the value $\mathbf{o}=\mathcal{M}_{c}(\sigma)$. It ignores the other output layer neurons and equivalently sets their relevance to zero. Second, it computes a relevance value for each intermediate lower-layer neuron depending on the neural connection type. A DNN’s layer can be described by one or more neural connections. In turns, the LRP procedure can be described layer-by-layer for different types of layers included in a DNN. Depending on the type of a neural connection, LRP defines heuristic propagation rules for attributing the relevance to lower-layer neurons given the relevance values of the upper-layer neurons [3].

In case of recurrent neural network layers, such as LSTM [11] layers, there are two types of neural connections: many-to-one weighted linear connections, and two-to-one multiplicative interactions [2]. Therefore, we restrict the definition of the LRP procedure to these types of connections. For weighted connections, let $\mathbf{z}_{j}$ be an upper-layer neuron. Its value in the forward pass is computed as $\mathbf{z}_{j}=\sum_{i}\mathbf{z}_{i}\cdot\mathbf{w}_{ij}+b_{j}$, while $\mathbf{z}_{i}$ are the lower-layer neurons, and $\mathbf{w}_{ij}$ as well as $\mathbf{b}_{j}$ are the connection weights and biases. Given each relevance $\mathbf{R}_{j}$ of the upper-layer neurons $\mathbf{z}_{j}$, LRP computes the relevance $\mathbf{R}_{i}$ of the lower-layer neurons $\mathbf{z}_{i}$. Initially, $\mathbf{R}_{j}=\mathcal{M}_{c}(\sigma)$ is set. The relevance distribution onto lower-layer neurons comprises two steps. First, by computing relevance messages $\mathbf{R}_{i\leftarrow j}$ going from upper-layer neurons $\mathbf{z}_{j}$ to lower-layer neurons $\mathbf{z}_{i}$. The messages $\mathbf{R}_{i\leftarrow j}$ are computed as a fraction of the relevance $\mathbf{R}_{j}$ accordingly to the following rule:

$N$ is the total number of lower-layer neurons connected to $\mathbf{z}_{j}$, $\epsilon$ is a stabiliser (small positive number, e.g. $0.001$) and $sign(\mathbf{z}_{j})=(1_{\mathbf{z}_{j}\geq 0}-1_{\mathbf{z}_{j}<0})$ is the sign of $\mathbf{z}_{j}$. Second, by summing up incoming messages for each lower-layer neuron $\mathbf{z}_{i}$ to obtain relevance $\mathbf{R}_{i}$. $\mathbf{R}_{i}$ is computed as $\sum_{j}\mathbf{R}_{i\leftarrow j}$. If the multiplicative factor $\delta$ is set to 1.0, the total relevance of all neurons in the same layer is conserved. If it is set to 0.0, the total relevance is absorbed by the biases.

For two-to-one multiplicative interactions between lower-layer neurons, let $\mathbf{z}_{j}$ be an upper-layer neuron. Its value in the forward pass is computed as the multiplication of two lower-layer neuron values $\mathbf{z}_{g}$ and $\mathbf{z}_{s}$, i.e. $\mathbf{z}_{j}=\mathbf{z}_{g}\cdot\mathbf{z}_{s}$. In such multiplicative interactions, there is always one of two lower-layer neurons that represents a gate with a value range $[0,1]$ as the output of a sigmoid activation function. This neuron is called gate $\mathbf{z}_{g}$, whereas the remaining one is the source $\mathbf{z}_{s}$. Given such a configuration, and denoting by $\mathbf{R}_{j}$ the relevance of the upper-layer neuron $\mathbf{z}_{j}$, the relevance can be redistributed onto lower-layer neurons by: $\mathbf{R}_{g}=0$ and $\mathbf{R}_{s}=\mathbf{R}_{j}$. With this reallocation rule, the gate neuron already decides in the forward pass how much of the information contained in the source neuron should be retained to make the overall classification decision.

The offline component receives as input an event log, pre-processes it, and outputs a Bi-LSTM model learned based on the pre-processed event log.

Pre-processing: The offline component’s pre-processing step transforms an event log $L$ into a data tensor $\mathsf{X}$ and a label matrix $\mathbf{Y}$ (i.e. next activities). The procedure comprises four steps. First, we transform an event log $L$ into a matrix $\mathbf{S}\in\mathbb{R}^{E\times U}$. $E$ is the event log’s size $|L|$, whereas $U$ is the number of an event tuple’s elements. Note that we add an activity to the end of each sequence to predict their end. Second, we onehot-encode the string values of the activity attribute in $\mathbf{S}$ because a Bi-LSTM requires a numerical input for calculating forward and backward propagations. After this step, we get the matrix $\mathbf{S}\in\mathbb{R}^{E\times H}$, where $H$ is the number of different activity values in the event log $L$. Third, we create prefixes and next activity labels. Thereby, a tuple of prefixes $R$ is created from $\mathbf{S}\in\mathbb{R}^{E\times H}$ by applying the function $f_{p}$, whereas a tuple of labels $K$ is created from $\mathbf{S}\in\mathbb{R}^{E\times H}$ through the function $f_{l}$. Lastly, we construct a third-order data tensor $\mathsf{X}\in\mathbb{R}^{|R|\times M\times H}$ based on the prefix tuple $R$ as well as a label matrix $\mathbf{Y}\in\mathbb{R}^{|K|\times H}$ based on the label tuple $K$, where $M$ is the longest trace in the event log $L$, i.e. $|max_{\sigma}(L)|$. The remaining space for a sequence $\sigma_{c}\in\mathsf{X}$ is padded with zeros, if $|\sigma_{c}|<|max_{\sigma}(L)|$.

Model learning: XNAP learns a Bi-LSTM model $\mathcal{M}$ that maps the prefixes onto the next activity labels based on the data tensor $\mathsf{X}$ and label matrix $\mathbf{Y}$ from the previous step. We use the Bi-LSTM architecture, an extension of “vanilla” LSTMs since Bi-LSTMs are forward and backward LSTMs that can exploit control-flow information from two directions of sequences. XNAP’s Bi-LSTM architecture comprises an input layer, a hidden layer, and an output layer. The input layer receives the data tensor and transfers it to the hidden layer. The hidden layer is a Bi-LSTM layer with a dimensionality of 100, i.e. the Bi-LSTM’s cell internal elements have a size of 100. We assign the activation function $tanh$ to the Bi-LSTM’s cell output. To prevent overfitting, we perform a random dropout of $20\%$ of input units along with their connections. The model connects the Bi-LSTM’s cell output to the neurons of a dense output layer. Its number of neurons corresponds to the number of the next activity classes. For learning weights and biases of the Bi-LSTM architecture, we apply the Nadam optimisation algorithm with a categorical cross-entropy loss and default values for parameters. Note that the loss is calculated based on the Bi-LSTM’s prediction and the next activity ground truth label stored in the label matrix $\mathbf{Y}$. Additionally, we set the batch size to $128$. Following Keskar et al. [12], gradients are updated after each 128^th trace of the data tensor $\mathsf{X}$. Larger batch sizes tend to sharp minima and impair generalisation. The number of epochs (learning iterations) is set to $100$, to ensure convergence of the loss function.

The online component receives as input a running trace, performs a pre-processing, creates a next activity prediction and concludes with the creation of a relevance value for each activity of the running trace regarding the prediction. The prediction is obtained by using the learned Bi-LSTM model from the offline component. Given the prediction, LRP determines the activity relevances by backwards passing the learned Bi-LSTM model.

Pre-processing: The online component’s pre-processing step transforms a running trace $\sigma_{r}$ into a data tensor and a label matrix, as already described in the offline component’s pre-processing step. Note that we terminate the online phase if $|\sigma_{r}|$ is $\leq 1$ since, for such traces, there is insufficient data to base prediction and relevance creation upon. Further, we assume that we have already observed all possible activities as well as the longest trace in the offline component. Thus, matrix $\mathbf{S}$ and tensor $\mathsf{X}$ lay in $\mathbb{R}^{1\times H}$ and $\mathbb{R}^{1\times M\times H}$. In the offline component, next activity labels are not known and based on the data tensor $\mathsf{X}_{r}$ for a running trace $\sigma_{r}$ a next activity is predicted.

Prediction creation: Given the data tensor $\mathsf{X}_{r}$ from the previous step, the trained Bi-LSTM model $\mathcal{M}$ from the offline component returns a probability distribution $\mathbf{p}^{1\times H}$, containing the probability values of all activities. We retrieve the prediction $p$ from $\mathbf{p}$ through $argmax(\mathbf{p}[j])$, with $1\leq j\leq H$.

Relevance creation: Lastly, we provide explainability of the prediction $p$ by applying LRP. For a next activity prediction $p$, LRP determines a relevance value for each activity in the course of a running trace $\sigma_{r}$ towards it by decomposing the prediction, from the output layer to the input layer, backwards through the model. Note the prediction $p$ was created in the previous step based on all activities of the running trace $\sigma_{r}$. In doing that, we apply the LRP approach proposed by Arras et al. [2] that is designed for LSTMs. As mentioned in Sec. 2, a layer of a DNN can be described by one or more neural connections. Depending on the layer’s type, LRP defines rules for attributing the relevance to lower-layer neurons given the relevance values of the upper-layer neurons. After backwards passing the model by considering conversation rules of different layers, LRP returns a relevance value for each onehot-encoded input activity of the data tensor $\mathsf{X}$. Finally, to visualise the relevance values, e.g. by a heatmap, positive relevance values are rescaled to the range $[0.5,1.0]$ and negative ones to the range $[0.0,0.5]$.

We demonstrate the benefit of XNAP with two real-life event logs that are detailed in Table 1.

First, we use the helpdesk event log³³3https://data.mendeley.com/datasets/39bp3vv62t/1.. It contains data of a ticketing management process form a software company. Second, we make use of the bpi2019 event log⁴⁴4https://data.4tu.nl/repository/uuid:a7ce5c55-03a7-4583-b855-98b86e1a2b07.. It was provided by a coatings and paint company and depicts an order handling process. Here, we only consider sequences of max. $250$ events and extract a 10%-sample of the remaining sequences to lower computation effort.

LRP is a model-specific method that requires a trained model for calculating activity relevances to explain predictions. Therefore, we report the predictive quality of the trained models, and then demonstrate the activity relevances.

Predictive quality: To improve model generalisation, we randomly shuffle the traces of each event log. For that, we perform a process-instance-based sampling to consider process-instance-affiliation of event log entries. This is important since LSTMs map sequences depending on the temporal order of their elements. Afterwards, for each event log, we perform a ten-fold cross-validation. Thereby, in every iteration, an event log’s traces are split alternately into a 90%-training and 10%-test set. Additionally, we use 10% of the training set as a validation set. While we train the models with the remaining training set, we use the validation set to avoid overfitting by applying early stopping after ten epochs. Consequently, the model with the lowest validation loss is selected for testing. To measure predictive quality, we calculate the average weighted Accuracy (overall correctness of a model) and average weighted F1-Score (harmonic mean of Precision and Recall).

Explainability: To demonstrate the explainability of XNAP’s LRP, we pick the Bi-LSTM model with the highest F1-Score value and randomly select two traces from all traces of each event log. One of these traces has a size of five; the other one has a size of eight. We use traces of different sizes to investigate our approach’s robustness.

Technical details: We conducted all experiments on a workstation with 12 CPU cores, 128 GB RAM and a single GPU NVIDIA Quadro RXT6000. We implemented the experiments in Python 3.7 with the DL library Keras⁵⁵5https://keras.io. 2.2.4 and the TensorFlow⁶⁶6https://www.tensorflow.org. 1.14.1 backend. The source code can be found on Github⁷⁷7https://github.com/fau-is/xnap..

The Bi-LSTM model of XNAP predicts the next most likely activities for the helpdesk event log with an average (Avg) Accuracy and F1-Score of 84% and 79.8% (cf. Table 2). For the bpi2019 event log, the model achieves an Avg Accuracy and F1-Score of 75.5% and 72.7%. For each event log, the standard deviation (SD) of the Accuracy and F1-Score values is between 1.0% and 1.5%.

We show the activity relevance values of XNAP’s LRP on the example of two traces per event log (cf. Fig. 3). The time steps (columns) represent the activities that are used as input. For each trace, we predict the next activity for different prefix lengths (rows). We start with a minimum of three and make one next activity prediction until the maximum length of the trace is reached (five and eight in our examples). The data-given ground truth is listed in the last column. We use a heatmap to indicate the relevance of the input activities to the prediction of the same row.

For example, in the traces (a) and (b), the activity “Resolve ticket” (C) has a high relevance on predicting the next activity “End (E)”. With that, a process analyst knows that the trace will end since the ticket was resolved. Another example is in the traces (c) and (d), where the activity “Record Invoice Receipt (C)” has a high relevance on predicting the next activity “Clear Invoice (D)”. Thus, a process analyst knows that the invoice can be cleared in the next step because the invoice receipt was recorded.