Learning DTW Global Constraint for
Time Series Classification

Dynamic Time Warping (DTW) [5] distance is a well-known similarity measure based on shape. It uses a dynamic programming technique to find all possible warping paths, and selects the one with the minimum distance between two time series. To calculate the distance, it first creates a distance matrix, where each element in the matrix is a cumulative distance of the minimum of three surrounding neighbors. Suppose we have two time series, a sequence $Q$ of length $n$ ($Q=q_{1},q_{2},\ldots,q_{i},\ldots,q_{n}$) and a sequence $C$ of length $m$ ($C=c_{1},c_{2},\ldots,c_{j},\ldots,c_{m}$). First, we create an $n$-by-$m$ matrix, where every ($i,j$) element of the matrix is the cumulative distance of the distance at ($i,j$) and the minimum of three neighboring elements, where $1\leq i\leq n$ and $1\leq j\leq m$. We can define the ($i,j$) element, $\gamma_{i,j}$, of the matrix as:

In reality, DTW may not give the best mapping according to our need because it will try its best to find the minimum distance. It may generate the unwanted path. For example, in Figure 1 [5], without global constraint, DTW will find its optimal mapping between the two time series. However, in many cases, this is probably not what we intend, when the two time series are expected to be of different classes. We can resolve this problem by limiting the permissible warping paths using a global constraint. Two well-known global constraints, Sakoe-Chiba band and Itakura Parallelogram [1], and a recent representation, Ratanamahatana-Keogh band (R-K band), have been proposed, Figure 2 [4] shows an example for each type of the constraints.

Sakoe-Chiba band (S-C band), shown in Figure 2 (b), is one of the simplest and most popular global constraints, originally introduced to be used for speech community. The width of this global constraint is generally set to be 10% of the time series length. However, recent work [5] has shown that the different sizes of the band can be used towards a more accurate classification. We therefore need to test all possible widths of the global constraint so that the best width could be discovered. An evaluation function is needed to justify the selection. We commonly use accuracy metric (a training accuracy) as a measurement. Table 1 shows an algorithm in finding the best warping window for S-C band by decreasing the band size by 1% in each step. This function receives a set of data $T$ as an input, and gives the best warping window (best_band) as an output. Note that if an evaluation value is equal to the best evaluation value, we prefer the smaller warping window size.

Ratanamahatana-Keogh band (R-K band) [4] is a general model of a global constraint specified by a one-dimensional array $R$, i.e., $R=r_{1},r_{2},\ldots,r_{i},\ldots,r_{n}$ where $n$ is the length of time series, and $r_{i}$ is the height above the diagonal in $y$ direction and the width to the right of the diagonal in $x$ direction. Each $r_{i}$ value is arbitrary, therefore R-K band is also an arbitrary-shape global constraint, as shown in Figure 2 (a). Note that when $r_{i}=0$, where $1\leq i\leq n$, this R-K band represents the Euclidean distance, and when $r_{i}=n$, where $1\leq i\leq n$, this R-K band represents the original DTW distance with no global constraint. The R-K band is also able to represent the S-C band by giving all $r_{i}=c$, where $c$ is the width of a global constraint. Moreover, the R-K band is a multi band model which can be effectively used to represent one band for each class of data. This flexibility is a great advantage; however, the higher the number of classes, the higher the time complexity, as we have to search through such a large space.

Since determining the optimal R-K band for each training set is highly computationally intensive, a hill climbing and heuristic functions have been introduced to guide which part of space should be evaluated. A space is defined as a segment of a band to be increased or decreased. In the original work, two heuristic functions, accuracy metric and distance metric, are used to evaluate a state. The accuracy metric is evaluated from the training accuracy using leaving-one-out 1-NN, and the distance metric is a ratio of the mean DTW distances of correctly classified and incorrectly classified objects. However, these heuristic functions do not reflect the true quality of a band because empirically, we have found that the resulting bands tend to ‘overfit’ the training data.

Two searching directions are considered, i.e., forward search, and backward search. In forward search, we start from the Euclidean distance (all $r_{i}$ in $R$ equal to 0), and parts of the band are gradually increased in each searching step. In the case where two bands have the same heuristic value, the wider band is selected. On the other hand, in backward search, we start from a very large band (all $r_{i}$ in $R$ equal to $n$, where $n$ is the length of time series), and parts of the band are gradually decreased in each searching step. If two bands have the same heuristic value, the tighter band is chosen.

Our learning algorithm starts from first enqueuing the starting- and ending-parts of the R-K Band. In each iteration, these values are dequeued, and used as a boundary for a band increase/decrease. And then the adjusted band is evaluated. If the heuristic value is higher than the current best heuristic value, the same start and end values are enqueued. If not, this part is further divided into two equal subparts before being enqueued, as shown in Figure 3 [4]. The iterations are continued until a termination condition is met. Table 2 shows the pseudo code for this multiple R-K bands learning.

Silhouette index (SI) [7] or Silhouette width is a well-known clustering validity technique, originally used to determine a number of clusters in a dataset. This index measures the ‘quality’ of separation and compactness of a clustered dataset, so the number of cluster is determined by selecting the number that gives maximum index value.

The Silhouette index is based on a compactness-separation measurement which consists of an inter-cluster distance (a distance between two different-cluster data) and an intra-cluster distance (a distance between two same-cluster data). A good clustered dataset means that the dataset has high inter-cluster distance and low intra-cluster distance. In other words, a good clustered dataset is the dataset that different-cluster are well separated, and the same-cluster data are well grouped together. The Silhouette index for each data $i$ is defined by the following equation:

Since the classification prediction time may because a major constraint, a data preprocessing step is needed. In this step, we approximate the calculation complexity and try to reduce the complexity exceeding the threshold. Our approximated complexity is mainly based on the number of items in the training data, its length, and the number of heuristic function evaluations. Suppose we have $n$ training data with $m$ data points in length, we can calculate the complexity by the following equation.

To decrease the complexity, we could reduce the length of each individual time series by using typical interpolation function. The new length of time series is set to be the current length divided by two. We keep reducing the time series length until the complexity is smaller than the user’s defined threshold. Table 5 shows the preprocessing steps on a set of training data $T$, a set of unlabeled data $P$, the original length $L$, and the complexity threshold. In this work, we set this threshold value to 9, according to resources and the time constraint for this 24-hour Workshop and Challenge on Time Series Classification.

Since the multi R-K band model allows a learning algorithm to create a different band for each different class. This boundary band extraction algorithm is derived from a simple intuition for each of the same-class data, we first calculate all their DTW distances, save all the warping paths, and plot those paths on a matrix (called a path matrix). After that, we will determine an appropriate R-K band. For each $r_{i}$ of this R-K band, the $r_{i}$ value is set to be the maximum between height above the diagonal in $y$ direction and width to the right of the diagonal in $x$ direction in the path matrix. We repeat these steps to every possible class in the dataset; we call this R-K band a $MaxBand$. Similarly, the band extraction is performed using mean average and mode instead of the maximum, resulting in a $MeanBand$ and a $ModeBand$, respectively. Figure 4 illustrates the steps in creating a $MaxBand$. From these calculations, three multiple R-K bands are generated. The evaluation function is used to select the best band to be returned as an output of this algorithm. Table 6 shows the band boundary extraction algorithm on a set of training data $T$ and return the best R-K band and the best heuristic value.

In this step, we try to achieve the best warping window of Sakoe-Chiba band to be an input of our proposed iterative R-K band learning. This function is slightly different from the original one in that we bound the maximum width of the warping window and we use our evaluation function (heuristic function) instead of the typical training accuracy. A simple pseudo code is described in Table 7 below. A set of training data $T$ and a maximum warping window size are required in discovering the best warping window $R$.

The iterative R-K band learning is extended from the original learning that it will repeat the learning again and again until a heuristic value no longer increases. In the first step, we initialize all the multi R-K bands with $R$% Sakoe-Chiba band, where $R$ is the output from the best finding warping window algorithm. We also set a learning threshold to be half of the time series length, and the initial bands are evaluated

In each iteration, our proposed algorithm learns a new R-K band starting with the previous R-K band learning result both in forward and backward direction. We also run both forward and backward learning and select the best band which gives a higher heuristic value. If the heuristic value is the same as the best heuristic value, the threshold is divided by two; otherwise we update the best heuristic value. We repeat these steps until the threshold falls below 1. Table 8 shows our proposed algorithm, iterative R-K band learning, which requires a set of training data $T$, a best warping window $R$, the length of time series $L$, and the bound of warping window.

We have modified the original multi R-K bands learning by changing its data structure. We replace multi queues, which are originally assigned for each class by only one single queue with an addition parameter label to each start-end object. This new queue will draw an object randomly instead of last-in-first-out (LIFO) manner. In addition, we also change an adjustment function by adding a new parameter bound to limit forward learning not to increase the band’s size exceeding limited bound. Table 9 shows the proposed R-K band learning on a set of training data $T$, a learning threshold, an initial band, and the bound of warping window.

From Section 2.4, we have briefly described the utility and the algorithm of the Silhouette index. This index is commonly used to measure the quality of a clustered dataset; however, we can utilize this Silhouette index as a heuristic function to measure the quality of a distance measure as well. The DTW distance with multi R-K bands is a distance measure that requires one additional parameter, $Band$, specifying the R-K band to be used (since the multi R-K bands contain one band for each class). Table 10 shows the evaluation function derived from the original Silhouette index.

After the best multi R-K bands are discovered, we use 1-Nearest Neighbor as a classifier and the Dynamic Time Warping distance measure with these best R-K bands for prediction in the test data to predict a set of unlabeled data. The LB_Keogh lower bound is also used to speed up the DTW computation.

We use the datasets from the Workshop and Challenge on Time Series Classification, held in conjunction with the thirteenth SIGKDD 2007 conference. The datasets are from very diverse domains (e.g., stock data, medical data, etc.); some are from real-world problems, and some are synthetically generated. The amount of training data and its length in each dataset also vary from the size of 20 to 1000 training instances and the length of 30 to 2000 data points. In addition, all data are individually normalized using Z-normalization. Examples of each dataset are shown in Figure 5, and the datasets’ properties are shown in Table 11.

The predicted result is generated after running our algorithm to find the best R-K band within the competition’s 24-hour time constraint. More specifically, the predicted accuracy is calculated by computing leaving-one-out cross validation on the training dataset. Table 12 shows our predicted accuracies and testing accuracies for all 20 datasets which are calculated and are returned to the contest organizer. Because of the small number of training data, the predicted accuracy and the test accuracy are different in some cases.