On Non-asymptotic Theory of Recurrent Neural Networks in Temporal Point Processes

Temporal point process (TPP) (Daley et al., 2003; Daley and Vere-Jones, 2008) is an important mathematical framework that provides tools for analyzing and predicting the timing and patterns of events in continuous time. TPP particularly deals with event streaming data where the events occur at irregular time stamps, which is different from classical time series analysis that often assumes a regular time spacing between data points. In real world applications, the events could be anything from transactions in financial markets (Bauwens and Hautsch, 2009; Hawkes, 2018) to user activities in online social network platforms (Farajtabar et al., 2017; Fang et al., 2023), earthquakes in seismology (Wang et al., 2012; Laub et al., 2021), neural spikes in biological experiments (Perkel et al., 1967; Williams et al., 2020), or failure times in survival analysis (Aalen et al., 2008; Fleming and Harrington, 2013).

With advent of artificial intelligence in last decades, the neural network (McCulloch and Pitts, 1943) has been proved to be a powerful architecture that can be adapted to different applications with distinct purposes. In modern machine learning, researchers have also incorporated deep neural networks into TPPs to handle complex patterns and dependencies in event data, leading to advancements in many areas such as recommendation systems (Du et al., 2015; Hosseini et al., 2017), social network analysis (Du et al., 2016; Zhang et al., 2021), healthcare analytics (Li et al., 2018; Enguehard et al., 2020), etc. Many new TPP models have been proposed in the recent literature, including but not limited to, recurrent temporal point process Du et al. (2016), fully neural network TPP model Omi et al. (2019), transformer Hawkes process Zuo et al. (2020); see Shchur et al. (2021); Lin et al. (2022) and the references therein for a more comprehensive review.

Despite the recent process in TPP’s applications as mentioned above, there is a lack of understanding in neural TPPs from the theoretical perspective. A fundamental question remains: whether the neural network-based TPP can provably have a small generalization error? In this paper, we provide an affirmative answer to this question for recurrent neural network (RNN, Medsker and Jain (1999))-based TPPs. To be specific, we establish the non-asymptotic rates of generation error bounds under mild model assumptions and provide the construction of RNN architectures that could approximate many widely-used TPPs, including homogeneous Poisson process, non-homogeneous Poisson process, self-exciting process, etc.

There are a few challenges in developing the theory of RNN-based TPPs. (a) Characterization of functional space. In the machine learning theory, it is necessary to specify the model space to derive any generalization errors. In our setting, the thing becomes more complicated since the model should be data-dependent (i.e., adapts to the past events). Otherwise, the model could not capture the information in event history and fail to provide a good fitting. (b) Expressive power of RNN architecture. RNN is the most widely adopted neural architecture in TPP modelling. However, it remains questionable whether the RNNs can approximate most well-known temporal point processes. If the answer is yes, it would be of great interest to know how many hidden layers and how large hidden dimensions will be sufficient for the approximation. (c) Expressive power of activation function. In modern neural networks, the activation function is chosen to be a simple non-linear function for the sake of computational feasibility. In RNNs, it is taken as the “tanh" by default. Then it is important to understand the approximability of tanh activation functions. (d) Variable length of event sequence. Unlike the standard RNN’s modelling where each sample is assumed to have the same number of observations (events), the event sequences in our setting may vary from one to another. In addition, their lengths are potentially unbounded. These add difficulties in computing the complexity of the model space.

To overcome the above challenges, we adopt the following approaches. (a) In TPPs, the intensity function is the core. We recursively construct the multi-layer hidden cells through RNNs to store the event information and adopt the suitable output layer to compute the intensity value. Equipped with suitable input embeddings, our construction can capture the information of event history and adapt to variable lengths of event sequence. (b) For four main categories of TPPs, homogeneous Poisson process, non-homogeneous Poisson process, self-exciting process, and self-correcting process, we carefully study their intensity formula. We can decompose the intensity function into different parts and approximate them component-wisely. Our construction explicitly gives the upper bounds on the model depth, the width of hidden layers, and parameter weights of the RNN architecture to achieve a certain level of approximation accuracy. (c) We use the results in a recent work (De Ryck et al., 2021), where they provide the approximation ability of one- and two-layer $\tanh$ neural networks. We adapt such results to our specific RNN structure and give the universal approximation results for each of the intensity components. (d) Thanks to the exponential decay property of the tail probability of the sequence length, we are able to use the truncation technique to decouple the randomness of independent and identically distributed (i.i.d.) samples and the lengths of event sequences. For the space of truncated loss functions, the space complexity can be obtained through calculating the covering number. The classical chaining methods in empirical process theory can hence be applied as well.

Our main technical contributions can be summarized as follows.

(i) In the analysis of the stochastic error in the excess risk of RNN-based TPPs, we provide a truncation technique to decompose the randomness into a bounded component and a tail component. By carefully balancing between the two parts, we establish a nearly optimal stochastic error bound. Additionally, we also derive the complexity of the multi-layer RNN-based TPP class, where we precisely analyze and compute the Lipschitz constant of RNN architecture. This extends the existing result in Chen et al. (2020) where they only give the Lipschitz constant of a single-layer RNN. Therefore, our truncation technique and the Lipschitz result of multi-layer RNNs can be useful and of independent interest for many other related problems.

(ii) We establish the approximation error bounds for the intensity functions of TPPs of four main categories. To the best of our knowledge, there is very few work (De Ryck et al., 2021) on studying the approximation property of $\tanh$ activation function. Our work is the first one to provide approximation results for RNN-based statistical models. Our construction procedure largely depends on the Markov nature (Laub et al., 2021) of self-exciting processes so that we can design hidden cells to store sufficient information of past events. Moreover, we decompose the excitation function into different parts. Each of them is a simple smooth function (i.e. either exponential function or trigonometric function) that can be well approximated by a single-layer $\tanh$ network. Our construction method can be viewed as a useful tool in analyzing other sequential-type neural networks.

(iii) We illustrate the differences between the architectures of classical RNNs and RNN-based TPPs. Note the fact that the observed events happen at the discrete time grids, while the TPP models should take into account the continuous time domain. Therefore, the interpolation of values in hidden cells at each time point is important and necessary. We show that improper interpolation mechanisms (e.g. constant, linear, exponential decay interpolation) may fail to provide RNN-based TPP with the universal approximation ability. Our result indicates that the input embedding plays an important role in interpolating the hidden states.

The rest of paper is organized as follows. In Section 2, the background of TPPs, the formulation of RNN-based TPPs, and useful notations are introduced. The main theories along with high-level explanations are given in Section 3. The technical tools for analyzing stochastic errors are provided in Section 4. The construction procedures for approximating different types of intensity functions are listed in Section 5. In Section 6, we provide explanations that the improper interpolation of hidden states in RNN-TPPs may lead to unsatisfactory approximation results. The concluding remarks are given in Section 7.

Recent applications in event stream analyses have witnessed the usefulness of TPPs with the incorporation of RNNs. However, there is no study in the existing literature to explain why the RNN structure in TPP modeling is so useful from the theoretical perspective. We attempt to answer the question of whether the RNN-TPPs can provably have small generalization error or excess risk. Our answer is positive! When the event data are generated according to the classical models described in Section 2.3, we show that the RNN-TPPs can perfectly generalize such data.

To make our presentation easier, we only need to focus on the self-exciting processes. ¹¹1Homogeneous Poisson, non-homogeneous Poisson, and self-correcting process can be treated similarly due to the following reasons. If we take $\mu(t)\equiv 0$ in (9), the linear self-exciting process reduces to the homogeneous Poisson or non-homogeneous Poisson process. In the RNN-TPP architecture, we can take the input embedding function $x(t;S)=(t,t-F_{S}(t),N(t-))$, i.e., using an additional input dimension to store the number of past events. Then establishing the excess risk of self-correcting process is technically equivalent to that of non-homogeneous Poisson process. To start with, we first consider the linear case (9).

Some regularity assumptions should be stated before we present the main theorem.

(A1) There exists a constant $B_{0}>0$ such that $\lambda_{0}\in W^{s,\infty}([0,T],B_{0})$, where $s\geq 1$, $s\in\mathbb{N}$.

(A2) $\int_{0}^{T}\mu(t)\mathrm{d}t:=c_{\mu}<1$.

(A3) There exists a positive constant $B_{1}$ such that $\inf_{t\in[0,T]}\lambda_{0}(t)\geq B_{1}$.

Assumption (A1) assumes the boundedness of the background intensity, which is also common in neural network approximation studies. Assumption (A2) is standard in the literature of Hawkes process, which guarantees the existence of a stationary version of the process when $\lambda_{0}(t)$ is constant. Assumption (A3) is an informative lower bound assumption, which ensures that sufficient intensity exists in any subdomain of $[0,T]$.

Now we can present the results on the non-asymptotic bound of excess risk (4) under model (9).

As suggested in Theorem 1, there exists a two-layer RNN-TPP model whose excess risk becomes vanishing when the size of the training set goes to infinity. The width of such network grows with the sample size, while the depth remains two.

Now we consider the true model to be a non-linear Hawkes process, which is given in (10). For simplicity, we only consider the case $\mu(t)=\alpha\exp(-\beta t)$, which is

The regularity of $\Psi$ is presented as Assumption (A4).

(A4) Function $\Psi$ is $L$-Lipschitz, positive and bounded. In other words, there exist $\tilde{B_{1}},\tilde{B_{0}}>0$ such that $\tilde{B_{1}}\leq\Psi\leq\tilde{B_{0}}$ and $|\Psi(x_{1})-\Psi(x_{2})|\leq L|x_{1}-x_{2}|$ for any $x_{1},x_{2}$.

We have a similar bound of excess risk (4) under model (15).

For the non-linear case, as indicated by Theorem 2, we require a deeper RNN-TPP with four layers to achieve the vanishing excess risk. Under the Lipschitz assumption of $\Psi$, the width of the hidden layers is of order $n^{1/4}$. When $\Psi$ is allowed to have higher-order smoothness, the width can reduce to that of the vanilla Hawkes case.

By Lemma 1, the excess risk of ERM is bounded by the sum of two terms, the stochastic error $2\sup_{\lambda\in\mathcal{F}}|\mathbb{E}[\text{loss}(\lambda,S_{test})]-n^{-1}\sum_{i\in[n]}\text{loss}(\lambda,S_{i})|$ and the approximation error $\mathbb{E}[\text{loss}(\check{\lambda}^{\ast},S_{test})]-\mathbb{E}[\text{loss}(\lambda^{\ast},S_{test})]$. The first term can be bounded by the complexity of the function class $\mathcal{F}$ using the empirical process theory, where the unboundedness of the loss function needs to be handled carefully; we present the details in section 4. The second term characterizes the approximation ability of the RNN function class $\mathcal{F}$ to the true intensity $\lambda^{\ast}$ under the measure of the expectation of the negative log-likelihood loss function. In order to bound this term, we need to carefully construct a suitable RNN which can approximate $\lambda^{\ast}$ well. This has not been studied yet in the literature; see section 5 for the details.

Based on Lemma 1, the results in Theorem 1 admit the following form,

where $C(N)/\sqrt{n}$ is the stochastic error and $1/R(N)$ is the approximation error. $C(N)$ is the complexity of RNN function class $\mathcal{F}$ and $R(N)$ is the corresponding approximation rate, where $N$ is a tuning parameter. For the Poisson case, we can construct a two-layer RNN-TPP with $O(N)$ width to achieve $O(N^{-s})$ approximation error. Hence $C(N)=O(N)$, $R(N)=O\left(N^{s}\right)$, and the final excess risk bound is $\tilde{O}(n^{-\frac{s}{2(s+1)}})$ in (12). For the vanilla Hawkes case, since the exponential function is $C^{\infty}$-smooth, we only need extra $O(\text{Poly}(\log N))$ hidden cells in each layer to obtain $\tilde{O}(N^{-s})$ approximation error, and then we have the same order excess risk bound. For the general case, motivated by the vanilla Hawkes case, we decompose $\mu\in C^{k,\infty}([0,T],C_{0})$ into two parts. One part is a polynomial of exponential functions which can be well approximated by $O(\text{Poly}(\log N))$-width tanh neural network. The other part is a function $\tilde{\mu}\in C^{k,\infty}([0,T],\tilde{C_{0}})$ satisfying $\tilde{\mu}^{(j)}(0+)=\tilde{\mu}^{(j)}(T-)$, $j=0,1,\cdots,k-1$. It is easy to check that the $r$-th Fourier coefficients of $\tilde{\mu}$, $\hat{\mu}_{r}$, decay at the rate of $r^{-k}$. Then it is sufficient to approximate the first $N$ functions in the Fourier expansion of $\tilde{\mu}$ to get $\tilde{O}(N^{-(k-1)})$ approximation error, which additionally costs $\tilde{O}\left(N^{5}\right)$ complexity (see section 5.3 for details). Combining this with the approximation result of $\lambda_{0}$, we get the final bound $\tilde{O}(n^{-\frac{1}{2}\left(\frac{s}{s+1}\wedge\frac{k-1}{k+4}\right)})$. Similarly, for the nonlinear Hawkes case, we need $\tilde{O}(N)$ complexity to obtain $\tilde{O}(N^{-1})$ approximation error, which leads to $\tilde{O}({n}^{-\frac{1}{4}})$ excess risk bound.

As we emphasize in the above remarks, the number of layers depends on the smoothness of $\lambda_{0}$. If $\lambda_{0}\in C^{\infty}([0,T])$ and $\|\lambda_{0}\|_{W^{s,\infty}}\leq C^{s}$, we only need one-layer $\tanh$ neural network to approximate $\lambda_{0}$, hence the number of layers in RNN-TPP can be reduced to one.

In this section, we focus on the stochastic error in (17). This type of stochastic error for the RNN function class has been studied in the recent literature, such as Chen et al. (2020) and Tu et al. (2020). However, they only consider the case where the lengths of the input sequences are bounded, which is not applicable under the TPP setting. Here we establish an upper bound of the stochastic error in (17) by a novel decoupling technique to make the classical results applicable. This technique can be used in many other related problems.