Interpretable Transformer Hawkes Processes: Unveiling Complex Interactions in Social Networks

The multivariate Hawkes process (Hawkes, 1971) is a widely used temporal point process model for capturing interactions among multiple event types. The key feature of the multivariate Hawkes process lies in its conditional intensity function. The conditional intensity function $\lambda_{k}(t|\mathcal{H}_{t})$ for event type $k$ at time $t$ is defined as the instantaneous event rate conditioned on the historical information $\mathcal{H}_{t}=\{(t_{i},k_{i})|t_{i}<t\}$:

where $\mu_{k}$ is the base rate for event type $k$ and $\phi_{k,k_{i}}(t-t_{i})$ is the trigger kernel representing the excitation effect from event $t_{i}$ with type $k_{i}$ to $t$ with type $k$. It expresses the expected number of occurrences of event type $k$ at time $t$ given the past history of events.

The interpretability of Hawkes processes stems from its explicit representation of event dependencies through the trigger kernel. The model allows us to quantify the impact of past events with different event types on the occurrence of a specific event, providing insights into the interactions between event types. As a result, the multivariate Hawkes process serves as a powerful tool in social network applications where understanding the interactions between event types (users or groups) is of utmost importance.

In contrast to the original Hawkes process, which assumes only non-negative trigger kernels (excitatory interactions) between events to avoid generating negative intensities, the nonlinear Hawkes process (Brémaud and Massoulié, 1996) offers a more flexible modeling framework by incorporating both excitatory and inhibitory effects among events. In the nonlinear Hawkes process, the conditional intensity function for event type $k$ at time $t$ is defined as:

where $\sigma(\cdot)$ is a nonlinear mapping from $\mathbb{R}$ to $\mathbb{R}^{+}$, ensuring the non-negativity of the intensity. Hence this trigger kernel can be positive (excitatory) or negative (inhibitory), thus enabling the modeling of complex interactions between different event types.

In the aforementioned models, the trigger kernel depends solely on the relative time $t-t_{i}$, implying that the trigger kernel is shift-invariant. However, in dynamic Hawkes process models (Zhou et al., 2020b, a; Bhaduri et al., 2021; Zhou et al., 2022), the trigger kernel is further extended to vary with absolute time, denoted as $\phi(t-t_{i},t_{i})$. By incorporating the absolute time, the trigger kernel becomes capable of capturing time-varying patterns, offering the model more degrees of freedom in its representation.

Our work is built upon THP (Zuo et al., 2020), so we concisely introduce the framework of THP here. Given a sequence $\mathcal{S}=\{(t_{i},k_{i})\}_{i=1}^{L}$ where each event is characterized by a timestamp $t_{i}$ and an event type $k_{i}$, THP leverages two types of embeddings, namely temporal embedding and event type embedding, to represent these two kinds of information. To encode event timestamps, THP represents each timestamp $t_{i}$ using an embedding vector $\mathbf{z}(t_{i})\in\mathbb{R}^{M}$:

where $z_{j}(t_{i})$ is the $j$-th entry of $\mathbf{z}(t_{i})$ and $j=0,\ldots,M-1$. The collection of time embeddings is represented as $\mathbf{Z}=\left[\mathbf{z}(t_{1}),\ldots,\mathbf{z}(t_{L})\right]^{\top}\in\mathbb{R}^{L\times M}$. For encoding event types, the model utilizes a learnable matrix $\mathbf{U}\in\mathbb{R}^{M\times K}$, where $K$ is the number of event types. For each event type $k_{i}$, its embedding $\mathbf{e}(k_{i})$ is computed as:

where $\mathbf{y}_{i}$ is the one-hot encoding of the event type $k_{i}$. The collection of type embeddings is $\mathbf{E}=[\mathbf{e}(k_{1}),\ldots,\mathbf{e}(k_{L})]^{\top}\in\mathbb{R}^{L\times M}$. The final embedding is the summation of the temporal and event type embeddings:

where each row of $\mathbf{X}$ represents the complete embedding of a single event in the sequence $\mathcal{S}$.

After embedding, the model focuses on learning the dependence among events using self-attention mechanism. The attention output $\mathbf{S}$ is computed as:

where $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ are the query, key and value matrices. Matrices $\mathbf{W}^{Q}\in\mathbb{R}^{M\times M_{K}}$, $\mathbf{W}^{K}\in\mathbb{R}^{M\times M_{K}}$ and $\mathbf{W}^{V}\in\mathbb{R}^{M\times M_{V}}$ are the learnable parameters. To preserve causality and prevent future events from influencing past events, we mask out the entries in the upper triangular region of $\mathbf{QK}^{\top}$.

Finally, the attention output $\mathbf{S}$ is passed through a two-layer MLP to produce the hidden state $\mathbf{H}$:

where $\mathbf{W}_{1}\in\mathbb{R}^{M\times M_{H}}$, $\mathbf{W}_{2}\in\mathbb{R}^{M_{H}\times M}$, $\mathbf{b}_{1}\in\mathbb{R}^{M_{H}}$ and $\mathbf{b}_{2}\in\mathbb{R}^{M}$ are the learnable parameters. The $k$-type conditional intensity function of THP is designed as:

where $t_{i}$ is the last event before $t$, $\alpha_{k},\mathbf{w}_{k},b_{k}$ are learnable parameters, the nonlinear function is chosen to be softplus to ensure that the intensity is non-negative, $\mathbf{h}(t_{i})$ is the transpose of the $i$-th row of $\mathbf{H}$ expressing the historical impact on event $t_{i}$.

In ITHP, we maintain the same temporal embedding and event type embedding methods as in THP. However, our modification lies in replacing the summation operation in Eq. 1 with concatenation:

The reason for this modification is that the original summation operation introduces a similarity between timestamps and event types (or vice versa) of the preceding and succeeding events. However, in statistical Hawkes processes, known for their interpretability, the interaction between two events is the magnitude of a kernel determined by the similarity (correlation) between their types and the similarity (distance) between their timestamps. No cross-similarity is introduced. To maintain a similar level of interpretability, we replace summation with concatenation here.

In ITHP, we still use self-attention to capture the influences of historical events on subsequent events. However, unlike THP, in the modified attention module, we use distinct query and key matrices:

where the $i$-th row of $\mathbf{S}$, $\mathbf{S}_{i}$, represents the historical influence on the $i$-th event. The calculation of $\mathbf{S}_{i}$ can be explicitly expressed as the summation over all events preceding event $i$, where the attention weights are normalized by the softmax:

The reason for this modification is that after removing $\mathbf{W}^{Q}$ and $\mathbf{W}^{K}$, the attention weights can be simply represented as $\mathbf{XX}^{\top}$. Compared to the original $\mathbf{QK}^{\top}$, $\mathbf{XX}^{\top}$ has a clearer physical meaning. In $\mathbf{XX}^{\top}$, the entry in the $i$-th row and $j$-th column can be expressed as a shift-invariant function $g_{k_{i},k_{j}}(t_{i}-t_{j})$. This representation allows for a more meaningful interpretation of the relationship between events. In contrast, $\mathbf{QK}^{\top}$ does not achieve this clarity.

The form of Eq. 6 naturally reminds us of the trigger kernel summation in statistical Hawkes processes. The only difference is that $\mathbf{g}$ in Eq. 6 is a vector, whereas the trigger kernel in statistical Hawkes processes is a scalar function. Taking inspiration from this, we propose a more interpretable conditional intensity function:

where $\mathbf{w}_{k}$ is a learnable parameter used to aggregate the vector $\mathbf{g}$ into a scalar value and $b_{k}$ is a learnable bias term. The newly designed conditional intensity aligns perfectly with the nonlinear Hawkes processes with a time-varying trigger kernel. The green term $b_{k}$ in Eq. 7 corresponds to the base rate $\mu_{k}$ in Section 3.2, the yellow term $\mathbf{g}^{\top}_{k,k_{i}}(t-t_{i},t_{i})\mathbf{w}_{k}$ in Eq. 7 corresponds to the time-varying trigger kernel $\phi_{k,k_{i}}(t-t_{i},t_{i})$ in Section 3.2, and the softplus function serves as a non-linear mapping ensuring the non-negativity of the intensity. This leads to improved interpretability as the trigger kernel can be explicitly expressed in our design, in contrast to the original THP.

In point process model training with maximum likelihood estimation (MLE), it is vital to compute the intensity integral over the entire time domain, which requires modeling the intensity both at event positions and on non-event intervals. In the RNN-based deep point process models (Du et al., 2016; Mei and Eisner, 2017), due to the limitations of the RNN framework in solely modeling latent representations at event positions, the aforementioned works adopted parameterized extrapolation methods to model the intensity on non-event intervals, see Eq. (11) in (Du et al., 2016) and Eq. (7) in (Mei and Eisner, 2017). THP (Zuo et al., 2020) also adopted the same approach to model the intensity on non-event intervals (the red term in Eq. 2). However, we emphasize that attention-based deep point process models do not necessarily require the parameterized extrapolation methods to model the intensity on non-event intervals. Our design Eq. 7 employs the attention mechanism to model the intensity function whether it is at event positions or not. Therefore, we refer to it as a “fully attention-based intensity function”. The fully attention-based intensity function circumvents the limitations of parameterization and ensures that the model can effectively capture intricate intensity patterns at non-event positions, thus enhancing the model’s expressive power.

For a given sequence $\mathcal{S}=\{(t_{i},k_{i})\}_{i=1}^{L}$ on $[0,T]$, the point process model training can be performed by the MLE approach. The log-likelihood of a point process is expressed in the following form:

where $\lambda(t|\mathcal{H}_{t})=\sum_{k=1}^{K}\lambda_{k}(t|\mathcal{H}_{t})$.

For ITHP, we estimate its parameters by maximizing the log-likelihood. Regarding the first term, we only need to compute the intensity function at event positions using Eq. 7. As for the second term, the intensity integral generally lacks an analytical expression. Here, we employ numerical integration by discretizing the time axis into a sufficiently fine grid and calculating the intensity function at each grid point using Eq. 7.

Complexity: The utilization of a fine grid does not significantly increase computational time. This is because the attention mechanism facilitates parallel computation of attention outputs for each point. This parallelized computation improves the scalability of ITHP. Parallel computation with more grid points would require additional memory. Fortunately, for one-dimensional temporal point processes, a large number of grids is not necessary. In subsequent experiments, all datasets can run smoothly with only 8GB memory.

We validate the interpretability of ITHP using two sets of 2-variate Hawkes processes data. Each dataset is simulated from a 2-variate Hawkes processes described in Section 3.1, using the thinning algorithm (Ogata, 1998). Both datasets share a common base rate ($\mu=0.2$), but they possess distinct trigger kernels:

• •

  Exponential Decay Kernel This kernel assumes that the influence of historical events decays exponentially as time elapses. The kernel function is given by: $\phi_{ij}(\tau)=\alpha_{ij}\exp(-\beta_{ij}\tau)$ for $\tau>0$, where $j$ is the source type and $i$ is the target type. Specifically, $\alpha_{11}=\alpha_{22}=3$, $\alpha_{12}=2$, $\alpha_{21}=1$ and $\beta_{ij}=5$ for all $i,j$.

• •

  Half Sinusoidal Kernel This kernel assumes the influence of historical events follows a sinusoidal pattern as time elapses and disappears when the interval surpasses $\pi$. The kernel function is given by: $\phi_{ij}(\tau)=\alpha_{ij}\sin(\tau)$ for $0<\tau<\pi$. Likewise, $j$ is the source type and $i$ is the target type. Specifically, $\alpha_{11}=\alpha_{22}=0.33$, $\alpha_{12}=0.1$, and $\alpha_{21}=0.05$.

Further elaboration on the simulation process and statistical aspects of the synthetic dataset can be found in Appendix A.

Results: We validate the interpretability of ITHP by reconstructing the trigger kernel. In ITHP, the trigger kernel is represented as $\mathbf{g}^{\top}_{k,k_{i}}(t-t_{i},t_{i})\mathbf{w}_{k}$, which is time-varying. To uncover the time-invariant trigger kernel inherent in the synthetic dataset, we evaluate trigger kernels at various time points and compute their mean. This approach enables us to extract the desired time-invariant trigger kernel (Zhang et al., 2020). The results are presented in Figs. 1 and 1, revealing a noticeable alignment between the learned kernel trends and the patterns exhibited by the ground-truth kernels. Moreover, as depicted in Figs. 1 and 1, the learned intensity function from ITHP exhibits a striking resemblance to the ground-truth intensity function. This observation underscores ITHP’s capability to accurately capture the true conditional intensity function for both exponential decay and half sinusoidal Hawkes processes.

We also visualize the learned attention map of ITHP, which provides a deeper insight into the influence patterns. As depicted in Fig. 2, this is the attention weight matrix of a testing sequence in the context of exponential decay Hawkes process data. The sequence encompasses both event timestamps and grids within the non-event intervals. In the matrix, the rows and columns correspond to events and grids on the sequence (arranged chronologically). The horizontal axis represents the source point, while the vertical axis represents the target point. Only events have the potential to impact subsequent points, whereas grids, lacking actual event occurrences, cannot affect future points. As a result, it is evident that numerous columns corresponding to grids have values of $0$. Due to a masking operation, the upper triangular section, including the diagonal, is set to $0$, which restricts events from influencing the past. Moreover, the color of event columns becomes progressively lighter as time advances, which aligns with the characteristics of the ground-truth exponential decay trigger kernel.

In this section, we extensively evaluate ITHP by comparing it to baseline models across several public datasets. We have selected several network-sequence datasets, including social media (StackOverflow), online shopping (Amazon, Taobao), traffic networks (Taxi), and a widely used public synthetic dataset (Conttime).

Our model has reduced the parameter count but still achieves comparable or even better results compared to THP. This improvement is attributed to the “fully attention-based intensity function” (Section 4.4). THP relies on the parameterized extrapolated intensity, assuming that the intensity function on non-event intervals follows an approximately linear pattern (red term in Eq. 2). However, such an assumption does not align with the actual patterns in real data and can impact the expressive capability of the model. We conduct further ablation studies to illustrate the limitations of the parameterized extrapolation method in Table 1. We implement an extra revised model Ex-ITHP which essentially is “interpretable Transformer” + “extrapolated intensity”. More details about Ex-ITHP is provided in Appendix C. THP, ITHP, and Ex-ITHP naturally constitute an ablation study. Ex-ITHP has fewer parameters as it removes the parameters $\mathbf{W}^{Q}$ and $\mathbf{W}^{K}$, and uses a less flexible extrapolated intensity. In Table 1, the Ex-ITHP exhibits the poorest performance due to its fewer parameters and restricted intensity flexibility. THP performs moderately, having more parameters but still restricted intensity flexibility. Conversely, the ITHP, despite having fewer parameters, outperforms THP on most datasets owing to its more flexible intensity expression.

Additionally, we visualize the difference between the learned fully attention-based intensity and the learned extrapolated intensity for a segment of the sequence in Half Sinusoidal Kernel Hawkes Synthetic dataset (Section 5.1). As depicted in Fig. 4, on non-event intervals, THP, constrained by the approximately linear extrapolation, struggles to capture the fluctuating intensity patterns and can only learn an intensity that is approximately linear. Additionally, due to the limited variation in intensity on non-event intervals, large jumps are required when a new event occurs to maintain a height similar to the ground-truth intensity. In contrast, our proposed ITHP demonstrates greater flexibility, successfully capturing the fluctuating pattern on non-event intervals, and accurately fitting the scale level.

Our model’s configuration primarily encompasses two dimensions: the encoding dimension, denoted as $M$, and the Value dimension, denoted as $M_{V}$. We maintain the skip connection within the implementation of the encoder which necessitates that $M_{V}$ must be equal to $2M$. We test the sensitivity of model performance to hyperparameters by using various hyperparameter configurations on one toy dataset and one public dataset: the half-sine and Taxi datasets. The results of our experiments are shown in Table 2. The results indicate that our model is not significantly affected by the hyperparameter variation. Additionally, it can achieve reasonably good performance even with fewer parameters.