A State Transition Model for Mobile Notifications
via Survival Analysis

Social networking services (e.g., Facebook, LinkedIn, Instagram, Twitter, WeChat) actively push information to their users through mobile notifications. As the content ecosystem and users’ connection networks grow, more and more information is generated on the social networking site that is worth informing the users. On the other hand, users have limited attention span, regardless of how much value notifications could inform them of. The discrepancy between increasing content and limited user attention is the challenge many mobile applications are facing, especially those social networking applications.

A mobile notification is a message displayed to the user either through the mobile application user interface (UI) itself, or through the operating system’s push notification services, such as Apple Push Notification Service (APNs). Instances of such messages include a user-to-user communication, a friend or connection request, an update from a friend or connection (e.g., birthday or job change), an article posted by a connection, etc. These notifications help keep the users informed of what is happening in their network. In addition, notifications also serve the purpose of promotions and product marketing for many mobile applications.

Compared with email communication, mobile notifications are more time sensitive and more promptly responded to (Fischer et al., 2011; Pielot et al., 2014a). Without an established way to determine delivery time, mobile notifications often arrive at inconvenient moments, failing to provide value to a user. Moreover, due to the pervasive nature of smartphones, such inconvenience may lead to complaints or even disablement on future notification deliveries, causing a permanent loss to both service providers and users. In short, sending notifications at the right time with the right content in many cases is critical.

In this paper, we focus on a quantitative way to measure the effectiveness of a mobile notification and to learn the pattern of how the effectiveness differs from user to user and from time to time. The overall objective is to improve personalization and ensure better delivery time and volume optimization.

The interaction of a user with mobile notifications can be very complex and depends on numerous aspects (Mehrotra et al., 2015; Mehrotra et al., 2014; Xu et al., 2013). It is common to link a notification event to one or more rewards to evaluate the effectiveness of a notification. For engagement, a typical reward is a visit from the user to the app. One challenge for such a study is how to attribute a reward, because users may receive multiple notifications before they open and visit the app. Simple strategies could be to attribute the reward to the most recent one or to several notifications delivered within a look-back time period. Such strategies are hard to justify theoretically and could introduce significant bias in learning. Our strategy is to leverage the survival analysis to attribute a reward without ambiguity and bias (Buckley and James, 1979; James and Smith, 1984).

Survival analysis is commonly used within medical and epidemiological research to analyze data where the outcome variable is the time until the occurrence of an event of interest. For example, if the event of interest is heart attack, then the time to event or survival time can be the time in years until a person develops a heart attack. In survival analysis, subjects are usually followed over a specified time period and the focus is on the time at which the event of interest occurs. Survival time has two components that must be clearly defined: a beginning point and an endpoint that is reached either when the event occurs or when the follow-up time has ended. If the event does not occur by the follow-up time, the observation is called censored. The censored observations are known to have a certain amount of time where the event of interest did not occur and it is not clear whether the event would have occurred if the follow-up time were longer. Such censoring is very common in observational notification data.

We introduce survival analysis to notification modeling as a new domain. The beginning point in this case is the delivery time of a notification and the endpoint is the reward time (e.g., the time of a visit) or a next notification delivery time, whichever happens first. When the next notification occurs first, the observation is censored. In this paper, we apply an accelerated failure-time model (Wei, 1992; Keiding et al., 1997) with a Weibull distribution to our large-scale user data for the reward prediction. This turns out to be not just novel, but also superior in prediction performance compared to baseline models in our offline analysis.

We provide two example formulations for notification volume optimization (VO) and delivery time optimization (DTO) separately. We then present an online use case on notification DTO, where our model is used to make send decisions. The A/B test results show significant improvement on user engagement and content consumption over a non-DTO control and a baseline DTO model.

The major contributions of this paper can be summarized as follows.

• •

  We develop a state transition model to measure the effectiveness of a notification through a delta effect $\Delta F(W_{0},T)$ in Section 3.

• •

  We propose to estimate the delta effect in the presence of censored data, using a log linear survival structure and a Weibull distribution.

• •

  We conduct offline evaluations with real-world notification data to demonstrate the accuracy and flexibility of our engagement prediction.

• •

  We carry out an online use case of determining the delivery time and show superiority of the proposed method with A/B tests in Section 7.

Email communication as a channel has a long history for social networking services. A volume optimization framework (Gupta et al., 2016; Gupta et al., 2017) can simultaneously minimize the number of emails sent, control the negative complaints, and maximize user engagement. While we share similar goals for mobile notifications, there are unique mobile aspects to be considered. Moreover, the volume optimization framework focuses on solving a Multi-Objective Optimization (MOO) problem (Agarwal et al., 2011, 2012), in which multiple objectives are optimized under given constraints. We focus on probabilistic nature of interactions between a user and a notification. Our work can be leveraged as a utility prediction model, which would be one of the utilities of interest in a MOO formulation for mobile notifications.

As more mobile applications push information to users, several studies have been carried out to understand how to make effective use of notifications. Sahami et al. (Sahami Shirazi et al., 2014) collect close to 200 million notifications from more than 40,000 users including users’ subjective perceptions and present the first large-scale analysis of mobile notifications. A number of findings about the nature of notifications, such as shorter responding time, have shed light on how to effectively use them. Pielot et al. (Pielot et al., 2014a) carry out an one-week, in-situ study involving 15 mobile phones users and suggested that an increase in the number of notifications is associated with an increase in negative emotions. Both works do not attempt to model the interactions probabilistically.

Xu et al. (Xu et al., 2013) developed an app usage prediction model that leverages the user’s day-to-day activities, app preferences and the surrounding environment. Mehrotra et al. (Mehrotra et al., 2015) developed a classification model to predict notification acceptance by considering both content and context information. Pielot et al. (Pielot et al., 2014b) proposed a machine learning model to predict whether the user will view a message within the next few minutes or not after a notification is delivered. Their study also suggests that indicators of availability, such as the last time the user has been online, not only create social pressure, but are also weak predictors of attentiveness of the message. Pielot et al. (Pielot et al., 2017) carried out a field study with hundreds of mobile phone users and built a machine-learning model to predict whether a user will click on the notification and subsequently engage with the content. The model can be used to determine the opportune moments to send notifications. These studies focus on cross-application study with complete device information, yet the scale of notifications and users are not comparable to our case.

On general user engagement, extensive studies (Attenberg et al., 2009; Dave et al., 2014; Wang et al., 2013; Ashley and Tuten, 2015; Khan, 2017) have been promoting relevant and high quality content to users to maximize long-term user engagement with the platform. Other works (Goldstein et al., 2013; Yoo and Kim, 2005) show that low-quality advertising has detrimental effect on long-term user engagement. Zhou et al. (Zhou et al., 2016) developed an ad quality model based on logistic regression to identify offensive ads that affect user engagement. The focus has been on the quality instead of the timing.

Most applications of the survival analysis in the literature have been in medicine, biology or public health, but there is an increasing interest in its applications to social behavior. Survival techniques based on Weibull distributions have been applied to understanding and predicting dwell time on web services (Liu et al., 2010; Vasiloudis et al., 2017). Yu et al. (Yu et al., 2017) proposed a temporally heterogeneous survival framework to model social behavior dynamics, whose model parameters can be solved by maximum likelihood estimation. The model can be applied to user-to-user communication. Gomez-Rodriguez et al. (Gomez-Rodriguez et al., 2013) studied the formation of an information cascade in a network based on survival theory. Last but not least, Li et al. (Li et al., 2017) applied survival analysis in modeling the career paths. They formulated the problem as a multi-task learning and achieved favorable performance against several other state-of-the-art machine learning methods.

A mobile notification may be delivered through one or many channels such as a sound, a badge count update on the app icon, and an alert shown on the lock screen or as banners. A UI push notification shown in Figure 1 refers to one with an alert showing the content of the message. Such notifications are more effective at drawing a user’s attention but they can also be intrusive or even annoying. As suggested in studies (Sahami Shirazi et al., 2014; Pielot et al., 2014a), the UI push channel is better for time-sensitive and potentially important notifications, e.g., a connection invitation or a user-to-user message. Other less time-sensitive ones, e.g., a connection’s birthday or work anniversary, can be served as badging notifications, meaning we only push a badge count update as shown in Figure 2 and a user has to open the app to see the content as an in-app notification within the mobile application’s UI in Figure 3. Unlike UI push notifications, such badging notifications are much less intrusive. On the other hand, the effect of them are more subtle. Users are not able to view and interact with the notification content without opening the app. It usually takes longer time for a user to respond to the badging than the UI push and it is harder to separate the effect of notifications from the organic visits. Attribution challenges also arise when multiple badging notifications have been delivered with more than one badge count. This challenge is further elaborated in Section 3.2 as data censoring. The content of badging notifications are usually less time-sensitive and hence we have more flexibility in their delivery time.

In this section, we develop a state transition model to describe and predict the interactions between users and notifications. We focus on badging notifications, which has a more subtle effect to model and is less studied in the literature. The methodology can be extended to UI push notifications with possibly different distribution assumptions as they are responded to more quickly.

Collecting large-scale unbiased training data is challenging, especially in the case of observational data. We collect data at LinkedIn from hundreds of millions of users for a given week including all badging notification events delivered to users and all user visit events. For each notification event, we include 3 broad categories of features in $\boldsymbol{X_{i}}$.

• •

  user’s profile features such as locale and network size.

• •

  State features such as badge count.

• •

  user’s activity features such as user’s last visit time, the number of site visits over the past week and the number of notifications received over the past week.

In addition, we also include interactions between the above features, such as interaction terms between the badge count and the profile features so that we can learn different sensitivity to the badge count from different users.

To get the response $T_{i}$ and censoring indicator $\delta_{i}$, we sort notification events and visit events in the temporal order for each user so that we can get the next event type and next event time $T_{i}$. If the next event is a visit, then $\delta_{i}=1$; otherwise $\delta_{i}=0$. Note that the next event and next visit may extend beyond the given week, and thus the following week’s data may be needed and joined accordingly. We then remove potential outliers by discarding records from users who receive too many notifications or visit too many times. Such records may come from erroneous tracking or abnormal accounts. Next, we split a week’s notification data into training and test data with a ratio of 4:1. The test data are held out for evaluation in Section 6.

We train the AFT model with the training data on using Spark MLlib (Meng et al., 2016) and obtain $\hat{\boldsymbol{b}}$ and $\hat{\sigma}$. Parameters in the conditional Weibull distribution can be calculated as $\hat{\alpha}=1/\hat{\sigma}$, $\hat{\lambda}_{i}=e^{-\hat{\boldsymbol{b}}\boldsymbol{X}_{i}/\hat{\sigma}}$.

The model we learned from training data suggests very different feature importance from that of a notification CTR model. For example, the badge count is a strong predictor and most people are more sensitive to one badge count increase when the badge count is low and become indifferent when the badge count is high. On the other hand, the badge count, the time after the last notification sent are usually not strong signals for a notification CTR model based on our previous experience. The two models can be complementary to each other in a MOO setup described in Section 5.1, since they seem to capture different aspects of notifications.

The $\hat{\sigma}$ we learned is greater than $1$, so we have $\hat{\alpha}\in(0,1)$, suggesting that the $\Delta F_{i}(W_{0,i},T)$ in (3) is increasing in $W_{0}$ according to Lemma 3.1. This is aligned with our intuition that the longer time spacing we have from the previous notification send time, the more incentive we have to send another notification.

The model we train also suggests that the marginal effect on user engagement diminishes as the badge count increases. The interaction between badge count and user features are significant, meaning different users have different sensitivity to badging.

In this section, we show how our model can be leveraged by different notification decision systems.

We compare our proposed survival-based approach with the conventional baseline logistic regression model. While there are potentially more accurate baseline models such as tree models and deep models, survival models can also be extended beyond a linear structure (Ishwaran et al., 2008; Ranganath et al., 2016). Such a comparison isolates the impact of data censoring and the survival approach from that of feature engineering. For any given time frame $T$, we train a logistic regression with the same set of features including their interactions $\boldsymbol{X}_{1,i}$ and a response of whether a user’s visit occurs within $T$ after the notification is delivered. One advantage of our formulation over a classification task is that the same model can be used to predict a user’s probability of visiting given any time frame $T$ through a Weibull distribution $F(T;\lambda,\alpha)$ in (7). Therefore, the same model can be deployed in different applications, where the prediction windows are chosen differently. On the other hand, we need to train an individual logistic regression model for each different $T$ since the response variables are different.

To evaluate the prediction performance, we calculate the area under the receiver operating characteristic curve (AUC) for selected $T$ values as binary classification problems. For the AFT model, we calculate $F(T;\hat{\lambda},\hat{\alpha})$ in (7) to be used in the same way as the logistic prediction for the AUC. Figure 6 shows how our model compares with the baseline model in terms of the AUCs as a function of the prediction time window $T$. The shorter the time window is, the harder the prediction as a binary classification is, since the randomness of the users’ engagement behavior tends to be dominating in the short term. At $4$ hours prediction window, the model already gives a reasonable AUC of about $0.74$ while the logistic regression model only gives $0.58$. At $24$ hours for daily engagement prediction, the model gives an AUC as high as $0.85$ while the baseline model reaches $0.73$. Interestingly, as we further increase the prediction window, the AUC of the logistic regression model starts to fall while our model reaches $0.89$ at $48$ hours. The decline of the logistic regression could come from bias introduced by attributing a visit event to multiple notification events within the time window $T$. This bias becomes more severe as the time window $T$ increases and likely covers more notifications. The AFT model, on the other hand, avoids such bias by correctly incorporating information from both censored and uncensored observations.

The results show that handling censoring properly is very crucial to mobile notification data. In addition, our model achieved great flexibility in $T$ and superior prediction power compared with the logistic classification models at every given $T$.

In this section, we present a case study deployed at LinkedIn to show how we improve our decision making for mobile notifications with our model.

To our best knowledge, this is the first work on probabilistic modeling of interactions between mobile notifications and user engagement at scale. We develop a state transition model and derive a delta effect to measure the effectiveness of a notification. With a common existence of censoring in observational mobile notification data, we estimate the delta effect through an AFT regression with a Weibull distribution. The prediction from this survival regression is both flexible to apply and superior in prediction accuracy compared to baseline models with the same feature set.

Our state transition model is generalizable and can have broader applications. While we focus on modeling the badging notifications, our model is applicable for all types of mobile notifications. For example, UI push notifications can be modeled with a distribution possibly different from a Weibull distribution.

We consider a user’s visit as a reward to a mobile notification. However, the reward can be generalized to a user’s purchase event for on-line shopping apps such as Amazon or a user’s content creation event for question-and-answer apps such as Quora. In the cases where data censoring is a major concern for modeling mobile notifications, we provide a general framework to evaluate the effectiveness of a notification towards driving a reward.

We would sincerely like to thank Rupesh Gupta, Matthew Walker, Kinjal Basu, Yan Gao, Haoyu Wang, Myunghwan Kim, Guangde Chen, Ajith Muralidharan for their detailed and insightful feedback during the development of this model.