CoSEM: Contextual and Semantic Embedding for App Usage Prediction

First, we discuss the preliminaries, including semantic and contextual information, to formulate app usage prediction problems.

Semantic Information ($\mathcal{S}$) : represents the goal, intent, or condition of users when they use a specific mobile application. The semantic information of a user $u$ in time window $T$ can be defined as $\mathcal{S}^{u}_{T}$ = {$s^{u}_{1},s^{u}_{2},...,s^{u}_{m}$}, where $s$ is the chunk of semantic information (e.g., words in the search query, tasks, location, time), and $m$ is the total number of pieces in semantic information.

Contextual information ($\mathcal{A}$) : used in this work is the user’s historical application usage. The contextual information of the user $u$ can be represented as $\mathcal{A}^{u}_{t<T}$ = {$a^{u}_{1},a^{u}_{2},...,a^{u}_{n}$}, where $a^{u}$ is the application, and $n$ is the number of historical applications used by a particular user $u$ before the time window $T$.

App usage prediction: A model $z$ is developed and trained with $\mathcal{S}$ and $\mathcal{A}$. Specifically, given a semantic information $\mathcal{S}^{u}_{T}$ in current time window $T$ and the contextual information $\mathcal{A}^{u}_{t<T}$, the trained model $z$ predicts a set of applications to be used during $T$. This problem can be formally defined as follows.

Here, $\mathcal{P}^{u}_{T}$ is the set of predicted applications.

This section discusses our app usage prediction model called Contextual and Semantic Embedding (CoSEM). The overview of CoSEM is illustrated in figure 1. In CoSEM, the chunk of semantic information in the time $T$ will be mapped to a unique vector $v_{s}$ in matrix $MS$, and an app $a$ in the historical app list, used in the previous period ($t<T$), is also mapped to a vector $v_{a}$ in matrix $MA$. The $MS$ and $MA$ matrices contain all possible semantic information chunks and applications used in each dataset. CoSEM takes $M$ semantic information vectors, and $N$ used app vectors as input to predict the set of apps used in time $T$. These matrices $MA$ and $MS$ are shared with all users, i.e., the app of ‘Youtube’ vector and the semantic information vector of a word ’recipe’ are the same for all users. The values in these matrices will be updated when we train the model.

The first layer is the embedding layer. Through this layer, the chunk of semantic information or historically used apps will map to a low-dimensional dense vector $V_{s}$ and $V_{a}$ using an embedding lookup table E $\in$ $R^{Q\times D}$, where $Q$ can be vocabulary size, location size, or possible application size depending on the type of inputs and D is the embedding dimension. The parameters of this embedding layer are tuned during model training.

Semantic information embedding The semantic information in each dataset can be different. This experiment evaluates our framework with three types of semantic information, including the task description, the task query, and the location id and time, from 3 different real-world datasets. To find the representation of semantic information, firstly, we divide the data into chunks and then remove the irrelevant chunks (e.g., for the task description, we remove all stop words contained in the task description). After that, we embed the chunk of semantic information via an embedding layer to vectors $Vs_{1}$, $Vs_{2}$,…,$Vs_{m}$ and then feed these vectors as the input to the vector summarizing process based on the following equation $SV_{s}=\frac{\sum_{l=1}^{m}{V_{s_{l}}}}{m}$. Where $SV_{s}$ is the summarization vector of semantic information, $V_{s_{l}}$ is the vector of each semantic chunk, and $m$ is the length of semantic chunks

Contextual information embedding To overcome the limitation of the user vector when the user’s data is inadequate, we introduce the concept of contextual information embedding that uses the historical app usage of an individual. The last $n$ most recent apps that a user used in the time slot before $T$ are taken as an input to the embedding layer for mapping the representation vector of each app. Then, these app vectors will be fed to the vector summarization process as illustrated by the equation $SV_{a}=\frac{\sum_{l=1}^{n}{V_{a_{l}}}}{n}$. Where $SV_{a}$ is the summarization vector of historical apps, $V_{a_{l}}$ is the vector of app $l$, and $n$ is the length of historical app sequences.

After obtaining the representation vectors of semantic information and contextual information, we forward them to the Dual-DNN model from the Appusage2Vec model (Zhao et al., 2019) as input features. In that paper, the Dual-DNN is designed to incorporate user behavior and app usage information. We adapted this Dual-DNN to utilize personalized user behavior regarding app usage and the semantic information for app prediction. It consists of two parallel deep neural networks, while each DNN is the fully connected layers using Tanh as the activation function. In this work, these DNNs will receive the vector from the semantic information embedding process, and the historical app usage vector. Finally, we combine these two vectors using the Hadamard product, which is the point-wise multiplication, instead of using a concatenation operation. The simple concatenation treats the contextual vector and semantic vector separately, and the point-wise multiplication can build the vector representing both semantic and contextual information. The outcome from the Hadamard product will be taken as input in the sigmoid layer to calculate the probability of each app.

We discuss three real-world datasets used in our research to demonstrate the generality and effectiveness of our proposed model for app usage prediction with varying semantic information.

Evaluation metrics. Efficiency is measured by the hit rate (HR) and the mean reciprocal rank (MRR). We include the MRR score instead of using only the hit rate score because the hit rate score calculates only the hit ratio of the top-k Apps without considering the order of predicted apps. As long as correct apps are in top-k predict apps, it is counted as one hit. In contrast, the MRR considers the order of predicted apps. Therefore, we choose both the MRR and HR as evaluation metrics. Several prior works have used these matrices to evaluate their model’s performance (Aliannejadi et al., 2018a, b; Liao et al., 2013a; Zhao et al., 2019; Liao et al., 2013b; Zhou et al., 2011). We use the following equation ${MRR}=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{f_{id}}$ to calculate the MRR score. Here, $|Q|$ is the number of testing instances, $f_{id}$ is the first position of correctly predicted apps in each instance. In this paper, we set the number of predicted apps to 5, so the MRR metric will be calculated with these five predicted applications, $f_{id}\in[1,5]$.

Data separation. To evaluate our proposed model, we sort the data of each user in each dataset chronologically and take the first 70% of each user for the training model, the next 10% for validation, and the rest for testing. This dataset setting will be applied to all baselines. In the tuning process, the MRR value on the validation set is used to select the hyperparameters for our model.

We compare our work to several state-of-the-art models, showing the performance of our proposed model for the app usage prediction problem from semantic information.

• Most recently used (MRU) considers the $k$ apps that were the most recently used by each user as predicted results. Note that for ISTAS data, we consider the last applications used for searching the query as the most recently used app.

• NTAS2 is the Neural Target Apps Selection model presented in (Aliannejadi et al., 2018b). We use the NTAS2 model to compare the performance of our work with the semantic information from each dataset.

• CNTAS is the Context-Aware Neural target apps selection, in (Aliannejadi et al., 2018a). This model is the improved version of NTAS, which considers contextual information when selecting the app as a predicted outcome. The CNTAS model creates the representation vector of candidate apps, semantic, and contextual information. The model is trained to ensure that the vector of the right candidate app more similar to the vector of semantic and contextual information than the wrong candidate app (negative app). This makes the CNTAS model different from our work, although we use both the semantic and contextual information. As CPS-Task and China Telecom dataset, we use the previous hour’s app usage as the contextual information because the last 24 hours’ app usage is not available in those datasets. Note that we use all possible apps as candidate apps, as the proposed paper (Aliannejadi et al., 2018a) does not mention the number of candidate apps used in the training process.

• Appusage2Vec (Zhao et al., 2019) embeds apps and users as vectors and treats them as an input for predicting the next app. The experiment with Appusage2Vec is presented only for the CPS-Task and China Telecom datasets, as the ISTAS dataset lacks the time spent on each app, which is a necessary feature in their framework.

We compared CoSEM with the baseline approaches, as illustrated in Table 1. Other approaches used the sequence of apps and an embedding dimension in the same size as CoSEM to predict the set of apps. As illustrated in Table 1, our CoSEM performs best in all of the three datasets. There is a significant improvement in our model on both MRR and HR metrics. This result indicates that combining the semantic information and historically used apps is helpful for the app prediction problem. Compared to the other baseline model, our CoSEM outperforms the Appusage2Vec in both the CPS-task dataset and the China telecom dataset around 0.09 and 0.34 on MRR score, respectively. A similar result compared with the CNTAS and NTAS2 models, our model also receives a better MRR and HR score in all datasets and achieves a 0.1 - 0.2 improvement over these models in terms of HR score. The reproduced result of the CNTAS model is different from their proposed paper because we drop users who have insufficient data, and the number of negative apps is used in the training process. The reason that CoSEM outperforms other baselines due to three reasons. Firstly, Appusage2Vec can achieve the best performance when applied to a vast dataset (more than 8k users, around 315 app sequences/ user), while the CPS-Task dataset only consists of 53, and the China telecom dataset is consists of 1000 users. Second, combining a user vector from the user id and app sequence vector is not enough for predicting the set of apps used to perform the task. Finally, Appusage2Vec, NTAS2, and CNTAS are designed for selecting the only app that will use by the user, not predicting the set of apps used.

To investigate the effectiveness of CoSEM, we perform ablation study with the following variants of the proposed model:

• DNN-A (LeCun et al., 2015) stacks fully-connected neural layers to learn nonlinear, hidden, and implicit features from a historical app list and then applies logistic regression with a sigmoid layer for predicting an app set. We use the embedding layer to vectorize the historical app list and then input it to fully-connected layers.

• DNN-S stacks fully-connected neural layers with the embedding layer to embed the information similar to DNN-A, except taking semantic information as input.

We found that the ablation of the different parts of our model influenced the average MRR score. The removal of semantic information caused a reduction in both MRR and HR scores which are 0.01 and 0.03 respectively. While the ablation of app usage information resulted in a significant decline in both scores, this indicates that the app usage information is the essential part of our model; however, the model needs the semantic information and app usage information to get the best performance.