GIANT: Scalable Creation of a Web-scale Ontology

We propose a novel algorithm to extract various types of attention phrases (e.g., concepts, topics or events), which represent user attentions or interests, from a collection of queries and document titles.

Problem Definition. Suppose a bipartite search click graph $G_{sc}=(Q,D,E)$ records the click-through information over a set of queries $Q=\{q_{1},q_{2},\cdots,q_{|Q|}\}$ and documents $D=\{d_{1},d_{2},\cdots,d_{|D|}\}$. We use $|*|$ to denote the length of $*$. $E$ is a set of edges linking queries and documents. Our objective is to extract a set of phrases $P=\{p_{1},p_{2},\cdots,p_{|P|}\}$ from $Q$ and $D$ to represent user interests. Specifically, suppose $p$ consists of a sequence of words $\{w_{p1},w_{p2},\cdots,w_{p|p|}\}$. In our work, each phrase $p$ is extracted from a subset of queries and the titles of correlated documents, and each word $w_{p}\in p$ is contained by at least one query or title.

Algorithm 1 presents the pipeline of our system to extract attention phrases based on a bipartite search click graph. In what follows, we introduce each step in detail.

Query-Doc Clustering. Suppose $c(q_{i},d_{j})$ represents how many times query $q_{i}$ is linked to document $d_{j}$ in a search click graph $G_{sc}$ constructed from user search click logs within a period. For each query-doc pair $<q_{i},d_{j}>$, suppose $N(q_{i})$ denotes the set of documents connected with $q_{i}$, and $N(d_{j})$ denotes the set of queries connected with $d_{j}$. Then we define the transport probabilities between $q_{i}$ and $d_{j}$ as:

From query $q$, we perform random walk (Spitzer, 2013) according to transport probabilities calculated above and compute the weights of visited queries and documents. For each visited query or document, we keep it if its visiting probability is above a threshold $\delta_{v}$ and the number of non-stop words in $q$ is more than a half. In this way, we can derive a cluster of correlated queries $Q_{q}$ and documents $D_{q}$.

Query-Title Interaction Graph Construction. Given a set of queries $Q_{q}$ and $D_{q}$, the next step is to extract a representative phrase that captures the underlying user attentions or interests. Figure 3 shows a query-doc cluster and the concept phrase extracted from it. We can extract ‘Hayao Miyazaki animated film (宫崎骏—动画—电影)” from the input query-title cluster. An attention phrase features multiple characteristics. First, the tokens in it may show up multiple times in the queries and document titles. Second, the phrase tokens are not necessarily consecutively or fully contained by a single query or title. For example, in Figure 3, additional tokens such as “famous (著名的)” will be inserted into the phrase tokens, making them not a consecutive span. Third, the phrase tokens are syntactically dependent even if they are not consecutively adjacent in a query or title. For example, “Hayao Miyazaki (宫崎骏)” and “film (电影)” forms a compound noun name. Finally, the order of phrase tokens may change in different text. In Figure 3, the tokens in the concept phrase “Hayao Miyazaki animated film (宫崎骏—动画—电影)” are fully contained by the query and two titles, while the order of the tokens varies in different queries or titles. Other types of attention phrases such as events and topics will feature similar characteristics.

To fully exploit the characteristics of attention phrases, we propose Query-Title Interaction Graph (QTIG), a novel graph representation of queries and titles to reveal the correlations between their tokens. Based on it, we further propose GCTSP-Net, a model that takes a query-title interaction graph as input, performs node classification with graph convolution, and finally generates a phrase by Asymmetric Traveling Salesman Decoding (ATSP-decoding).

Figure 3 shows an example of query-title interaction graph constructed from a set of queries and titles. Denote a QTIG constructed from queries $Q_{q}$ and the titles $T_{q}$ of documents $D_{q}$ as $G_{qt}(Q_{q},T_{q})$. The queries and documents are sorted by the weights calculated during the random walk. In $G_{qt}(Q_{q},T_{q})$, each node is a unique token belonging to $Q_{q}$ or $T_{q}$. The same token present in different input text will be merged into one node. For each pair of nodes, if they are adjacent tokens in any query or title, they will be connected by a bi-directional “seq” edge, indicating their order in the input. In Figure 3, the inverse direction of a “seq” edge points to the preceding words, which is indicated by a hollow triangle pointer. If the pair of nodes are not adjacent to each other, but there exists syntactical dependency between them, they will be connected by a bi-directional dashed edge which indicates the type of syntactical dependency relationship and the direction of it, while the inverse direction is also indicated by a hollow triangle pointer.

Algorithm 2 shows the process of constructing a query-title interaction graph. We construct the nodes and edges by reading the inputs in $Q_{q}$ and $T_{q}$ in order. When we construct the edges between two nodes, as two nodes may have multiple adjacent relationships or syntactical relationships in different inputs, we only keep the first edge constructed. In this way, each pair of related nodes will only be connected by a bi-directional sequential edge or a syntactical edge. The idea is that we prefer the “seq” relationship as it shows a stronger connection than any syntactical dependency, and we prefer the syntactical relationships contained in the higher-weighted input text instead of the relationships in lower-weighted inputs. Compared with including all possible edges in a query-title interaction graph, empirical evidence suggests that our graph construction approach gives better performance for phrase mining.

After constructing a graph $G_{qt}(Q_{q},T_{q})$ from a query-title cluster, we extract a phrase $p$ by our GCTSP-Net, which contains a classifier to predict whether a node belong to $p$, and an asymmetric traveling salesman decoder to order the predicted positive nodes.

Node Classification with R-GCN. In the GCTSP-Net, we apply Relational Graph Convolutional Networks (R-GCN) (Kipf and Welling, 2016; Gilmer et al., 2017; Schlichtkrull et al., 2017) to our constructed QTIG to perform node classification.

Denote a directed and labeled multi-graph as $G=(V,E,R)$ with labeled edges $e_{vw}=(v,r,w)\in E$, where $v,w\in V$ are connected nodes, and $r\in R$ is a relation type. A class of graph convolutional networks can be understood as a message-passing framework (Gilmer et al., 2017), where the hidden states $h^{l}_{v}$ of each node $v\in G$ at layer $l$ are updated based on messages $m^{l+1}_{v}$ according to:

where $N(v)$ denotes the neighbors of $v$ in graph $G$, $M_{l}$ is the message function at layer $l$, and $U_{l}$ is the vertex update function at layer $l$.

Specifically, the message passing function of Relational Graph Convolutional Networks is defined as:

where $\sigma(\cdot)$ is an element-wise activation function such as ReLU$(\cdot)=\text{max}(0,\cdot)$. $N^{r}(v)$ is the set of neighbors under relation $r\in R$. $c_{vw}$ is a problem-specific normalization constant that can be learned or pre-defined (e.g., $c_{vw}=|N^{r}(v)|$). $W^{l}_{r}$ and $W^{l}_{0}$ are learned weight matrices.

We can see that an R-GCN accumulates transformed feature vectors of neighboring nodes through a normalized sum. Besides, it learns relation-specific transformation matrices to take the type and direction of each edge into account. In addition, it adds a single self-connection of a special relation type to each node to ensure that the representation of a node at layer $l+1$ can also be informed by its representation at layer $l$.

To avoid the rapid growth of the number of parameters when increasing the number of relations $|R|$, R-GCN exploits basis decomposition and block-diagonal decomposition to regularize the weights of each layer. For basis decomposition, each weight matrix $W^{l}_{r}\in\mathbb{R}^{d^{l+1}\times d^{l}}$ is decomposed as:

where $V^{l}_{b}\in\mathbb{R}^{d^{l+1}\times d^{l}}$ are base weight matrices. In this way, only the coefficients $a^{l}_{rb}$ depend on $r$. For block-diagonal decomposition, $W^{l}_{r}$ is defined through the direct sum over a set of low-dimensional matrices:

where $W^{l}_{r}=\text{diag}(Q^{l}_{1r},Q^{l}_{2r},\cdots,Q^{l}_{br})$ is a block-diagonal matrix with $Q^{l}_{br}\in\mathbb{R}^{(d^{l+1}/B)\times(d^{l}/B)}$. The basis function decomposition introduces weight sharing between different relation types, while the block decomposition applies sparsity constraint on the weight matrices.

In our model, we apply R-GCN with basis decomposition to query-title interaction graphs to perform node classification. For each node in the graph, we represent it by a feature vector consisting of the embeddings of the token’s named entity recognition (NER) tag, part-of-speech (POS) tag, whether it is a stop word, number of characters in the token, as well as the sequential id that indicates the order each node be added to the graph. Using the concatenation of these embeddings as the initial node vectors, we pass the graph to a multi-layer R-GCN with a $\text{softmax}(\cdot)$ activation (per node) on the output of the last layer. We label the nodes belonging to the target phrase $p$ as $1$ and other nodes as $0$, and train the model by minimizing the binary cross-entropy loss on all nodes.

Node Ordering with ATSP Decoding. After classified a set of tokens $V_{p}$ as target phrase nodes, the next step is to sort the nodes to get the final output. In our GCTSP-Net, we propose to model the problem as an asymmetric traveling salesman problem (ATSP), where the objective is to find the shortest route that starts from the “sos” node, visits each predicted positive nodes, and returns to the “eos” node. This approach is named as ATSP-decoding in our work.

We perform ATSP-decoding with a variant of the constructed query-title interaction graph. First, we remove all the syntactical dependency edges. Second, instead of connecting adjacent tokens by a bi-directional “seq” edge, we change it into unidirectional to indicate the order in input sequences. Third, we connect “sos” with the first predicted positive token in each input text, as well as connect the last predicted positive token in each input with the “eos” node. In this way, we remove the influence of prefixing and suffixing tokens in the inputs when finding the shortest path. Finally, the distance between a pair of predicted nodes is defined as the length of the shortest path in the modified query-title interaction graph. In this way, we can solve the problem with Lin-Kernighan Traveling Salesman Heuristic (Helsgaun, 2000) to get a route of the predicted nodes and output $p$.

We shall note that ATSP-decoding will produce a phrase that contains only unique tokens. In our work, we observe that only less than $1\%$ attention phrases contain duplicated tokens, while most of the duplicated tokens are punctuations. Even if we need to produce duplicate tokens when applying our model to other tasks, we just need to design task-specific heuristics to recognize the potential tokens (such as count their frequency in each query and title), and construct multiple nodes for it in the query-title interaction graph.

Attention Phrase Normalization. The same user attention may be expressed by slightly different phrases. After extracting a phrase using GCTSP-Net, we merge highly similar phrases into one node in out Attention Ontology. Specifically, we examine whether a new phrase $p_{n}$ is similar to an existing phrase $p_{e}$ by two criteria: i) the non-stop words in $p_{n}$ shall be similar (same or synonyms) with that in $p_{e}$, and ii) the TF-IDF similarity between their context-enriched representations shall be above a threshold $\delta_{m}$. The context-enriched representation of a phrase is obtained by using itself as a query and concatenating the top $5$ clicked titles.

Training Dataset Construction. To reduce human effort and accelerate the labeling process of training dataset creation, we design effective unsupervised strategies to extract candidate phrases from queries and titles, and provide the extracted candidates together with query-title clusters to workers as assistance. For concepts, we combine bootstrapping with query-title alignment (Liu et al., 2019). The bootstrapping strategy exploits pattern-concept duality: we can extract a set of concepts from queries following a set of patterns, and we can learn a set of new patterns from a set of queries with extracted concepts. Thus, we can start from a set of seed patterns, and iteratively accumulate more and more patterns and concepts. The query-title alignment strategy is inspired by the observation that a concept in a query is usually mentioned in the clicked titles associated with the query, yet possibly in a more detailed manner. Based on this observation, we align a query with its top clicked titles to find a title chunk which fully contains the query tokens in the same order and potentially contains extra tokens within its span. Such a title chunk is selected as a candidate concept.

For events, we split the original unsegmented document titles into subtitles by punctuations and spaces. After that, we only keep the set of subtitles with lengths between $L_{l}$ (we use 6) and $L_{h}$ (we use 20). For each remaining subtitle, we score it by counting how many unique non-stop query tokens within it. The subtitles with the same score will be sorted by its click-through rate. Finally, we select the top ranked subtitle as a candidate event phrase.

Attention Derivation. After extracting a collection of attention nodes (or phrases), we can further derive higher level concepts or topics from them, which automatically become their parent nodes in our Attention Ontology.

On one hand, we derive higher-level concepts by applying Common Suffix Discovery (CSD) to extracted concepts. We perform word segmentation over all concept phrases, and find out the high-frequency suffix words or phrases. If the suffixes forms a noun phrase, we add it as a new concept node. For example, the concept “animated film (动画—电影)” can be derived from “famous animated film (著名的—动画—电影)”, “award-winning animated film (获奖的—动画—电影)” and “Hayao Miyazaki animated film (宫崎骏—动画—电影)”, as they share the common suffix “animated film (动画—电影)”

On the other hand, we drive high-level topics by applying Common Pattern Discovery (CPD) to extracted events. We perform word segmentation, named entity recognition and part-of-speech tagging over the event phrases. Then we find out high-frequency event patterns and recognize the different elements in the events. If the elements (e.g., entities or locations of events) have isA relationship with one or multiple common concepts, we replace the different elements by the most fine-grained common concept ancestor in the ontology. For example, we can derive a topic “Singer will have a concert (歌手—开—演唱会)” from “Jay Chou will have a concert (周杰伦—开—演唱会)” and “Taylor Swift will have a concert (泰勒斯威夫特—开—演唱会)”, as the two phrases sharing the same pattern “XXX will have a concert (XXX—开—演唱会)”, and both “Jay Chou (周杰伦)” and “Taylor Swift (泰勒斯威夫特)” belong to the concept “Singer (歌手)”. To ensure that the derived topic phrases are user interests, we filter out phrases that have not been searched by a certain number of users.

The previous step produces a large set of nodes representing user attentions (or interests) in different granularities. Our goal is to construct a taxonomy based on these individual nodes to show their correlations. With edges between different user attentions, we can reason over it to infer a user’s real interest.

In this section, we describe our methods to link attention nodes and construct the complete ontology. We will construct the isA relationships, involve relationships, and correlate relationships between categories, extracted attention nodes and entities to construct a ontology. We exploit the following action-driven strategies to link different types of nodes.

Edges between Attentions and Categories. To identify the isA relationship between attention-category pairs, we utilize the co-occurrence of them shown in user search click logs. Given an attention phrase $p$ as the search query, suppose there are $n^{p}$ clicked documents of search query $p$ in the search click logs, and among them there are $n^{p}_{g}$ documents belong to category $\mathbf{g}$. We then estimate $\mathbb{P}(g|p)$ by $\mathbb{P}(g|p)=n^{p}_{g}/n^{p}$. We identify that there is an isA relationship between $p$ and $g$ if $\mathbb{P}(g|p)>\delta_{g}$ (we set $\delta_{g}=0.3$).

Edges between Attentions. To discover isA relationships, we utilize the same criteria when we perform attention derivation: we link two concepts by the isA relationship if one concept is the suffix of another concept, and we link two topic/event attentions by the isA relationship if they share the same pattern and there exists isA relationships between their non-overlapping tokens. Note that if a topic/event doesn’t contain an element of another topic/event phrase, it also indicates that they have isA relationship. For example, “Jay Chou will have a concert” has isA relationship with both “Singer will have a concert” and “have a concert”. For the involve relationship, we connect a concept to a topic if the concept is contained in the topic phrase.

Edges between Attentions and Entities. We extract: i) isA relationships between concepts and entities; ii) involve relationships between topics/events and entities, locations or triggers.

For concepts and entities, using strategies such as co-occurrence will introduce a lot of noises, as co-occurrence doesn’t always indicate an entity belongs to a concept. To solve this issue, we propose to train a concept-entity relationship classifier based on the concept and the entity’s context information in the clicked documents. Labeling a training dataset for this task requires a lot of human efforts. Instead of manual labeling, we propose a method to automatically construct a training dataset from user search click graphs. Figure 4 shows how we construct such a training dataset. We select the concept-entity pairs from search logs as positive examples if: i) the concept and the entity are two consecutive queries from one user, and ii) the entity is mentioned by a document which a user clicked after issuing the concept as a search query. Besides, we select entities belonging to the same higher-level concept or category, and insert them into random positions of the document to create negative examples of the dataset. For the classifier, we can train a classifier such as GBDT based on manual features, or fine-tune a pre-trained language model to incorporate semantic features and infer whether the context indicates a isA relationship between the concept-entity pair.

For events/topics and entities, we only recognize the important entities, triggers and locations in the event/topic, and connect them by an involve relationship edge. We first create an initial dataset by extracting all the entities, locations, and trigger words in the events/topic based on a set of predefined trigger words and entities. Then the dataset is manually revised by workers to remove the unimportant elements. Based on this dataset, we reuse our GCTSP-Net and train it without ATSP-decoding to perform $4$-class (entity, location, trigger, other) node classification over the query-title interaction graphs of the events/topics. In this way, we can recognize the different elements of an event or topic, and construct involve edges between them.

Edges between Entities. Finally, we construct the correlate relationship between entity pairs by the co-occurrence information in user queries and documents. We utilize high frequency co-occurring entity pairs in queries and documents as positive entity pairs, and perform negative sampling to create negative entity pairs. After automatically created a dataset from search click logs and web documents, we learn the embedding vectors of entities with Hinge loss, so that the Euclidean distance between two correlated entities will be small. After learned the embedding vectors of different entities, we classify a pair of entities as correlated if their Euclidean distance is smaller than a threshold.

Note that the same approach for correlate relationship discovery can be applied to other type of nodes such as concepts. Currently, we only constructed such relationships between entities.

Table 1 shows the statistics of different nodes in the attention ontology. We extract attention phrases from daily user search click logs. Therefore, the scale of our ontology keeps growing. Currently, our ontology contains $1,206$ predefined categories, $460,652$ concepts, $12,679$ topics, $86,253$ events and $1,980,841$ entities. We are able to extract around $27,000$ concepts and $400$ events every day, and around $11,000$ concepts and $120$ events are new. For online concept and event tagging, our system processes $350$ documents per second.

Table 2 shows the statistics and accuracies of different types of edges (relationships) in the attention ontology. Currently, we have constructed more than $490K$ isA relationships, $1,080K$ correlate relationships, and $160K$ involve relationships between different nodes. The human evaluation performed by three managers in Tencent shows the accuracies of the three relationship types are above $95\%$, $95\%$ and $99\%$, respectively.

To give intuition into what kind of attention phrases can be derived from search click graphs, Table 3 and Table 4 show a few typical examples of concepts and events (translated from Chinese), and some topics, categories, and entities that share isA relationship with them. By fully exploiting the information of user actions contained in search click graphs, we transform user actions to user attentions, and extract concepts such as “Actors who committed suicide (自杀的演员)”. Besides, we can also extract events or higher level topics of users’ interests, such as “Taylor Swift and Katy Perry” if a user often inputs related queries. Based on the connections between entities, concepts, events and topics, we can also infer what a user really cares and improve the performance of recommendation.

We evaluate our GCTSP-Net on multiple tasks by comparing it to a variety of baseline approaches.

Datasets. To the best of our knowledge, there is no publicly available dataset suitable for the task of heterogeneous phrase mining from user queries and search click graphs. To construct the user attention ontology, we create two large-scale datasets for concept mining and event mining using the approaches described in Sec. 3: the Concept Mining Dataset (CMD) and the Event Mining Dataset (EMD). These two datasets contain $10,000$ examples and $10,668$ examples respectively. Each example is composed of a set of correlated queries and top clicked document titles from real-world query logs, together with a manually labeled gold phrase (concept or event). For the event mining dataset, it further contains triggers, key entities and locations of each event. We use the earliest article publication time as the time of each event example. The datasets are labeled by $3$ professional product managers in Tencent and $3$ graduate students. For each dataset, we utilize $80\%$ as training set, $10\%$ as development set, and $10\%$ as test set. The datasets will be published for research purposes ²²2https://github.com/BangLiu/GIANT.

Methodology and Models for Comparison. We compare our GCTSP-Net with the following baseline methods on the concept mining task:

• •

  TextRank. A classical graph-based keyword extraction model (Mihalcea and Tarau, 2004).³³3https://github.com/letiantian/TextRank4ZH

• •

  AutoPhrase. A state-of-the-art phrase mining algorithm that extracts quality phrases based on POS-guided segmentation and knowledge base (Shang et al., 2018).⁴⁴4https://github.com/shangjingbo1226/AutoPhrase

• •

  Match. Extract concepts from queries and titles by matching using patterns from bootstrapping (Liu et al., 2019).

• •

  Align. Extract concepts by the query-title alignment strategy described in Sec. 3.1.

• •

  MatchAlign. Extract concepts by both pattern matching and query-title alignment strategy.

• •

  LSTM-CRF-Q. Apply LSTM-CRF (Huang et al., 2015) to input query.

• •

  LSTM-CRF-T. Apply LSTM-CRF (Huang et al., 2015) to titles.

For the TextRank and AutoPhrase algorithm, we extract the top 5 keywords or phrases from queries and titles, and concatenate them in the same order with the query/title to get the extracted phrase. For MatchAlign, we select the most frequent result if multiple phrases are extracted. For LSTM-CRF-Q/LSTM-CRF-T, it consists of a 200-dimensional word embedding layer initialized with the word vectors proposed in (Song et al., 2018), a BiLSTM layer with hidden size 25 for each direction, and a Conditional Random Field (CRF) layer which predicts whether each word belongs to the output phrase by Beginning-Inside–Outside (BIO) tags.

For event mining task, we compare with TextRank and LSTM-CRF. In addition, we also compare with:

• •

  TextSummary (Mihalcea and Tarau, 2004). An encoder-decoder model with attention mechanism for text summarization.⁵⁵5https://github.com/dongjun-Lee/text-summarization-tensorflow

• •

  CoverRank. Rank queries and subtitles by counting the covered nonstop query words, as described in 3.1.

For TextRank, we use the top $2$ queries and top $2$ selected subtitles given by CoverRank, and perform re-ranking. For TextSummary, we use the 200-dimensional word embeddings in (Song et al., 2018), two-layer BiLSTM (256 hidden size for each direction) as encoder, and one layer LSTM with 512 hidden size and attention mechanism as decoder (beam size for decoding is 10). We feed the concatenation of queries and titles into TextSummary to generate the output. For LSTM-CRF, the LSTM layer in it is configured similarly to the encoder of TextSummary. We feed each title individually into it to get different outputs, filter the outputs by length (number of characters between 6 and 20), and finally select the phrase which belongs to the top clicked title.

For event key elements recognition (key entities, trigger, location), it is a 4-class classification task over each word. We compare our model with LSTM and LSTM-CRF. The difference between LSTM and LSTM-CRF is that LSTM replaces the CRF layer in LSTM-CRF with a softmax layer.

For each baseline, we individually tune the hyper-parameters in order to achieve its best performance. As to our approach (GCTSP-Net), we stack $5$-layer R-GCN with hidden size $32$ and number of bases $B=5$ in basis decomposition for graph encoding and node classification. We will open-source our code together with the datasets for research purposes.

Metrics. We use Exact Match (EM), F1 and coverage rate (COV) to evaluate the performance of phrase mining tasks. The exact match score is 1 if the prediction is exactly the same as ground-truth or 0 otherwise. F1 measures the portion of overlap tokens between the predicted phrase and the ground-truth phrase (Rajpurkar et al., 2016). The coverage rate measures the percentage of non-empty predictions of each approach. For the event key elements recognition task, we evaluate by the F1-macro, F1-micro, and F1-weighted metrics.

Evaluation results and analysis. Table 5, Table 6, and Table 7 compare our model with different baselines on the CMD and EMD datasets for concept mining, event mining, and event key elements recognition. We can see that our unified model for different tasks significantly outperforms all baseline approaches. The outstanding performance can be attribute to: first, our query-title interaction graph efficiently encodes the information of word adjacency, word features, dependencies, query-title overlapping and so on in a structured manner, which are critical for both attention phrase mining tasks and event key elements recognition tasks. Second, the multi-layer R-GCN encoder in our model can learn from both the node features and the multi-resolution structural patterns from query-title interaction graph. Therefore, combining query-title interaction graph with R-GCN encoder, we can achieve great performance in different node classification tasks. Furthermore, the unsupervised ATSP-decoding sorts the extracted tokens to form an ordered phrase efficiently. In contrast, heuristic-based approaches and LSTM-based approaches are not robust to the noises in dataset, and cannot capture the structural information in queries and titles. In addition, existing keyphrase extraction methods such as TextRank (Mihalcea and Tarau, 2004) and AutoPhrase (Shang et al., 2018) are better suited for extracting key words or phrases from long documents, lacking the ability to give satisfactory performance in our tasks.

Story Tree Formation. We apply the story tree formation algorithm described in Sec. 4 to real-world events to test its performance. Figure 5 gives an example to illustrate what we can obtain through our approach. In the example, each node is an event, together with the documents that can be tagged by this event. We can see that our method can successfully cluster events related to “China-US Trade”, ordering the events by the published time of the articles, and show the evolving structure of coherent events. For example, the branch consists of events $3\sim 6$ are mainly about “Sino-US trade war is emerging”, the branch $8\sim 10$ are resolving around “US agrees limited trade deal with China”, and $11\sim 14$ are about “Impact of Sino-US trade war felt in both countries”. By clustering and organizing events and related documents in such a tree structure, we can track the development of different stories (clusters of coherent events), reduce information redundancy, and improve document recommendation by recommending users the follow-up events they are interested in.

Document Tagging. For document tagging, our system currently processes around $1,525,682$ documents per day, where about $35\%$ of them can be tagged with at least one concept, and $4\%$ can be tagged with an event. We perform human evaluation by sampling $500$ documents for each major category (“game”, “technology”, “car”, and “entertainment”). The result shows that the precision of concept tagging for documents is $88\%$ for “game” category, $91\%$ for “technology”, $90\%$ for “car”, and $87\%$ for “entertainment”. The overall precision for documents of all categories is $88\%$. As to event tagging on documents, the overall precision is $96\%$.

We evaluate the effect of our attention ontology on recommendations by analyzing its performance in the news feeds stream of Tencent QQ Browser which has more than $110$ million daily active users. In the application, both users and articles are tagged with categories, topics, concepts, events or entities from the attention ontology, as shown in Figure 2. The application recommends news articles to users based on a variety of strategies, such as content-based recommendation, collaborative filtering and so on. For the content-based recommendation, it matches users with articles through the common tags they share.

We analyze the Click-Through Rate (CTR) given by different tags from July 16, 2019 to August 15, 2019 to evaluate their effects. Click-through rate is the ratio of the number of users who click on a recommended link to the number of total users who received the recommendation. Figure 6 compares the CTR when we recommend documents to users with different strategies. Traditional recommender systems only utilize the category tags and entity tags. We can see that including more types of attention tags (topics, events, or concepts) in recommendation can constantly help improving the CTR on different days: the average CTR improved from 12.47% to 13.02%. The reason is that the extracted concepts, events or topics can depict user interests with suitable granularity and help to solve the inaccurate recommendation and monotonous recommendation problems.

We further analyze the effects of each type of tags in recommendation. Figure 7 shows the CTR of the recommendations given by different types of tags. The average CTR for topic, event, entity, concept and category are 16.18%, 14.78%, 12.93%, 11.82%, and 9.04%, respectively. The CTR of both topic tags and event tags are much higher than that of category and entities, which shows the effectiveness of our constructed ontology. For events, the events happening on each days dramatically different with each other, and they are not always attractive to users. Therefore, the CTR of event tags is less stable than the topic tags. For concept tags, they are generalization of fine-grained entities and has isA relationship with them. As there may have noises when we perform user interest inference using the relationships between entities and concepts, the CTR of concepts are slightly lower than entities. However, compared to entities, our experience show that concepts can significantly increase the diversity of recommendation and are helpful in solving the problem of monotonous recommendation.