Word Embedding-based Text Processing for Comprehensive Summarization and Distinct Information Extraction

The flowchart of the proposed framework for review clustering and summarization is shown in Fig. 1. It consists of two major parts: the first one creates a similarity network graph joining different sentences collected from the review dataset, while the second part, assigns textual tags for each clustered community. For the first part, the input is constituted by the reviews that we split into independent sentences denoted by $S_{i,j}$ where the $i$ is the review index and $j$ is the index of the sentence in review $i$. Each sentence will be represented by a node in the graph. The edges connecting two nodes represent the similarity between the corresponding sentences. In order to calculate the similarity score denoted by $\sigma_{j,j^{\prime}}$, we map the sentences to the vector space using word-embedding algorithm and compute the cosine similarity value as the distance between two sentences. Instead of using traditional Word2Vec [22] or GloVe [17] models, we use BERT to represent each word by a vector having 768 real numbers in $[-1,1]$ as elements. Their values depend on both the context of the sentence and the word itself. So, different vector representations are given for the same word when they are under different contexts. An example is provided in Fig. 2. We notice that the word “bank” has three different vector representations as the contexts where it is used are different. However, the cosine similarity score is high when the meanings of the word are similar.

Consequently, if two sentences have $N$ words then, both of them will be represented with a vector space of length $768N$. Hence, the cosine similarity score of the two sentences having the same dimension can be computed. Otherwise, if the other sentence has $M$ words, then a sliding window browsing the longest sentence is applied to compare phrases with the same number of words. Hence, $|N-M|$ comparisons are made and the highest obtained score will represent the similarity between those two sentences as shown in Fig. 3, where $\sigma_{1,2}=0.764$ represents the similarity between sentence $1$ and sentence $2$.

The next step is to create a network graph modeling the similarities between sentences. The vertices of the graphs are the phrases/sentences and the edges connecting two vertices indicate a certain similarity between them. Note that we set a certain threshold for the similarities, the edge exists only if the similarity is larger than the threshold. With the similarity network, our objective is to cluster the sentences into different “topics” and assign to them tags by selecting the most meaningful sentences. The clustering is based on the Louvain method designed by Blondel, which is a greedy optimization method that rapidly extract communities from large networks [23]. In this clustering problem, the objective function to maximize is a modularity metric defined as follows:

where $k_{j}$ and $k_{j^{\prime}}$ are the sum of the weights of the edges attached to nodes $j$ and $j^{\prime}$, respectively, $m$ is the sum of the weights in the graph, $\delta$ is Kronecker delta function with binary value, and $c_{j}$ and $c_{j^{\prime}}$ are the communities of the nodes.

Afterwards, we assign another correlation score, denoted by $C_{j}$, to each sentence in the graph that reflects its similarity with the rest of the phrases (i.e., nodes of the graph) using the TextRank algorithm, which counts the number and quality of links to a sentence to determine how important the corresponding node is. Consequently, the sentences with the highest correlation score is the sentence that is most similar to others and will be used to tag each detected community.

In this section, we illustrate and evaluate the output of our proposed framework applied on the case of restaurant reviews. We randomly select a restaurant located in Manhattan, NYC, having 1586 reviews in Google website as shown in Fig. 4. From this dataset, we pick the recent one hundred reviews, and feed them into our framework pipeline described earlier. We then split each review into sentences. In case of short phrases, we combine them with previous sentences to avoid having inaccurate results when computing similarities. In Fig 5, and for tractability, we provide a graph representing the connections between 568 sentences composing the 100 reviews. The isolated nodes (sentences) are not illustrated. We then highlight the different detected communities colored differently as well as their corresponding tags. Nine independent communities are obtained with the Louvain algorithm, each one is tagged by the sentence having the highest correlation score. Most of the communities show positive comments since the rating of the restaurant is 4.5, in other words, the positive side of the restaurant is mostly being discussed in the reviews.

To tackle this problem, we propose the second information extraction framework presented in Fig. 6. It is composed of two major parts: the first part is dedicated to collect answers for the questions that we formulated according to the context of the product/service. We apply the “BertForQuestionAnswering” question-answering model trained using the SQUAD dataset [24]. The model is batch trained for two epochs using in total 100,000 questions where each batch consists of eight questions. It is shown that the model achieves a matching score of 80.1% and a F1 score of 83.1% that are very close to human performance which are 86.8 and 89.5, respectively. Note that the F1 score measures the average overlap between the prediction and ground truth answer¹¹1In our future study, we will explore BERT and ALBERT ensemble models which are expected to achieve better performance over human but require large computational resources.. The question-answering model provided by BERT is not valid for multi-responses questions. Therefore, to overcome this issue, we proceed by formulating new different questions having the same meaning of the original questions to get all possible answers. The first part outputs, for each question, is a set of answers.

In the second part, we adopt the framework described in Section II so that, for each original question, all the possible answers are collected together and clustered into communities. Then, tags are assigned to them according to their correlation scores. Hence, for each original question, we determine a number of distinct answers corresponding to the number of the detected communities. Optionally, the framework can be used to answer human entered questions by returning the most relevant answers.

We employ the same restaurant reviews data from which we pick the most recent one thousand reviews. In Table I, we present some examples of answers extracted from a one review text by applying the question-answering model. The latter is applied on the first review text (106 words) given in Fig 4.

From Table I, we can notice that with the intentional selected questions, we are able to extract most of the required information from a single review including the delicious dish, the comments about the price, and the quality of service in the restaurant. In addition to the original question: “What is the delicious food to order in the restaurant?”, we use six other similar questions (the first six questions given in the TABLE I) to extract all the possible answers to the original question. Note that we need to remove the redundant answers, e.g, “The appetizers”.

In Fig 7, we present all the possible answers from one thousand reviews corresponding to this original question. After filtering and clustering the results, we obtain 25 communities with different tags representing 25 different menus items recommended by reviewers. The answer “Noodles with pork and crab” has the highest correlation score and has been recommended by the highest number of reviewers as it is reflected by the community size.

Finally, we compare our results with the “Ask a question” service provided by Google shown in Fig 8. For the same original question, the service provides only ten reviews where seven of them does not provide any useful details. Hence, customers can hardly get comprehensive information or an exhaustive list about their requests. The information provided by the proposed framework are more specific and directed towards the customers need which can ease their purchase decisions.