InsightNet : Structured Insight Mining from Customer Feedback

The InsightNet architecture (Figure 1) is based on decomposed prompting Khot et al. (2023), allowing to solve the complex task of extracting actionable verbatims and assigning a topic and a polarity to each verbatim. It consists of three phases of prompting, one for topic generation, one for polarity generation, and one for verbatim extraction.

We propose a bottom-up method to generate a hierarchical auto-taxonomy from reviews with weak supervision. This means we start with identifying Granular Topics from the reviews, then group them into broader (high-level) topics. This helps us preserve structure and create hierarchy for the output. We segment raw reviews (refer to Appendix section B.1 for exact segmentation steps) and assign a polarity to each segment (see equation 4). We discard segments with neutral polarity. The following steps illustrate the process of Taxonomy creation: captured in 1. Clustering review segments: We cluster the positive and negative segments separately using Fast clustering²²2https://github.com/UKPLab/sentence-transformers/blob/master/examples/applications/clustering/fast_clustering.py, a sentence transformer Reimers and Gurevych (2019) method. It uses cosine similarity to cluster sentence embeddings based on a threshold value. We obtain clusters for each sentiment class, representing different aspects or topics that the reviewers mentioned in their feedback.

2. Merging similar clusters & Cluster naming: Human annotators merge duplicate clusters and name each cluster using pre-defined taxonomy guidelines. They name each cluster with fine-grained topic names that reflect the main idea of that cluster, forming the Granular Topics (Level-3) of the taxonomy.

3. Creating hierarchy: To structure the taxonomy, we group similar Granular Topics into Hinge Topics (Level 2) and Coarse Topics (Level 1), resulting in a multi-level hierarchical taxonomy.

4. Keyword generation: To get an exclusive and exhaustive set of keywords, we refined the clusters (of segments) obtained in step 2 (above). We applied the semantic similarity function (equation 5) to perform Intra-cluster (refer section 4.2.1) cleaning to remove redundant and semantically duplicated keywords and Inter-cluster (refer section 4.2.2) cleaning of keywords, to eliminate ambiguous and overlapping keywords.

SegmentNet is a semantic matching algorithm that generates high quality training data with minimal training. It extracts insights from reviews using the taxonomy alone. It assumes that insights are often in short phrases within a review. It produces insights at a segment level and then aggregates them at review level. This involves 3 major steps:

1.Segmentation: We use language syntax heuristics to split a review into segments. We observe that a segment typically has one sentiment and at most one topic.

2. Sentiment classification: We train a BERT-based model with two linear heads (one for +ve and one for -ve) to get the sentiment of segment

We use $\delta_{p}=0.7$ as the classification threshold. A segment is neutral if $p<\delta_{p}$ and $n<\delta_{p}$, where $p$ and $n$ are the probabilities of a verbatim being positive and negative polarities respectively. We fine-tune sentiment classification model on $80k$ segments with almost equal data for each label, which has 99.1% accuracy when evaluated manually.
SimST: We formulate semantic similarity function SimST (equation 5) between two texts $text_{i}$ and $text_{j}$, where sbert computes the Sentence-BERT Reimers and Gurevych (2019) embedding of text.

where $\cos(\mathbf{u},\mathbf{v})=\frac{\mathbf{u}\cdot\mathbf{v}}{|\mathbf{u}||\mathbf{v}|}$ is the cosine similarity.

3. Topic matching: We devised heuristics based on the semantic matching function SimST (equation 5) and a signalling algorithm (see BTS Algorithm 4) to assign the best matching topic to a segment from a list of taxonomy topics. The signalling algorithm outputs the topic with the maximum similarity score and the value of that score among the given topic and similarity score pairs. Let $S$ denote a segment and $T$ its most relevant topic. We find $T$ from the list of taxonomy topics ($\tau^{\prime}$), $[T_{i}]_{i=1}^{N}$ with each topic $T_{i}$ has keywords $[k_{i,j}]_{j=1}^{M_{i}}$. We define three signals (using Algorithm 4), where the first signal (equation 6) is the semantically closest topic name and its score, the second signal (equation 7) is the topic with best mean score with the five closest keywords, and the last signal (equation 8) is the topic with the best mean score with all keywords.

To identify most relevant topic, we use heuristics on the three signals for topic matching:

(a.) High confidence match: if any of the three signal scores is high, match with high scoring topic ($score\geq\delta_{h}$). This matches a segment that is very similar to a topic or keyword,

(b.) Majority vote: If any two signals give the same topic, match with the common topic ($score\geq\delta_{m}$). Since each of the three signals is an independent weak predictor of the correct topic, the fact that any two signals agree on a topic is a strong indicator of correctness,

(c.) Best average score: Match with the topic with the best average score across all three signals $T_{avg}$  ($score\geq\delta_{a}$).

We present the topic matching algorithm (Algorithm 3) which is more robust to noisy keywords and identifies topics with higher precision than simple semantic matching.

During inference, we leverage syntactic and semantic matching to tackle topics generated that are out-of-taxonomy and re redundant. We either enrich taxonomy with these topics as fine-grained subtopics (L4 topics) or as novel topics (new L3 topics).

We show that SegmentNet can generate training data that is better or comparable to human annotated data. Figure 4 compares the performance of InsightNet trained with SegmentNet on different dataset sizes with a fixed human annotated dataset (fixed due to human bandwidth limitations). SegmentNet improves the performance by $9\%$ over human annotated data, given only $30$ samples/topic by manual annotation. This limitation is due to the heavy-tailed data (See Appendix section C.1) and the need for more data to cover the underrepresented topics. We also see that we need about three times more synthetic data to surpass the human-annotated baseline. We also show the model performance trained with k samples per topic. We find that InsightNet outperforms SegmentNet around 20 samples per topic and stabilizes around 100 samples per topic.

The choice of prompt can significantly affect the performance of language models like ours, especially in multi-task settings. We devised different variations of decomposed prompting for our multi-task problem of extracting actionable insights from customer reviews. We experimented with different orders of prompts for verbatim extraction ($V$), topic identification ($T_{c}$ and $T_{g}$), and polarity detection ($P$). We also explored different approaches for prompting for topic identification, either top-down (from coarse to granular) or bottom-up (from granular to coarse). We measured the performance of each variation using precision, recall, and F1-score metrics. We discovered that the optimal prompting strategy was to first prompt for the granular topics ($T_{g}$) from the review, then prompt for polarity ($P$) for each topic, and finally prompt for the verbatims ($V$) that correspond to the topics. This strategy achieved an F1-score of 0.80, which was considerably higher than the other variations. We also observed that using bottom-up prompting for topic identification was more efficient than using top-down prompting, as it minimized the errors in conditional prompting and enhanced the quality of topic extraction. We could deduce the coarse topics ($T_{c}$) from the granular topics ($T_{g}$) using the taxonomy. We refer to Level-1 and Level-2 topics as coarse topics and Level-3 topics as granular topics. We provide more detailed explanation of experiments in Appendix section A.2.

We applied unsupervised pre-training Li et al. (2021) to fine-tune a pre-trained model with unlabeled data from the target domain to enhance its transferability. We used the T5-base model Raffel et al. (2020) with review data (20M raw reviews) and the i.i.d. noise, replace spans objectives to do this. We named the resulting model AmaT5 and it showed better performance than the original. We also tried other variations, like T5-base, along with Sentence Shuffling (see section C.4), BART Lewis et al. (2019), FlanT5 Chung et al. (2022) and the results are shown in Table 2.

We conducted a comprehensive evaluation of our proposed methodology across a diverse set of 43 categories, encompassing over 2200+ distinct product types, which collectively represent more than 95% of the global volume of reviews. Our evaluation employed a dataset extracted from Amazon reviews1. To ensure a robust assessment, both the training and test datasets were meticulously stratified at a granular topic level. Specifically, we used around $75k$ reviews for training and $10k$ reviews for testing, thereby ensuring coverage across all product categories and granular topics.

To facilitate an equitable comparison across different approaches, we carried out post-processing procedures, as outlined in Section 4.4. Our findings reveal that our approach outperforms Aspect Clustering Sircar et al. (2022), a state-of-the-art method for topic extraction, in terms of coverage, diversity, and standardization. Specifically, our approach can generate over 1200+ unique topics that capture both positive and negative aspects of the reviews, while Aspect Clustering produces many redundant topics for the same level of coverage. Moreover, our approach ensures that the topics are consistent and coherent across reviews and product categories, with only 12% of them being duplicates that can be easily merged in post-processing. On the other hand, Aspect Clustering approach faces some challenges in reducing high duplication rate and no standardization, meaning that many topics are redundant and suffer with duplicate entries.

Table 1 shows the comparison of our results with the baselines. We also observed around 15% new topics have emerged which were not part of taxonomy (detailed analysis in Appendix section A.3).

In contrast to existing large language models like ChatGPT/OpenAI-GPT3 Brown et al. (2020), Llama2/Meta Touvron et al. (2023), Bard/Google-LaMDA Thoppilan et al. (2022), Falcon/TII Penedo et al. (2023), which fall short in extracting structured insights from customer reviews due to issues like generating redundant topics, domain-specificity, struggle to distinguish actionable vs non-actionable verbatims, and inefficiencies due large model size and inference latency. Additionally, they lack adherence to taxonomy topics as evidenced by our experiments (see Appendix section A.1 and Table 6 for results), where only 7% of reviews produced correct outputs, 11% reasonable outputs, and the majority, 82%, yielded random results.