High Quality Related Search Query Suggestions using Deep Reinforcement Learning

Variants of mono-lingual supervised $Seq2Seq_{NMT}$ models are used in industry applications for query suggestions [14]. In the pre-training step, we train the supervised $Seq2Seq_{NMT}$ model using co-occurring consecutive query pairs in a search sessions. A search-session (Cao et al., 2008) is a stream of queries entered by a user in a 5-min¹¹1Based on guidance from internal search team’s proprietary analysis. time window. N-1 Consecutive query pairs ($q_{i}$, $q_{i+1}$) are extracted from a search session consisting of a sequence of N queries ($q_{1}$, $q_{2}$,…,$q_{N}$). Consecutive queries could be unrelated, semantically and (or) syntactically related. Our model is weakly supervised as we use all query pairs and do not filter them using sparse click data, costly human-evaluations and weak association rules. Weak supervision allows the training process to scale, minimize selection-bias and we conjecture that it improves model generalization too. For example, unlike [14], we do not apply syntactic similarity heuristics to filter query pairs, as queries could be semantically related yet syntactically dissimilar (e.g., “artificial intelligence” and “machine learning”) . The $Seq2Seq_{NMT}$ encoder-decoder framework consists of a BiLSTM (Graves et al., 2013) encoder, that encodes a batch (batch-size $B$) of input queries ($q^{1:B}_{i}$) and the LSTM (Hochreiter and Schmidhuber, 1997) decoder generates a batched sequence of words y = ($y^{1:B}_{1}$,…, $y^{1:B}_{T}$). Where, $T$ is the sequence length. During training, we use teacher forcing (Williams and Zipser, 1989), i.e., use the co-occuring query ($q^{1:B}_{i+1}$) as input to the decoder. Context attention vector is obtained from the alignment model (Bahdanau et al., 2015) . Categorical cross entropy loss is minimized during training and hyper-parameters of the model are fine-tuned (see Section 3.2).

This section describes the reward estimation and Deep Reinforcement Learning (DRL) model training steps to fine-tune and improve the policy obtained via pre-trained supervised model.

From user search-query logs, we randomly sampled 0.61 million (90% train), 34k (5% valid) and 34k (5% test) query pairs to train the supervised $Seq2Seq_{NMT}$ and DRL models. Dataset size to train the naturalness-estimator model is 5x the aforementioned amount (See Section 2.2.3). Max-length of a query is 8 and mean-length is ~2 words. Vocabulary size is 32k and out of vocabulary words in validation and test sets are replaced with “¡UNK¿” unknown-token.

We implemented all models in Tensorflow (Abadi et al., 2015) and tuned the parameters using Ray[Tune] (Liaw et al., 2018) on Kubernetes (kubernetes.io, 2020) distributed cluster. As described in Section 2, the query suggestion policy is initialized with fine-tuned $Seq2Seq_{NMT}$ model. $Seq2Seq_{NMT}$ model parameters are updated using Adam (Kingma and Ba, 2015) optimizer and categorical-cross-entropy loss is minimized during training. During inference, six⁴⁴4In production environment six related queries are suggested for each user query. queries are generated per input query ($q_{i}$) using beam-search (Freitag and Al-Onaizan, 2017) decoding. Negative examples to train the two-layered BiLSTM contextual-naturalness-estimator are obtained from pre-trained $Seq2Seq_{NMT}$ model. At inference, naturalness probability ($D_{\phi}(y^{bk}_{1:T}$)) is obtained from the output of fully-connected layer with last time-step’s hidden state as its input.

The initial policy is fine-tuned using REINFORCE policy-gradient algorithm, using future-reward described in Section 2.2.2. During training, “K” samples for MC roll-out are generated using beam-search or from categorical distribution of inferred word probabilities at each time-step (See Figure 1). DRL model training stability is monitored using reward weighted Negative Log Likelihood convergence performance, with $\frac{1}{KB}\Sigma_{(k\epsilon K,b\epsilon B)}[-R_{D_{\phi}}(y^{bk}_{1:T})*log(G_{\theta}(y^{bk}_{1:T}))]$ as the computed loss at each model training step.

We use SGD optimizer (Kiefer and Wolfowitz, 1952) to update the weights of the agent (Equation 4). Appendix Figure 2 shows the convergence performance of the DRL model for different values of unnaturalness penalty ($\eta$), number of MC samples ($K$) and choice of sampling strategy. Complete set of hyper-parameters we tuned are in Appendix Table A. Best combination of hyper-parameters are chosen are based on performance on validation set (See Appendix Table A).

The binary “natural/unnatural” class prediction performance of the contextual naturalness estimator is evaluated using F1⁵⁵5F1 = $\frac{2*Precision*Recall}{Precision+Recall}$ score and Accuracy⁶⁶6Categorical accuracy: calculates how often predictions match one-hot labels. metrics. We use the mean of the following metrics calculated on the test set, to evaluate the relevance, engagement, accuracy and diversity of generated queries.

• •

  Sessions with positive user-action ($Sessions^{+}@6$): Long-term binary engagement metric indicating if recommended queries lead to a successful session. Its value is “1”, if any of the six generated queries belong to a search-session in test-data with an associated down-stream positive user action (Section 2.2.2).

• •

  Unique@6: Diversity metric indicating the percentage of unique sentences in (six) query suggestions made per query $q^{test}_{i}$. Queries containing unknown word token (“¡UNK¿”) are filtered out as only high-quality suggestions are presented to the end user.

• •

  Precision@6: Measures relevance with respect to the query a user would enter next. Is “1” if ($q^{test}_{i+1}$) is in the set of six query suggestions made for ($q^{test}_{i}$) and “0” otherwise.

• •

  Word-repetitions per sentence ($Repetitions_{S}$): Fraction of word repetitions per generated query ($S$). Unwanted word repetitions lead to lower quality.

• •

  Prior Sentence Probability ($P_{S}$): $P_{S}=\Sigma_{w_{i}\epsilon S}log(p_{w_{i}})$, measures the prior sentence probability. $p_{w_{i}}$ is prior word probability defined in Section 2.2.3. Lower sentence probability indicates higher diversity as generated queries contain less frequent words.

The contextual-naturalness-estimator achieved $90\%$ accuracy and $80\%$ F1 performance on test set. Table 1 shows the performance of supervised ($Seq2Seq_{NMT}$) and proposed DRL model on the five metrics mentioned in previous section. $DRL_{beam}$ and $DRL_{sampling}$ use “beam-search” and “sampling from categorical distribution” MC sampling strategies respectively (see Section 3.2). In order assess the impact of applying heuristics to filter and improve quality of suggestions provided by supervised models, we analyzed the performance of $Seq2Seq^{-}_{NMT}$, which is $Seq2Seq_{NMT}$ model with post-processing filters to remove suggestions with repeated words.

On offline test data set, in comparison to the baseline $Seq2Seq_{NMT}$ model, $Seq2Seq^{-}_{NMT}$ removed query suggestions with repeated words completely, however the heuristics-based model performed poorly in-terms of diversity (4.5% relative drop in mean Unique@6) and average number of successful sessions (0.6% relative drop in mean $Sessions^{+}@6$). On the other hand, both versions of our proposed DRL models outperformed the baseline model on all metrics. DRL variants achieved significant relative improvement in-terms of user-engagement (mean $Session^{+}@6$) up to 4.2% ($0.1108\rightarrow 0.1155$), query suggestions’ diversity (mean Unique@6) up to 3% ($5.8244\rightarrow 5.9956$), sentence-level diversity (mean Prior Sentence Probability) up to 0.7% ($-6.4442\rightarrow-6.4897$) and reduction in errors per sentence up to 82% ($2.21\rightarrow 0.40$). Non significant improvement in relevance (mean Precision@6) is not surprising as the supervised $Seq2Seq_{NMT}$ model is also trained with consecutive query pairs.