Comparison-based Conversational Recommender System with Relative Bandit Feedback

In the above conversational recommender system, the agent is only allowed to ask the user absolute questions like “Are you interested in Nolan’s films?” We argue that it is usually unpractical, as the user may have very biased attitudes towards the key-terms, in which case the agent can only collect limited information. Comparative preferences in the meanwhile are potentially more informative and cheaper to collect. Therefore, we design the comparison-based conversation scenario. To conduct conversations, the agent queries relative questions over pairs of key-terms. For example, as illustrated in Figure 1, the agent may ask “Whose films do you prefer, Nolan or Michael Bay?”. It then receives the feedback inferring the user’s comparative preference. To simplify, we assume no abstention, making the feedback binary. Furthermore, we assume the feedback may be corrupted by Gaussian noise. Suppose the user has an internal valuation towards key-term $k$:

where $\epsilon_{k,t}\sim\mathcal{N}(0,\sigma_{g}^{2})$ denotes the random noise. The relative feedback for key-term $k_{1}$ and $k_{2}$ is:

Similarly to (Zhang et al., 2020), we model the conversation frequency by a function $b(\cdot)$. Specifically, if $b(t)-b(t-1)>0$, the agent will conduct

conversations with user at the $t$-th round of interaction. In such a manner, the agent will conduct $b(t)$ conversations with the user up to round $t$.

To simulate the long-run process of interaction with multiple users, we consider a sequence of $N$ iterations described in Algorithm 1. Assume there is an arbitrary distribution $p$ over users indicating the user frequency. At each iteration¹¹1It needs to clarify that ‘iteration’ considers interaction with multiple users, while ‘round’ refers to interaction with a specific user., an arbitrary user drawn randomly from $p$ will arrive. The agent, unknowing the user’s real feature, will then interact with the user as described in Line 1-1 of Algorithm 1. This framework extends the original contextual bandit scenario by considering one particular user may not arrive consecutively.

The overall objective for an agent in the comparison-based CRS is to minimize the cumulative regret induced by all interacted users in a long run:

where index set $\mathcal{U}_{N}$ denotes the users that have interacted with the agent for at least one round and $R_{u}$ denotes the cumulative regret induced by the historical interaction with user $u$ defined in Equation 2.

On the first attempt, we adopt an absolute model for relative questions inspired by (Christakopoulou et al., 2016). The insight behind the model is that relative feedback inferring the user’s comparative preference can be interpreted as two individual observations of key-term. For example, if the user shows more inclination to comedy movies than sci-fi ones, the agent will translate it into absolute information: 1) the user likes comedy movies; and 2) the user dislikes sci-fi movies. In this manner, we estimate the key-term level preference of user $u$ by:

where $(\tilde{\boldsymbol{x}},\tilde{r})$ stands for an absolute observation of key-term with $\tilde{\boldsymbol{x}}\in\mathcal{K}$ and $\tilde{r}\in\{0,1\}$, and $\mathcal{D}_{u}$ denotes the collected observation dataset of user $u$. Equation 6 has a closed-form solution $\tilde{\boldsymbol{\theta}}_{u}=\tilde{\mathbf{M}}_{u}^{-1}\tilde{\boldsymbol{b}}_{u}$, where

Based on the above model, we formulate two variants of the proposed algorithm Pos and Pos&Neg. They use the same mechanism to select relative questions. Suppose at $t$-th round of interaction with user $u$, the agent has access a subset of arms $\mathcal{A}_{t}$ and a subset of key-term $\mathcal{K}_{t}$. The contextual vectors for arms are denoted by $\mathbf{X}_{t}$, whose rows are composed of $\{\boldsymbol{x}_{a}^{\top}:a\in\mathcal{A}_{t}\}$. To collect the most informative observations, we expect to select key-terms that minimize the estimation error $\mathbb{E}\left[\left\|\mathbf{X}_{t}\boldsymbol{\theta}_{u}-\mathbf{X}_{t}\boldsymbol{\theta}_{u}^{*}\right\|_{2}^{2}\right]$. To select the optimal key-term to observe, we can approximately choose:

according to Theorem 2 in (Zhang et al., 2020). The essence behind Equation 7 is to greedily choose an key-term that could reduce the most uncertainty in the upcoming round of recommendation.

In this manner, we apply the following mechanism to choose the pair of key-terms: 1) we select a key-term $k_{1}$ with feature $\tilde{\boldsymbol{x}}_{k_{1}}$ according to Equation 7; 2) then, we conduct a pseudo update by updating the model with $\tilde{\boldsymbol{x}}_{k_{1}}$ according to Algorithm 3; 3) finally, we select another key-term $k_{2}$ according to Equation 7 with the updated model.

The only difference between two variants is how to interpret relative feedback to update the model or, equivalently, how to construct observation samples.

Suppose during the conversation at round $t$, user $u$ expresses that $k_{1}$ is preferred rather than $k_{2}$. The agent will therefore receive relative feedback $\tilde{r}_{k_{1},k_{2},t}=1$. Then the agent will update the model for user $u$ according to Algorithm 3. For Pos, it incorporates only positive information $(\tilde{\boldsymbol{x}}_{k_{1}},1)$. For Pos&Neg, the relative feedback is interpreted as two observations of absolute feedback: a positive observation $(\tilde{\boldsymbol{x}}_{k_{1}},1)$ for the preferred key-term and a negative observation $(\tilde{\boldsymbol{x}}_{k_{2}},0)$ for the less preferred key-term.

The above updating methods deal with relative feedback the same as with absolute feedback, which may not fully utilize the advantages of relative feedback. To improve, we consider training a linear classifier directly on paired data, and develop the updating method Difference.

We still apply the estimation of $\tilde{\boldsymbol{\theta}}_{u}$ according to Equation 6. But for Difference, we incorporate the relative feedback as a single observation $(\tilde{\boldsymbol{x}}_{k_{1}}-\tilde{\boldsymbol{x}}_{k_{2}},1)$. This is essentially training a linear classifier of feature difference using ridge regression. Note that the unknown parameters can not be fully identified by relative feedback, as $\boldsymbol{\theta}^{*}$ and $2\boldsymbol{\theta}^{*}$ induce the same comparison results for any pair $\tilde{\boldsymbol{x}}_{k_{1}},\tilde{\boldsymbol{x}}_{k_{2}}$ in expectation. Rather, what the linear classifier actually estimates is the normalized feature vector $\frac{\boldsymbol{\theta}^{*}}{\|\boldsymbol{\theta}^{*}\|_{2}}$. On the other hand, in the arm-level recommendation, the agent learns the norm of the feature $\|\boldsymbol{\theta}^{*}\|_{2}$. In this manner, RelativeConUCB learns users’ preferences effectively in the sense of both absolute correctness and relative relationship.

To design relative questions, we follow the maximum error reduction strategy which leads us to choose the pair based on the feature difference $\Delta\tilde{\boldsymbol{x}}_{k_{1},k_{2}}=\tilde{\boldsymbol{x}}_{k_{1}}-\tilde{\boldsymbol{x}}_{k_{2}}$:

The key-term selection strategy described above induces the standard Difference variant. However, since the number of key-terms can be extremely large, the strategy may not be practical as it has a quadratic time complexity. To eschew the problem, we further propose an optimized variant called Difference (fast). When selecting key-terms, it first selects a key-term $k_{1}$ by Equation 7. Then, with the first key-term fixed, it chooses another key-term $k_{2}$ based on Equation 8. In Section 4, we will compare these two variants, showing that Difference (fast) only suffers little decrease of performance.

The following algorithms are selected as the representative baselines to be compared with:

• •

  LinUCB (Li et al., 2010): A state-of-the-art non-conversational contextual bandit approach.

• •

  ConUCB (Zhang et al., 2020): A recently proposed conversational contextual bandit approach. When conducting conversations with user $u$, it chooses the key-term $k$ which maximizes error reduction, queries the user’s preference on it, and receives absolute feedback $\tilde{r}\sim{\rm Bernoulli}(\tilde{\boldsymbol{x}}_{k}^{\top}\boldsymbol{\theta}_{u}^{*})$.

For LinUCB, we set hyper-parameters $\lambda=1,\alpha=0.5$. For both ConUCB and RelativeConUCB, we set hyper-parameters $\lambda=0.5$, $\tilde{\lambda}=1$, $\sigma=0.05$, $\alpha=0.25$, $\tilde{\alpha}=0.25$ according to the default configuration in ConUCB’s public source code.

For all experiments, we sequentially simulate $N=400\times|\mathcal{U}|$ iterations. At each iteration $i$, the incoming user $u$ with $t=t_{u}$ rounds of historical interaction is chosen randomly from the user set $\mathcal{U}$. For item recommendation, the agent can select one out of 50 arms in the candidate set $\mathcal{A}_{t}$, which is randomly drawn from $\mathcal{A}$ without replacement. The key-term candidate set $\mathcal{K}_{t}$ contains those key-terms that are related to any arm in $\mathcal{A}_{t}$. Furthermore, we set the conversation frequency function $b(t)=5\lfloor\log(t)\rfloor$ which induces more conversations at the early stage to quickly capture preferences (Zhang et al., 2020). The reward for arms and relative feedback for key-terms are generated by Equation 1 and Equation 3 where $\sigma_{g}=0.1$. The distribution $p$ over users is set uniform, suggesting that each user will averagely interact with the agent for 400 rounds. The final results are reported by conducting 10 repeated runs.

We mainly use cumulative regret in Equation 5 to measure the performance of algorithms. As a widely used metric in the multi-armed bandit scenario, it captures the degree of satisfaction the user feels towards recommendation in a long run. Besides, in Section 4.2, we report averaged reward over iterations $A(N)\stackrel{{\scriptstyle\bigtriangleup}}{{=}}\frac{1}{N}\sum_{i=1}^{N}r_{i}$ to help clearly show the gap between different algorithms.

The designed experiments aim to answer the following evaluation questions:

• RQ1.

  Can relative feedback be incorporated well to help preference learning, and which variant of our algorithm is more effective?

• RQ2.

  Can we observe the benefit brought from feedback sharing?

• RQ3.

  Under which circumstances will relative feedback be more superior to absolute feedback?

We consider a setting of a user set $\mathcal{U}$ with $|\mathcal{U}|=200$, an arm set $\mathcal{A}$ with $|\mathcal{A}|=5,000$ and a key-term set $\mathcal{K}$ with $|\mathcal{K}|=500$. The relation matrix $\mathbf{W}\in\mathbb{R}^{|\mathcal{A}|\times|\mathcal{K}|}$ between arms and key-terms is constructed by the following procedures: 1) for each key-term $k$, we uniformly sample a subset of $n_{k}$ related arms $\mathcal{A}_{k}\subseteq\mathcal{A}$ without replacement, where $n_{k}$ is an integer drawn uniformly from $[1,10]$; 2) we assign $\mathbf{W}_{a,k}=1/n_{a}$, assuming arm $a$ is related to a subset of $n_{a}$ key-terms $\mathcal{K}_{a}$.

To generate the arms’ feature vector, we follow similar procedures in (Zhang et al., 2020): 1) we first sample a pseudo feature in $d-1$ dimension $\dot{\boldsymbol{x}}_{k}\sim\mathcal{N}\left(\mathbf{0},\sigma^{2}\mathbf{I}\right)$ for each $k\in\mathcal{K}$, where $d=50$ and $\sigma=1$; 2) for each arm $a$, its feature vector $\boldsymbol{x}_{a}$ is drawn from $\mathcal{N}\left(\sum_{k\in\mathcal{K}_{a}}\dot{\boldsymbol{x}}_{k}/n_{a},\sigma^{2}\mathbf{I}\right)$; 3) finally, feature vectors are normalized and added one more dimension with constant $1$: $\boldsymbol{x}_{a}\leftarrow(\frac{\boldsymbol{x}_{a}}{\|\boldsymbol{x}_{a}\|_{2}},1)$.

For the ground-truth feature of each user $u$, we first randomly sample a feature vector $\boldsymbol{\theta}_{u}^{*}\in\mathbb{R}^{d-1}$ from $\mathcal{N}\left(\mathbf{0},\sigma^{2}\mathbf{I}\right)$. Then we introduce two parameters: $c_{u}$ and $b_{u}$ They are drawn uniformly from $[0,0.5]$ and $[c_{u},1-c_{u}]$ respectively, controlling the variance and mean of expected reward. The final feature vector of user $u$ is $\boldsymbol{\theta}_{u}^{*}\leftarrow(c_{u}\frac{\boldsymbol{\theta}_{u}^{*}}{\|\boldsymbol{\theta}_{u}^{*}\|_{2}},b_{u})$. Note that the normalization step on both item features and user features aims to bound the expected reward $\boldsymbol{x}_{a}^{\top}\boldsymbol{\theta}_{u}^{*}\in[0,1]$, $\forall a\in\mathcal{A},u\in\mathcal{U}$.

The results are given in Figure 2, which plots the averaged reward over 80,000 iterations. We can first notice that all other algorithms outperform LinUCB, indicating the advantage of conducting conversations. The improvements are mainly reflected in the early stage, when the agent frequently converses with users.

Before we start, we randomly choose 100 users and formulate a binary matrix $H\in\{0,1\}^{100\times 2\text{k}}$ which stands for the interaction history. We derive feature vectors in $d-1$ ($d=50$) dimension for items by applying one-class matrix factorization (Chin et al., 2016) on $H$, and then normalize them by $\boldsymbol{x}_{a}\leftarrow(\frac{\boldsymbol{x}_{a}}{\|\boldsymbol{x}_{a}\|_{2}},1)$, $\forall a\in\mathcal{A}$. The constructed features will be used to conduct experiments on the records of the remaining 400 users.

For missing feedback of the remaining users, we again apply the matrix factorization technique to reconstruct the feedback matrix $F\in\{0,1\}^{400\times 2\text{k}}$. Next, in order to derive the ground truth user features, we use a simple autoencoder (Rumelhart et al., 1985) with four hidden layers and ReLU activation function. The autoencoder takes the user’s feedback (i.e., the corresponding row of $F$) as its input, encodes it into a latent representation $\boldsymbol{\theta}_{u}^{*}\in\mathbb{R}^{d-1}$, and normalizes it by $\boldsymbol{\theta}_{u}^{*}\leftarrow(\frac{\boldsymbol{\theta}_{u}^{*}}{\|\boldsymbol{\theta}_{u}^{*}\|_{2}},1)/2$. Finally, we reconstruct the feedback by $\boldsymbol{x}_{a}^{\top}\boldsymbol{\theta}_{u}^{*}$.

When training, we use MSE loss as the criterion and Adam with learning rate $3\times 10^{-4}$ as the optimizer. The training runs 200 epochs.

In this experiment, we compare the performance of algorithms both in the original setting and the feedback sharing setting. For ConUCB, it shares conversational absolute feedback to all users. For RelativeConUCB, relative feedback is shared instead. The experimental results on real-world dataset are shown in Figure 3.

In the drifted environment, we assume the users’ comparative preferences are inherently the same. To be specific, we model the feature vectors of user to be different only in the bias term. Following similar procedures in Section 4.3.1, we apply a four-layer autoencoder with two trainable embeddings $\boldsymbol{\theta}_{base}\in\mathbb{R}^{d-1}$ and $c\in[0,0.5]$ to construct drifted user features. $\boldsymbol{\theta}_{base}$ aims to extract the common preferences of users and $c$ controls the variance of expected reward. Taking the user’s feedback as the input, the autoencoder outputs a real number $\tau\in[0,1]$. The user feature is then constructed by $\boldsymbol{\theta}_{u}^{*}\leftarrow(c\frac{\boldsymbol{\theta}_{base}}{\|\boldsymbol{\theta}_{base}\|_{2}},b_{u})$, where $b_{u}=c+(1-c)\tau$. In this manner, the autoencoder constructs a drifted user group where preferences only differ in $b_{u}$.

When training on the autoencoder, we use MSE loss as the criterion and Adam with learning rate $3\times 10^{-3}$ as the optimizer. The training runs 1,000 epochs.

The results are shown in Figure 4.

We modify the procedures of user feature generation based on the synthetic experiment. For each user $u$, we consider the ground-truth feature is composed of three parts: users’ common feature $\boldsymbol{\theta}_{base}$, the user’s individual feature $\boldsymbol{\theta}_{u}$⁷⁷7It is a slight abuse of notion, as $\boldsymbol{\theta}_{u}$ usually stands for the estimated user feature., and the user’s bias $b_{u}$. $\boldsymbol{\theta}_{base}$ and $\boldsymbol{\theta}_{u}$ are both in $d-1$ dimension, and are drawn randomly from $\mathcal{N}\left(\mathbf{0},\sigma^{2}\mathbf{I}\right)$. We introduce two parameters $c\in[0,0.5]$ and $\beta\in[0,+\infty)$ to control the reward variance and user similarity. The final feature vector of user $u$ is $\boldsymbol{\theta}_{u}^{*}\leftarrow(c\frac{\boldsymbol{\theta}_{base}+\beta\boldsymbol{\theta}_{u}}{\|\boldsymbol{\theta}_{base}+\beta\boldsymbol{\theta}_{u}\|_{2}},b_{u})$, where $b_{u}$ is drawn uniformly from $[c,1-c]$. In this experiment we set $c=0.4$.

Figure 5 shows the outcome of the experiment, a comparison of the performances of algorithms under different user individual partitions (i.e., $\beta$). As the value of $\beta$ gets smaller, users share more similarity in preferences. Meanwhile, the scale of user bias over the user group remains the same.