CoMAC: Conversational Agent for Multi-Source Auxiliary Context with Sparse and Symmetric Latent Interactions

Deep neural network (DNN) based conversational systems have recently become prevalent due to superior scalability and applicability [2] compared to traditional systems. Neural networks can extract signals directly from end-to-end training driven by data without human intervention. With the ever-increasing learning capabilities of LLMs, especially pre-trained Transformer-based LLMs [16], DNN-based agents are being continually developed for new applications. Retrieval methods [5, 12] and Generative methods [18, 9] are two common categories of DNN-based conversational methods. Many document retrieval solutions [7] can also be easily adapted to solve conversational challenges.

Knowledge-based models [1, 3] are focused on better utilizing auxiliary facts to enhance the accuracy of responses. Knowledge and facts are crucial for many conversational tasks since providing accurate responses that avoid hallucinations and adhere to known facts is often a minimum requirement.

Persona-based models [12, 5, 17, 9] customize responses using auxiliary data that can represent the values, beliefs, characteristics, and/or history of a specific user. Models without persona information often fail to align responses with the user’s preferences, which can be problematic when personalization is an expectation [11]. Many pre-fusion-based methods [5] concatenate personas with input queries as long text inputs to the model. These approaches have been shown to be sub-optimal because they process heterogeneous data from multiple sources in a homogeneous way, which might ignore the critical source-specific signals and increase the learning complexity of the model. Liu et al. [12] recently proposed a retrieval model in a post-fusion framework that handles heterogeneous persona and query data in specialized encoding streams.

The above methods are limited to a single source of auxiliary data, while many real applications contain multiple auxiliary sources that can provide valuable perspectives of the conversation. Jang et al. [6] proposed a pre-fusion-based method, PK-FoCus, that identifies relevant persona and knowledge data based on a single highly summarized context embedding. This type of pre-fusion approach can ignore important signals in low-level word interactions. Consequently, it is unable to effectively extract useful information for response generation.

Liu et al. [10] used low-level word similarities in the PK-NCLI method, which leverages a variation of ColBERT [7] similarity in grounding networks. However, ColBERT is designed particularly for one-to-many document retrieval problems, where there is only one query and many documents. PK-NCLI failed to address critical issues of ColBERT when used for many-to-many context retrieval when it involves multiple sources, each with multiple entries. First, ColBERT has a bias based on query length. While this bias is harmless when only one query is involved, it can favor longer, but potentially irrelevant, context entries in multi-source, multi-entry scenarios. Second, it is an asymmetric metric that finds “one-way” relevance from the query to the documents. It ignores the critical reverse relevance from the documents to the queries when multiple auxiliary sources are involved. Third, it considers all word pairs equally, which makes it sensitive to frequent but mostly irrelevant words and suppresses subtle yet critical relevance among meaningful notional words.

The inputs, $U$, $P$, and $K$, are first embedded separately by $LM$ as $E_{U}\in{}^{1\times s_{u}\times d}$, $E_{P}\in{}^{N_{p}\times s_{p}\times d}$, and $E_{K}\in{}^{N_{k}\times s_{k}\times d}$, where $s_{u}$/$s_{p}$/$s_{k}$ are the padded sequence lengths (i.e., number of tokens) of the $U$/$P$/$K$ entries, and $d$ is the embedding size. Here, each source of data is encoded in a separate stream to preserve low-level authentic information of the corresponding source, instead of using a single highly summarized embedding for all sources as in [6].

Identifying the relevant auxiliary entries starts with a metric that evaluates the relevance of an entry (e.g., $P$) to the rest of the conversational context (e.g., $K$ and $U$). We first review the ColBERT similarity $\mathcal{S}\textsubscript{{C}}$ [7]. $\mathcal{S}\textsubscript{{C}}$ measures the token-level latent interaction similarity between a query, $x$, and a document, $y$, in document retrieval, computed as $\mathcal{S}\textsubscript{{C}}(x,y)=\sum_{x_{i}\in x}\max_{y_{j}\in y}E_{x_{i}}\cdot E_{y_{j}}^{T}$, where $E_{x_{i}}$/$E_{y_{j}}$ are the embeddings of the $i$-th/$j$-th token of $x$/$y$, respectively. $\mathcal{S}\textsubscript{{C}}$ can be generalized to any two texts, $x$ and $y$.

In our problem, the vanilla $\mathcal{S}\textsubscript{{C}}$ suffers from several issues mentioned in Section 2.3. Here, we develop a novel metric, denoted as $\mathcal{S}\textsubscript{{SSN}}$, that introduces normalization, symmetry and sparsity in ColBERT-style latent interactions.

Normalization To eliminate the dominating bias of longer but potentially less relevant $x$’s in a batch (e.g., longer vs. shorter paragraphs in the auxiliary knowledge base), $\mathcal{S}\textsubscript{{C}}$ is normalized by $|x|$, the number of tokens in $x$. That is,

Symmetry Both $\mathcal{S}\textsubscript{{C}}$ and $\mathcal{S}\textsubscript{{N}}$ are asymmetric metrics that exchange information from only $x$ to $y$. In a multi-source auxiliary setup, it is critical to allow information to flow bi-directionally among all sources. A simple yet effective approach is to combine the asymmetric similarities obtained in both directions, i.e., $\mathcal{S}\textsubscript{{SN}}(x,y)=\mathcal{S}\textsubscript{{N}}(x,y)+\mathcal{S}\textsubscript{{N}}(y,x)$.

Sparsity Another common issue of existing ColBERT-style similarities is that, when measuring pairwise token similarity, certain frequent tokens (such as function words) can dominate overall similarities, diminishing the subtle but critical signals from informative tokens (like notional words). To mitigate the potential adverse effects of such language noise, we introduce sparsity and selectivity into the similarity measure, using only a subset of tokens chosen through specific sampling strategies, denoted as $\mathcal{S}\textsubscript{{SSN}}(x,y)=\mathcal{S}\textsubscript{{SN}}(\hat{x},\hat{y})$, where $\hat{x}$/$\hat{y}$ are the tokens sampled from $x$/$y$, respectively, following the sampling strategies.

The main assumption of sparsity and selectivity is that frequent tokens are less informative than those appearing sparsely in the dataset. TF-IDF [15] is a common solution that fits the assumption. We propose a sampling strategy that selects tokens with highest TF-IDF weights for $\mathcal{S}\textsubscript{{SSN}}$, referred to as “CoMAC (TF-IDF)”. We denote the percentage of selected tokens as $\mathcal{P}_{sr}$. In Section 5.2, we also discuss an alternative sampling strategy which learns the token weights by a feed-forward network using $E_{U}$/$E_{P}$/$E_{K}$, referred to as “CoMAC (FF)”.

Figure 2 demonstrates the computation of $\mathcal{S}\textsubscript{{SSN}}$ similarity by using token-level similarities. Calculating a similarity matrix for a large set of text entries, $P$ and $K$, incurs an exponentially growing computational cost. To improve efficiency, we reduce $E_{U}/E_{P}/E_{K}\in{}^{d}$ to $E^{\prime}_{U},E^{\prime}_{P},E^{\prime}_{K}\in{}^{d_{0}}$ ($d_{0}=d/4$) by a dimension reduction layer prior to $\mathcal{S}\textsubscript{{SSN}}$.

In this section, we aim to identify two subsets of relevant persona and knowledge entries, $\hat{P}\in P$, and knowledge entries, $\hat{K}\in K$, that are helpful for response generation in the PG and KG networks with $\mathcal{S}\textsubscript{{SSN}}$.

Persona Grounding In PG, we consider $P$ as the “query” set and $U$/$K$ as two distinct sets of “documents.” We construct two similarity matrices for: (I) persona and utterance entry pairs ($S^{PU}$), and (ii) persona and knowledge entry pairs ($S^{PK}$). The elements in $S^{PU}$ and $S^{PK}$ are calculated as follows:

where $E^{\prime}_{\cdot_{i}}$/$E^{\prime}_{\cdot_{j}}$ is a reduced embedding of an entry token-by-token. We further calculate the similarities between each persona entry and the overall knowledge by taking the average entry-wise similarity, i.e., $\tilde{S}^{PK}=\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}S^{PK}_{i}$.

A post-fusion strategy scores each persona entry by fusing $S^{PU}$ and $\tilde{S}^{PK}$ in an attention layer with sigmoid activation, i.e., $\tilde{P}=\sigma(w_{1}\tilde{S}^{PK}+w_{2}S^{PU}+b)$, where $w_{1}$/$w_{2}$/$b$ are the attentions and bias, and $\tilde{P}\in{}^{N_{p}}$. Persona entries are considered relevant when $\hat{P}=P[\tilde{P}>0.5]$.

Knowledge Grounding KG is similarly constructed by calculating the relevance scores of each knowledge entry with the utterance and persona, $S^{KU}_{i,j}$ and $\tilde{S}^{KP}$, respectively, followed by the post-fusion strategy $\tilde{K}=\texttt{softmax}(w_{1}\tilde{S}^{KP}+w_{2}S^{KU}+b)$. The knowledge candidate with the highest $\tilde{K}$ score is considered relevant, i.e., $\hat{K}=K[\texttt{argmax}(\tilde{K})]$. Here softmax and argmax are used, following the dataset assumption [6] that a response is based on only one knowledge entry, while multiple persona entries could contribute to it.

With estimated relevant persona and knowledge entries ($\hat{P}$ and $\hat{K}$), a new input $U^{*}$ is formed by combining $\hat{P}$, $\hat{K}$ and the original input $U$ for language model $LM$, i.e., $E_{U^{*}}$=$LM(U^{*})$=$LM([\hat{K};\hat{P};U])$. A probability distribution $p_{r}$ over the vocabulary space is estimated based on $E_{U^{*}}$ by a projection layer.

For inference, $LM$ follows the auto-regressive framework [4] to generate a sequence of tokens as the response. For training, we train the model with loss function $L^{SSN}=\alpha L_{K}+\beta L^{*}_{P}+\gamma L_{M}$, where $L_{K}$ is the cross-entropy loss over $\tilde{K}$ for knowledge grounding, $L^{*}_{P}$ is the cross-entropy loss over $\tilde{P}$ for persona grounding, $L_{M}$ is the cross-entropy loss over $p_{r}$, and $\alpha$/$\beta$/$\gamma$ are the weighting hyper-parameters. The loss term $L^{*}_{P}$ addresses an extreme imbalance issue overlooked by the baselines [6, 10] that nearly 87% of persona entries are irrelevant. First, we apply weights $w^{*}$/$(1-w^{*})$ to positive/negative persona labels, respectively, to emphasize the importance of relevant persona entries. Second, we randomly discard $L^{*}_{P}$ with probability of $p^{*}$ if an utterance has all negative persona labels. However, $L_{K}$ and $L_{M}$ are not affected by the persona labels.

We train CoMAC models on the FoCus dataset [6], which is a conversational dataset powered by persona and knowledge. Each conversation includes five text-based persona entries describing the user’s personal profile, such as demographic data or hobbies (e.g., “I find heritage-listed buildings interesting”), and several knowledge entries extracted from a Wikipedia page about a landmark (e.g., “Thorps Building is a heritage-listed commercial building at Macrossan Street…”). For each round, zero or more persona entries are relevant to the conversation, while only one knowledge entry is relevant. Table 1 shows some basic statistics of dataset. For more detailed information and examples, please refer to [6].

We compare CoMAC with two baseline methods, PK-FoCus [6] and PK-NCLI [10]. The source code of CoMAC is available online ¹¹1https://github.com/jliu-v/CoMAC. All models are initialized with pre-trained BART [8] weights, trained on NVIDIA RTX2080-Ti GPUs over the training set for two epochs and evaluated on the validation set. For TF-IDF, the IDF weights are pre-computed offline for efficiency, and the TF and final TF-IDF weights are computed on the fly. It is worth noting that, theoretically, CoMAC could use any existing language model $LM$. GPT-2 [14] is sub-optimal compared to BART (encoder-decoder architecture) for all methods, as GPT-2’s decoder-only architecture is not suitable for the PG/KG tasks which require advanced capabilities on comprehension and summarization.

Hyper-parameters Besides the default setup of $\alpha/\beta/\gamma$=1/1/10 in [6], we conduct additional hyper-parameter search, limiting $\alpha$+$\beta$+$\gamma$=10 to avoid arbitrary optimization solutions. In our experiments, the optimal values of other hyper-parameters are $w^{*}$=$0.9$, $p^{*}$=$0.1$ and $\mathcal{P}_{sr}$=$0.35$. The $w^{*}$/$p^{*}$ values roughly match the persona label distribution in the dataset.

Evaluation Metrics We evaluate CoMAC in terms of both the quality of generated responses and the grounding accuracies. For response quality, we use the perplexity (PPL), ROUGE and BLEU, and F1 scores. For grounding, we use PG/KG accuracies. Due to the imbalanced persona label distribution, we also report F1, precision, and recall for PG (PG-F1/PG-PR/PG-RC). All reported metrics are based on the validation dataset. Lower PPL values indicate better performance, while the other metrics are the higher the better.

Asymmetric similarities like $\mathcal{S}\textsubscript{{C}}$ or $\mathcal{S}\textsubscript{{N}}$ learn one-way information flow from the “queries” (e.g., $P$ in PG network) to the “documents” (e.g., $U$/$K$ as the context in PG network). They are not suitable for the purpose of the grounding networks in conversational problems, where all sources of inputs (whether “query” or “documents”) will contribute to the final output.

To demonstrate that the supplemental two-way signals are critical among multiple auxiliary data, we compare the CoMAC method with the two-way $\mathcal{S}\textsubscript{{SN}}$ (no sparsity, denoted as CoMAC-$\mathcal{S}\textsubscript{{SN}}$) against the baseline PK-NCLI with the one-way $\mathcal{S}\textsubscript{{N}}$. Table 2 shows that CoMAC-$\mathcal{S}\textsubscript{{SN}}$ has significant improvements over the baseline PK-NCLI on PG accuracy (51.82% vs 44.75%), and provides minor improvements on KG accuracy (90.42% vs 90.25%) and other language generation metrics. For PG tasks with extremely imbalanced and sparse data, it is especially important to exploit symmetry and leverage other context data. The KG task benefits less from the symmetry, primarily because most contexts in the dataset are highly correlated with knowledge entries. Most questions are related to information from the knowledge, making them more similar to traditional document retrieval tasks. Therefore, the asymmetric $\mathcal{S}\textsubscript{{N}}$ already captured most of the useful signal, and the symmetric $\mathcal{S}\textsubscript{{SN}}$ only provides marginal improvement.

The baselines failed to reduce the extra noise brought into the grounding networks from less informative tokens. Such noise not only dilutes the overall information in texts but could also obscure the importance of meaningful words. We conduct two sets of CoMAC experiments with two different strategies (TF-IDF and FF) to sample tokens in the PG/KG networks.

Table 2 shows that both strategies can significantly improve the PG/KG and generation performance over the two baselines and CoMAC-$\mathcal{S}\textsubscript{{SN}}$. TF-IDF/FF improved PG over the best baseline PK-NCLI by 45.74%/62.46% (65.22/72.70 vs 44.75), PG-F1 by 27.41%/43.47% and KG by 6.76%/2.47%, respectively. They also improved the generation quality, e.g., F1 by 5.26%/3.51%. Table 3 shows an example of the tokens sampled by the CoMAC (TF-IDF). For efficiency, CoMAC outperforms the baseline by 9.77% (2872 vs 3183 sec.) in terms of inference time. In real applications, further efficiency and scalability can be achieved through pre-sampling of persona and knowledge fully offline.

An interesting observation is that TF-IDF is better for KG and generation performances, while FF is better for PG performance. This is probably because many conversations in the dataset are highly correlated with knowledge, and personas do not have as much influence as knowledge on the responses, as observed in [10]. The FF strategy is a fully trained layer directly driven by the $L^{*}_{P}$ loss of the PG network. In contrast, the token distribution and contribution to the PG training target is disconnected. Therefore, FF performs well on PG accuracy. However, this doesn’t make TF-IDF a sub-optimal choice overall, as it is directly tied to the token distributions in the dataset, which can preserve rare but informative tokens in the grounding stage for response generation later.

To evaluate the quality of generated responses, we randomly selected 75 conversations (408 rounds) from the validation set for human evaluation. We presented to human volunteers the original utterance, ground truth and predicted knowledge, as well as the persona entries, reference response and two predicted responses from CoMAC and PK-FoCus (denoted as $\mathcal{A}_{C}$ and $\mathcal{A}_{F}$, respectively). In order to compare the performance of CoMAC with the baseline, each volunteer was asked to answer three questions: Q1. do they prefer $\mathcal{A}_{C}$ or $\mathcal{A}_{F}$, Q2. did $\mathcal{A}_{C}$ successfully address the original question, and Q3. did $\mathcal{A}_{F}$ successfully address the original question. For Q1, 46.81% of $\mathcal{A}_{C}$ were rated as better than $\mathcal{A}_{F}$, 42.40% were similar to $\mathcal{A}_{F}$, and only 10.78% were worse than $\mathcal{A}_{F}$. For Q2 and Q3, 90.44% and 75.98% of $\mathcal{A}_{C}$ and $\mathcal{A}_{F}$, respectively, addressed the original question. These results demonstrate that CoMAC is better than PK-FoCus at addressing questions and producing higher quality responses.