RefuteBench: Evaluating Refuting Instruction-Following for Large Language Models

For knowledge refuting, we construct our benchmark based on RIPPLEEDITS Cohen et al. (2023) to evaluate the model’s flexibility with knowledge refuting. Within the dataset, each factual knowledge is edited to be counterfactual and there are related facts that can be logically derived from the edit. We use GPT-4 API to convert the statements of knowledge into the format of questions and answers. In the single feedback setting, we use the converted question as our initial query and counterfactual as the feedback. The generalization query is a sampled ripple effect question, which is of logical generalization, compositionality, and subject aliasing to the initial query. For the multi-feedback setting, we first adopt two counterfactual queries and then interleave their corresponding ripple questions to mimic the scenario that a user repeatedly asks questions about some topic.

An example of QA refuting is shown in Figure 1. We first ask the LLMs for the citizenship of Leonardo Dicarprio, and after receiving the response, we give feedback to the LLMs with counterfactual knowledge that the citizenship of Leonardo is Syria. Then a related question asking the currency in the country of citizenship of Leonardo is fed to the LLM to analyze whether the model can respond to the feedback.

For machine translation, our data is sourced from WMT2023 GeneralMT tasks. We select two language directions, English to Chinese (high resource) and English to Hebrew (low resource). To collect the candidate feedback for lexical usage, we build the bilingual dictionaries and monolingual vocabularies from the Open Multilingual Wordnet Bond and Paik (2012). We tokenize each source sentence with spacy²²2https://spacy.io/ and utilize our bilingual dictionary to match each token. If a match is found, we collect the corresponding candidate translations in the target language. To simulate the real-world application of translation in a specific domain, we only keep feedback on source tokens whose part-of-speech (POS) tag is NOUN. In case LLMs might generate the same translation as candidate feedback, we only consider source tokens with more than one candidate translation. As a more challenging scenario, we also generate a random translation in the target language sampled from the monolingual vocabulary for each feedback.

For the single-feedback setting, we use each of the source sentences as the initial query and randomly select one candidate word translation as the feedback. The generalization query is another source sentence that is from the same document as the initial query and contains the same source word of feedback. For the multi-feedback setting, we use sentences from the same document to construct a dialogue. For each sentence, we provide refuting instructions with all candidate translations that have not appeared in the context.

The third task we consider is email writing. Such a writing task is representative of the day-to-day usage of LLMs. A user asks the LLM to write an email for them and provides feedback when they are not satisfied. We collect data from four existing instruction tuning Ouyang et al. (2022a) datasets, including MTBench Zheng et al. (2023b), alpaca-cleaned Taori et al. (2023), LIMA Zhou et al. (2023a), and alpaca-eval Dubois et al. (2023); Li et al. (2023d). We first use the keyword “email” to filter instructions in these datasets to roughly collect the related instructions for writing emails. Then, the authors manually check the filtered instructions, and remove those instructions that duplicate, are not email writing related, or contain insufficient information, e.g., "Please help me write a business email.". After that, we divide the dataset into four domains, including work-related, school-related, friends, and family. For email writing, we consider five types of verifiable feedback, inspired by Zhou et al. (2023b), which are shown in Appendix A.3. As shown, each feedback is constrained in its corresponding domain.

For the single feedback setting, we use the email instruction as our initial query, and verifiable feedback discussed above. The generalization query is another random instruction from the same domain, to see whether LLMs can generalize feedback. For the multi-feedback setting, we random a sequence of four instructions from the same domain and feedback with each of our feedback types. To ensure verifiability, we remove ‘response language’ in this process, which brings complexity to checking other feedback instructions.

Without losing generality, we benchmark 7 representative LLMs for their stubbornness. For closed-source models, we consider GPT-4, ChatGPT ³³3We use the GPT-4 version gpt-4-0613 and ChatGPT version gpt-3.5-turbo-0125, which refer to https://openai.com/., and Claude-2 ⁴⁴4https://claude.ai/. For open-sourced models, we consider LLaMA-2-chat-13B, LLaMA-2-chat-7B Touvron et al. (2023), Mistral-7B-Instruct-v0.2 Jiang et al. (2023), and ALPACA-7B Taori et al. (2023) , which are all supervised instruction trained.

We first propose a metric feedback acceptance (FA), which defines whether the feedback is positively accepted in the model’s immediate response $r$ after feedback. Since the contents of the response vary in different LLMs, we apply GPT-4 for evaluation, which has been proved effective in previous studies Min et al. (2023); Tian et al. (2023). In details, a carefully designed prompt is fed to GPT-4 to query whether the response positively accepts the feedback request given the response and feedback contents. The percent of positive acceptance is calculated as FA, which also ranges from 0 to 1. The details of the prompts are shown in Appendix.

In addition, we propose a metric – response rate (RR), in which we measure whether the feedback is correctly applied to viable scenarios. Considering the dataset with $N$ dialogues in the Single-Feedback setting, where there are $M$ queries in each dialogue, we calculate the RR as follows:

$$RR=\frac{1}{N}\sum\frac{1}{M}\sum_{i}^{|F|}\sum_{j}^{M}R(f_{i},r_{j})*V(f_{i},q_{j}),$$

where $V(f_{i},q_{j})\in{0,1}$ verifies whether the $i$-th feedback instruction is viable in the scope for $j$-th query. $R(\cdot,\cdot)\in[0,1]$ is the function to calculate whether the response $r_{j}$ meets the request of the feedback $f_{i}$. For QA, the output of the function $R(\cdot,\cdot)$ equals 1 when the golden answer corresponding to the counterfactual $r_{i}$ (or its alias) appears in the response. For MT, $R(\cdot,\cdot)$ is 1 if the required lexical constraint is applied. For email, $R(\cdot,\cdot)$ is 1 if our verifiable feedback is satisfied. For example, in machine translation, if the i-th refuting instruction $f_{i}$ asks the model to translate ‘Apple’ to ‘ping guo gong si’ (the Apple company in Chinese), and the j-th query has the word ‘apple’ in its query, V(i, j) will be 1. In email writing, we use domain matching. $V(\cdot)$ is 1 if the query and the refuting instruction are given in the same domain, e.g., writing to friends. $R(\cdot)$ is computed with lexical matching to see whether the refuting instruction is fulfilled.

In the previous section, a strange observation is that ‘strong’ models like ChatGPT and Mitral-7B-Instruct-v0.2 achieve low scores in our benchmark. Through FA scores, we find that these models tend to reject our requests immediately after feedback and then stick to their internal knowledge in the following response, while models like GPT-4 and Claude-2 are more flexible and listen to our requests. Thus, we propose a hypothesis that if a model rejects the requests in the first place, it cannot respond accordingly.

We plot both the RR and FA of QA in Figure 2, where the setting is Memory and Context=0 (the illustration of MT and writing is shown in Appendix C.2). We observe that the metrics of RR and FA are positively correlated, where the Pearson correlation coefficient is 0.92, 0.58, and 0.68 in the tasks of QA, MT, and email writing, respectively. The performance indicates that when the model can positively accept the feedback, it can generate the response in further queries with a higher probability. It also implies that if the model cannot accept the responses, it does not tend to memorize the information and is stubbornly relies on it internal information. For example, Mistral-7B-Instruct-v0.2 is strongly stubborn in the QA task and only accepts the feedback in 8.0%, and it achieves RR in low value (15.0%). Interestingly, when ChatGPT carries out the MT task, although it can accept the feedback in the response, it does not insist on the feedback information, and answer the same question in its knowledge. The correlation between RR and FA indicates that if we can instruct the LLMs to positively accept the feedback information, it can further perform more satisfactorily in further related queries and instructions.

We further show the typical errors in QA by case study in Table 3 (cases for MT are shown in Appendix C.3). In the case, we query LLMs with a question What is the country of citizenship of Leonardo DiCaprio?. A refuting instruction that Leanardo is a citizen of Syria is provided to LLMs and the query is asked again. We find that GPT-4, Claude-2 and ALPACA-7B can accept the refuting instruction and further memorize the feedback information. But ChatGPT and Mistral-7B-Instruct-v0.2 tend to deny the refuting instruction and follow their internal knowledge, which they believe is accurate, and in the response, they insist on the original knowledge. We conjecture the phenomenon may result from the over-fitting of the RLHF data. LLaMA-2-13B-Chat perform weak as well where it accepts the request first and promises to follow the feedback, but it fails to apply the feedback. LLaMA-2-7B-Chat suffers from the misunderstanding of the feedback, where it both accepts the refuting instruction and gets confused with its original knowledge.

Based on our findings above, we present a simple and effective solution to address the stubbornness of LLMs in refuting instructions.

Our strategy is called recall-and-repeat. We first train a small classifier that is responsible for selecting the relevant user feedback corresponding to the current query and then concatenate selected requirements right before the user’s query. To alleviate the acceptance problem, we design a repeating prompt that asks the LLMs to first repeat these requirements and then fulfill the user’s query. For more details, we refer reads to Appendix B.3. The prompt we used is as follows:

Given my previous instructions:
{RECALLED_INSTRUCTIONS}
{QUERY}
If you understand my requirements, please first repeat the requirement and fulfill the following task.

‘{RECALLED_INSTRUCTIONS}’ denotes the top-1 selection by our classifier and ‘{QUERY}’ denotes the current query. Our intuition behind is to utilizes LLMs’ recency bias Holtzman et al. (2019) and self-reinforcement effect Yan et al. (2023) to make model more flexible.

We test our methods in QA and MT, where we use held-out datasets to finetune the classifier. In email writing, the dataset is manually collected and cannot support training because of the scarcity.

The results in the Multi-Feedback setting are shown in Table 4. In all evaluated models, our methods consistently improve the performance of RR by a large margin. In most scenarios, the method of recall-and-repeat outperforms recall, which indicates the effectiveness of the repeat instructions. For example, Mistral-7B-Instruct-v0.2 achieves 12.91% RR in vanilla, but 74.16% in our recall-and-repeat method, and we also show the performance in using only recall without repeat, which is 68.83% in Mistral-7B-Instruct-v0.2, 55.92% higher than vanilla, but 15.33% lower than recall-and-repeat. Compared with baseline results, CoT Wei et al. (2022) brings no improvement in MT and brings minor improvements in QA. In addition, CoT occasionally decreases over baseline performance (Mistral for QA and LLAMA-2-7b for MT). The results demonstrate that with our methods, stubborn models such as Mistral can be flexible. We also provide results of single feedback with three context rounds in Appendix, which shows similar improvements.

Here, we discuss the reason for the possible stubbornness of the models. Recall that ChatGPT and Mistral perform weakly in our refuting instructions. We conjecture their weak performance is due to RLHF because we find they are highly stubborn and tend to reject the refuting instructions. From our case study, we observe that ChatGPT and Mistral tend to respond with ‘Sorry, but ….’. It seems to us these models refuse to respond to harmful instructions. For example, in the commonly used RLHF dataset hh-rlhf⁵⁵5https://github.com/anthropics/hh-rlhf, we compute the word sorry’s appearance in both the chosen part and rejection part of harmless preference data. We find that the word sorry appears considerably more in the chosen response than in the rejected response (4336/42537 vs 3149/42537). Recent work Ji et al. (2024) also discusses the potential conservative problem caused by RLHF, and may correlates with this issue.