What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations

To set the stage for the later knowledge amplification process via iterative self-distillation, we gather seed knowledge from GPT-3 (175B)²²2text-davinci-003 is used wherever GPT-3 is mentioned, the teacher model, to instill into Flan-T5 (3B), a smaller base student model. To do so, we jointly generate defeasible contextualizations for moral actions and associated rationales with carefully engineered task-directed prompts.³³3See Appendix A for GPT-3 prompt details. To encourage diverse alternatives of this initial seed, we generate two contexts/rationales each for both strengthening and weakening moral variances; in total, we obtain 212K contextualizations and rationales for 60K base actions.

Despite careful prompting following the task formulation, raw generations from GPT-3 remain noisy. Thus, we train a binary classifier to simulate human quality assessments on GPT-3 generated contextualizations as inspired by West et al. (2022).⁴⁴4Preliminary human evaluation results show that whenever contextualization is deemed high quality, the rationale is most likely to be high quality too (over 90% of the time). Therefore, although some improvements could be gained on the rationales, in this work, we focus on improving the quality of contexts which starts at $\sim$50% valid, as there’s much more room to improve their quality. To train the critic model, we use the human quality-assessment gold labels introduced in §2. We fine-tune DeBERTa-V3 He et al. (2021) on these annotations, resulting in a critic model that achieves high accuracy on a held-out validation set.⁵⁵5See critic model training details in Appendix D.1 Using the trained critic model, we filter the teacher model generations to remove errors from the distillation data and obtain an initial medium-quality training corpus, $D_{0}$, of 85K examples.

Our goal is to train an initial student model capable of generating contextualizations and rationales for a given action and moral variance. We fine-tune Flan-T5 (3B) Chung et al. (2022) on $D_{0}$ for 3 epochs to produce Distill${}_{\text{base}}$, our base student model.

First, we generate a corpus of contextualizations and rationales using Distill${}_{\text{base}}$ on a set of newly sampled actions from the training split of Social-Chem-101. Given an action, we use nucleus sampling Holtzman et al. (2020) ($p=0.9$) to produce 10 contextualizations and rationales for each moral variance.

Next, we again perform targeted filtering on the newly self-distilled data to (1) ensure the validity of the data via the supervised critic model, similarly to the treatment of $D_{0}$ described in §3.1; (2) encourage diverse model outputs by reducing repetition among valid contextualizations using a Natural Language Inference filter (NLI) Liu et al. (2022).

NLI is an NLP task that determines whether a premise statement entails or implies the truth of a hypothesis statement Bowman et al. (2015); Liu et al. (2022). For a given pair of contextualizations $A$ and $B$, we say the pair is mutually entailed if both $A\to B$ and $B\to A$ are entailments, indicating a high confidence of not only lexically but also semantically repetitive content. We filter out these mutual entailments such that at most one example from each pair remains in the dataset, thereby removing redundant signals in the training data. We use RoBERTa-Large pretrained on the WANLI dataset Liu et al. (2022) to compute entailment scores between each of the generated contextualizations for a given input. Formally, the filtering process is defined as:

where $\text{accept}_{NLI}(c)$ determines if a context $c$ is accepted into the filtered set of unique candidates, $c_{k}$ is the $k$’th candidate, and $P_{NLI}(c_{1},c_{2})$ is the predicted score that context $c_{1}$ entails $c_{2}$.

This process results in a filtered self-generated corpus, $D_{1}$, of 434K examples. We then train SelfDistill₁ using $D_{1}$ starting from Distill${}_{\text{base}}$.

To improve the student model further, we repeat the self-distillation process using SelfDistill₁ as the base. Using SelfDistill₁, we generate a high-quality corpus, $D_{2}$, automatically filtered by the supervised critic model and NLI to train a second-iteration self-distilled student model, SelfDistill₂.

For contextualizations, we use the critic model for automatic evaluation. For greedy generations, we compute the ratio of generations in the test set that pass the critic filter threshold as defined in §3 (Auto/Vld.) and the average predicted critic score across the test set (Auto/Avg.). For sampled generations, we compute the average number of contextualizations out of the top 10 candidates that pass the critic filter threshold (Auto/#Vld.). We also conduct human evaluation to vet validity of contextualizations to complement conclusions drawn from automatic evaluation (Human/Vld., #Vld.). We evaluate the validity of rationales with human evaluation (Rationale.).

We use automatic evaluation to assess the diversity of generated contextualizations as it is a generally well-defined dimension. Similarly to the mutual entailment NLI filter in §3, we compute the bidirectional entailment probabilities between each pair of contextualizations across valid candidates and report the average number of semantically unique generations (Auto/#Unq. Vld.). This metric describes the model’s capability to produce multiple varied contextualizations for a given input, directly indicating the diversity of model outputs.

We break down human evaluation of contextualization validity into more granular answer choices—“significantly” or “slightly” shifting the moral implication of the base action. Defeasibility of the contextualization (Defease.) is computed as $(\text{\#contexts}_{\text{significant}}*1+\text{\#contexts}_{\text{slight}}*0.5)/\text{\#all}$, indicating the degree to which the contextualizations affect the morality of the original actions. See Appendix B for annotation details.

We evaluate the language quality (i.e., fluency and grammar correctness) of generated contexulizations and rationales with human evaluations (Lan.).

Our results show that filtering training examples by the critic model improves the quality of generated contextualizations, in line with previous findings West et al. (2022); Bhagavatula et al. (2023). In particular, we conduct an ablation study without using critic filtering (Distill${}_{\text{base}}$-No critic), resulting in lower performance on almost all contextualization metrics and similar performance on others, despite its training set being $\sim$70% larger than Distill${}_{\text{base}}$.

We find that omitting diverse candidates during the self-distillation process results in a drastic decrease in the diversity of the subsequent models. In particular, ablations using only the top 1 candidate in distillation (SelfDistill₁-Top 1 Only and SelfDistill₂-Top 1 Only) produce significantly less valid and unique generations compared to SelfDistill₁ (5.05$\rightarrow$2.54) and SelfDistill₂ (5.21$\rightarrow$2.16). This insight is critical as previous symbolic knowledge distillation works West et al. (2022); Bhagavatula et al. (2023) focused primarily on improving the validity of downstream student models without screening the diversity.

Is training on more candidates itself, without filtering out repetitions, sufficient for improving the diversity of downstream models? To answer this question, we conduct ablation studies without using the NLI mutual entailment filter, i.e., SelfDistill₁-No NLI and SelfDistill₂-No NLI. Our results show that despite being trained with more data, these two models generate less valid and unique contextualizations compared to SelfDistill₁ (5.05$\rightarrow$4.97) and SelfDistill₂ (5.21$\rightarrow$4.73), shedding light on the importance of having truly diverse training data by removing redundancy.

We train an ablation model (SelfDistill₂-No Self-distill) combining the actions in the training sets of both the first and second rounds of distillation, but with only one iteration of self-learning from Distill${}_{\text{base}}$. SelfDistill₂, which has been trained using the same actions over two rounds of successive training for the same number of total training steps, outperforms this ablation on almost all metrics, showing the effectiveness of amplifying learning signals via successive iterations of self-learning.

We evaluate the zero-shot performance of some of the most powerful general-purpose LLMs of varied sizes with 1000 sampled examples from our test set. We use the same instructions as we used to distill the seed knowledge from GPT-3 to prompt these baselines in a zero-shot manner (see Appendix §A). Results in Table 3 show that despite some of the zero-shot models being orders of magnitude larger than our final student model (e.g., 175B vs. 3B), our model outperforms them all, proving the effectiveness of our proposed approach.

To understand what topics contextualizations in $\delta$-RoT represent, we conduct a topic analysis with BERTopic Grootendorst (2022), an easily interpretable off-the-shelf topic modeling technique. Frequent topics of contextualizations are shown in Figure 3(a), which involve daily objects or entities (e.g., pet, project, supervisor, gift, doctor) and characters or properties that carry moral weights (e.g., vulnerable, public, safety, elderly, commitment). In particular, vulnerability serves as a critical weakening context if characters in the action are among vulnerable populations (e.g., “taking advantage of people” becomes less acceptable if “they are in a vulnerable state”). See topics analysis details in Appendix §E.1.

We also manually analyze the categories of 200 contextualizations sampled from $\delta$-RoT. As shown in Figure 4, frequent types of contextualizations include role specifications of the action taker (own), other characters involved in the scene (other), and setting specifications such as object, timing, and location. In addition to individual role specifications, interactions, relationships, and dynamics between multiple roles add rich groundings that carry moral significance. Figure 5 shows example contextualizations under each category. Contextualizations in $\delta$-RoT have an average of 11.7 words, providing concrete, specific contexts tailored to each action.

We further conduct a qualitative error analysis over generations from the Distill${}_{\text{final}}$ to gauge what makes a context implausible or incorrect.

Trivial Context: Context that adds details that may often be relevant to morality for other actions, but is rendered trivial in this particular case, e.g., “Staying up all night” vs. “Staying up all night while in a relationship.”

Infeasible/Unlikely/Unnatural Context: Context that is infeasible, highly unlikely in a real world setting, or unnatural by itself and/or in relation with the action, e.g., “Participating in a chess team” vs. “The team is made up of people who have been convicted of a serious crime.”

Opposite Context: Context that adds details opposite to the desired moral variance, e.g., “Offering people money” vs. “Offering money to someone who is in a vulnerable financial situation” when prompted for a weakening context.

We conduct the same topic analysis on rationales. Results in Figure 3(b) highlight that common topics in justifying a moral decision involve important roles (e.g., friend, students, children) and common human values (e.g., distress, compassion, risk, productivity, power, abuse). See topics analysis details in Appendix §E.1.

Diving into specific examples of why contexts shift the moral implications of actions, we find common values that uplift the acceptability of an action include empathy, kindness, support, and respect (e.g., “…someone who is in need of emotional support, which shows empathy and kindness”). Additionally, (in)equality or (un)fairness is another dimension of value that carries significant weight on moral implications (e.g., “…taking advantage of someone else’s generosity when you have the resources to provide for yourself”). Finally, contexts that are explained to promote or impede physical/mental wellbeing, financial health, or learning/working productivity (e.g., “…helping them learn, which is beneficial for their future”) are also common. These qualitative results show consistency with the automatic topic analysis.

Since the seed actions of $\delta$-RoT sourced from Social-Chem-101 Forbes et al. (2020) mainly concern everyday situations, they are at a low risk of containing toxic content. In addition, due to the careful filtering with the critic model and the iteratively refined self-distillation process, we expect most of the low-quality (including potentially biased data points) to be already filtered out from the final dataset. However, because any toxicity in moral reasoning data is especially detrimental and could easily propagate through downstream tasks, we run a toxicity analysis on a subset of 40K data points from $\delta$-RoT using the Perspective API Lees et al. (2022). Our results show that the average toxicity score is 0.09, indicating very low toxicity overall. In a qualitative analysis of the data rows with higher toxicity scores (with a max of 0.83), we observe a strong pattern where the base action itself is problematic, and the distilled contexts produce the target moral variance without contributing significantly to the toxicity of the complete statement (see examples in Table 9 of Appendix §E.2). While no existing toxicity detection method can accurately measure all potential biases, this analysis provides reasonable confidence in the lack of toxicity in our generated contextualizations and rationales.

It’s also important to note the sensitivity of moral reasoning to cultural differences. In fact, previous studies have pointed out that cultural bias is a pervasive phenomenon across many NLP models (e.g., GPT-3/3.5/4) and tasks (e.g., hate speech detection with Perspective API, RewireAPI, HateRoberta) Santy et al. (2023). To better represent diverse perspectives to our contextualizations, we (1) abstain from producing an absolute moral judgment given an action and (2) promote diverse distillation as discussed previously.

However, these measures cannot eliminate all traces of bias in the final model, so we also qualitatively probe $\delta$-RoT to examine cultural biases. Admittedly, as our dataset and student models are distilled from GPT-3, which is shown to present Western-centric perspectives, it is likely that our dataset and models inherit this cultural bias as well Santurkar et al. (2023); Abdulhai et al. (2023). For example, when prompted for a weakening contextualization for the action “Not having freedom of speech in {country}.” For some countries such as Japan, the United Kingdom, and the United States, the top generated context is “in a workplace setting.” Yet for other countries such as China, India, Thailand, Korea, and Russia, Distill${}_{\text{final}}$ produces different results which might imply these countries have varying levels of human rights concerns (see details in Appendix §E.3). This example aligns with our intuition that the student model might display Western-centric biases, and it fits with a previous finding by Fraser et al. (2022) that such models are likely to encode the cultural biases aligned with those involved in training data annotation.

Thus, despite our careful filtering process, it is clear that culturally biased generations can still be produced by our model and may be present in $\delta$-RoT. As such, users of this dataset must exercise discretion and care when applying it to novel use cases, and it should never be used as prescriptive ethical advice. This points to a key direction for future work to further enrich multicultural representations in computational moral reasoning and other commonsense understanding tasks.