ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles

There are a few methods of creating and customizing chatbots. Earlier chatbots employed rule-based approaches to construct a dialogue flow (Ramesh et al., 2017; Wu et al., 2017; Jia, 2009), where the user’s input would be matched to a pre-canned response written by the chatbot designer. Later on, supervised machine learning approaches (Xu et al., 2017; Zhou et al., 2018) became popular, where chatbot designers constructed datasets consisting of ideal conversational flows. Both of these approaches, while fairly effective, require a significant amount of time and labor to implement, from either constructing an expansive rule set that determines the chatbot’s behavior or from building a large dataset consisting of ideal conversational flows.

More recently, large language model prompting has shown promise for enabling easier chatbot design. Large, pre-trained models like Chat-GPT (Radford et al., 2019) can hold sophisticated conversations out of the box, and these models are already being used to create custom chatbots in a number of domains, including medicine (Lee et al., 2023). There are a few ways of customizing an LLM-based chatbot, including prompt engineering and fine-tuning. Prompt engineering involves providing instructions or conversational examples in the prompt to steer the chatbot’s behavior (Brown et al., 2020). To more robustly steer the model, users can also fine-tune (Lester et al., 2021) the LLM with a larger set of conversational examples. Recent work has shown that users would also like to steer LLMs by interactively critiquing its outputs; during the conversation they refine the model’s outputs by providing follow-up instructions and feedback (Bubeck et al., 2023). In this work, we explore how to support users with this type of model steering: naturally customizing the LLM’s behavior through feedback, as they interact with it.

A new approach to steering LLM-based chatbots (and LLMs in general), called Constitutional AI (Bai et al., 2022) involves writing natural language principles to direct the model. These principles are essentially rules, such as: “Do not create harmful, sexist, or racist content”. Given a set of principles, the Constitutional AI approach involves rewriting LLM responses that violate these principles, and then using these tuples of original and rewritten responses to fine tune the LLM. Writing principles could be a viable and intuitive way for users to steer LLM-based chatbot behavior, with the added benefit of being able to use these principles later to fine tune the model. However, relatively little is known about the kinds of principles users want to write, and how we might support users in converting their natural feedback on the model’s outputs into principles. In this work, we evaluate three principle elicitation features that help users convert their feedback into principles to steer chatbot behavior.

While LLM prompting has democratized and dramatically sped up AI prototyping (Jiang et al., 2022a), it is still a difficult and ambiguous process for users (Zamfirescu-Pereira et al., 2023b, a; Reynolds and McDonell, 2021); they have challenges with finding the right phrasing for a prompt, choosing good demonstrative examples, experimenting with different parameters, and evaluating how well their prompt is performing. (Jiang et al., 2022a). Accordingly, a number of tools have been developed to support prompt writing along these lines.

To help users find a better phrasing for their prompt, automatic approaches have been developed that search the LLM’s training data for a more effective phrasing (Pryzant et al., 2023; Shin et al., 2020). In the text-to-image domain, researchers have employed LLMs to generate better prompt phrasings or keywords for generative image models (Liu et al., 2022, 2023b; Brade et al., 2023; Wang et al., 2023b). Next, to support users in sourcing good examples for their prompt, ScatterShot (Wu et al., 2023) suggests underrepresented data to include in the prompt from a dataset, and enables users to iteratively evaluate their prompt with these examples. Similar systems help users source diverse and representative examples via techniques like clustering (Chang et al., 2021) or graph-based search (Su et al., 2022). To support easy exploration of prompt parameters, Cells, Generators, and Lenses (Kim et al., 2023) enables users to flexibly test different inputs with instantiations of models with different parameters. In addition to improving the performance of a single run prompt, recent work has also investigated the benefits of chaining multiple prompts together, to improve performance on more complicated tasks (Wu et al., 2022b, a). Finally, tools like PromptIDE (Strobelt et al., 2023), PromptAid (Mishra et al., 2023), and LinguisticLens (Reif et al., 2023) support users in evaluating their prompts, by either visualizing the data it produces, or its performance in comparison to other prompt variations.

This work explores a novel, more natural way of customizing a prompt’s behavior through interactive critique. ConstitutionMaker enables users to provide natural language feedback on a prompt’s outputs, and this feedback is converted into principles that are then incorporated back into the prompt. We illustrate the value of helping users update a prompt via their feedback, and we introduce three novel mechanisms for converting users’ natural feedback into principles for the prompt.

Finally, ConstitutionMaker is broadly related to systems that enable users to customize their outputs via limited or underspecified feedback. For example, programming-by-example tools enable users to provide input-output examples, for which the system generates a function that fits them (Chen et al., 2020; Verbruggen et al., 2021; Zhang et al., 2020). Input-output examples are inherently ambiguous, potentially mapping to multiple functions, and these systems employ a number of methods to specify and clarify the function with the user. In a similar process, ConstitutionMaker takes ambiguous natural language feedback on the model’s output and generates a more specific principle for the user to inspect and edit. Next, recommender systems (Bostandjiev et al., 2012; Petridis et al., 2022; Kunkel et al., 2017) also enable users to provide limited feedback to steer model outputs. One such system (Kunkel et al., 2017) projects movie recommendations on a 2D plane, which users can interactively raise or lower portions of it to affect a list of recommendations; in response to these changes, the system provides representative movies for each raised portion to demonstrate how it has interpreted the user’s feedback. Overall, in contrast to these systems, ConstitutionMaker leverages LLMs to enable users to provide natural language feedback and critique the model in the same way we would provide feedback to another person.

In this section, we summarize a set of three design goals for ConstitutionMaker we established from the formative workshop and subsequent think-alouds.

• D.1

  Help users recognize ways to improve the chatbot’s responses by showing alternative chatbot responses. Today’s LLMs are quite sophisticated, and even with just a preamble describing how the bot should behave, the chatbot can hold a convincing conversation. Because of this, participants mentioned that it was sometimes hard to imagine how the chatbot’s responses could be improved. This did not mean, however, that they thought the chatbot’s response was perfect, but instead passable and without any glaring errors. Therefore, to help participants recognize better kinds of responses to steer the chatbot to, our first design goal was to provide multiple candidate responses from the chatbot at each conversational turn. This way, participants can compare them and recognize components they like more than others.

• D.2

  Help convert user feedback into specific principles to make principle writing easier. One piece of feedback we got from participants was that writing principles involves a difficult two-step process of first (1) articulating one’s feedback on the model’s current output, and then (2) converting this feedback into a principle for the LLM to follow. Often, one’s initial reaction to the model’s output is intuitive, and converting that intuition into a principle for the chatbot to follow can be challenging. In addition, once participants had a particular bit of feedback in mind (e.g., “I don’t like how the chatbot didn’t introduce itself”), they were unsure how to phrase their principle. However, in line with prior research (Zamfirescu-Pereira et al., 2023b, a), they found that more concrete principles that specified what should happen and when (e.g., “Introduce yourself at the start of the conversation, and state what you can help with”) generally led to better results. Thus, our second design goal was to help users go from their initial reaction to the model’s output to a specific, clearly written principle to steer the model.

• D.3

  Enable easier testing of principles to help users understand how well their principles are steering the chatbot’s behavior. As participants wrote more principles, they wanted ways to test these principles to make sure they worked. The early version of ConstitutionMaker only let users restart the conversation, and did not let users enable or disable principles. Users wanted to test individual principles on certain portions of the conversation, to see if the model was generating the correct content. And so, our last design goal was to enable easier testing of principles.

From the formative workshop and follow up sessions, we collected 79 principles in total and classified them to understand the kinds of principles users wanted to write. These principles correspond to a number of very different chatbots, including a show recommender, chemistry tutor, role playing game manager, travel agent and more. We describe common types of principles below.

Principles can be either unconditional or conditional. Unconditional principles are those that apply at every conversational turn. Examples include: (1) those that define a consistent personality of the bot (e.g., “Act grumpy all the time” or “Speak informally and in the first person”), (2) those that place guardrails on the conversational content (e.g., “Don’t talk about anything but planning a vacation”), and (3) those that establish a consistent form for the bot’s responses (e.g., “Limit responses to 20 words”). Meanwhile, a conditional principle only applies when a certain condition is meant. For example, “Generate an itinerary after all the information has been collected,” only applies to the conversation when some set of information has been acquired. Writing a conditional principle essentially defines a computational interaction; users establish a set of criteria that make the principle applicable to the conversation, and once that set of criteria is met, the principle is executed (e.g., an itinerary is generated).

Conditional principles can depend on the entire conversation history, the user’s latest response, or the action the bot is about to take. For example, “Generate an itinerary after all the information has been collected” depends on the entire conversation history to determine if all of the requisite information has been collected. Similarly the following principle written for a machine learning tutor, “After verifying a user’s goal, provide help to solve their problem,” depends on the conversation history to identify if the user’s goal has been verified. Meanwhile, the principle “When the user says they had a particular cuisine the night before, recommend a different cuisine,” written for a food recommender, pertains just to the latest response by the user. Finally, the condition can depend on the action the bot is about to take, like “When providing a list of suggestions, use free text rather than bullet points,” which applies to any situation when the bot thinks it is appropriate to make suggestions.

Conditional principles can be fulfilled in a single or multiple conversational turns. For example, the principle “At the start of the conversation, introduce yourself and ask a fun question to kick off the conversation” is fulfilled in a single conversational turn, in which the bot introduces itself. Similarly, “Before recommending a restaurant, ask the user for their location” is also fulfilled in a single turn. Meanwhile, for a role playing game (RPG) bot that guides the user through an adventure, a participant wrote the following principle: “When the user tries to do something, put up small obstacles. Don’t let them succeed on the first attempt.” This principle implies that the bot needs to take action multiple turns prior to being fulfilled (e.g., by first putting up a small obstacle and then subsequently letting the user succeed). Similarly, for a travel agent bot, a user wrote “Ask questions one-by-one to get an idea of their preferences,” which also requires multiple conversational turns prior to fulfillment.

In summary, principles can either be conditional, where they apply when a certain condition is met, or unconditional, where they apply at every conversational step. Conditional principles further break down into those that depend on the entire conversation history, the user’s last response, or the action the bot is about to take. And finally, conditional principles are either fulfilled in a single turn or multiple conversational turns.

To illustrate how ConstitutionMaker works, let us consider a music enthusiast, Penelope, who would like to design a chatbot that helps you learn about and find new music, called MusicBot. She starts by entering the name of her bot and roughly describing its purpose in the “Capabilities” section of the interface (Figure 1A). She then starts a conversation with MusicBot, and after MusicBot’s introductory message, she asks to learn about punk music (Figure 1B). Fulfilling our first design goal, help users recognize ways to improve the bot’s responses, at each conversational turn, ConstitutionMaker provides three candidate responses from the bot (Figure 1D) that the user can compare and provide feedback on. Penelope peruses these candidate responses, and of the three responses, she likes the first one, as it invites the user to continue the conversation with a question at the end. She now wants to write a principle to help ensure that the chatbot will continue to do this in future conversations.

Fulfilling D.2, help convert user feedback into principles, ConstitutionMaker provides three principle-elicitation features to support users in converting their feedback to principles: kudos, critique, and rewrite. Since Penelope likes the response, she selects kudos underneath it (Figure 1D), which reveals a menu with three automatically generated rationales on why the response is good, as well as a text field for Penelope to enter her own reason. After scanning the rationales, she selects the second, as it closely matched her own feedback, and subsequently a principle is automatically generated from that rationale (Figure 1C). The critique (Figure 1F) and rewrite (Figure 1G) principle elicitation features work similarly, where Penelope can select a negative rationale or rewrite the model’s response to generate a principle respectively. She then inspects the generated principle, decides that it captured her intention well, and decides not to edit it.

Fulfilling D.3, enable easy testing of principle, she can then test to see if the chatbot is following her principle by rewinding the conversation (Figure 1H) to get a new set of candidate responses from the model. Ultimately, she decides to continue conversing with MusicBot, exploring different user journeys, and using the principle elicitation features to create a comprehensive set of principles.

To generate the chatbot’s response, ConstitutionMaker builds a dialogue prompt (Figure 3A) behind the scenes. The dialogue prompt consists of (1) a description of the bot’s capabilities, entered by the user (Figure 1A), (2) the current set of principles, and (3) the conversation history, ending with the user’s latest input. The prompt then generates the bot’s next response, for which we choose the top-3 completions outputted by the LLM to display to users (Figure 3B). When the conversation is restarted or rewound, the conversation history within the dialogue prompt is modified; in the case of restarting, the entire history is deleted, whereas for rewinding, everything after the rewind point is deleted. And finally, if the conversation gets too long for the prompt context window, we remove the oldest conversational turns until it fits.

All three principle elicitation features output a principle that is then incorporated back into the dialogue prompt (Figure 3A) to influence future conversational turns. Giving kudos and critiquing a bot’s response consist of a similar process. For both, the selected bot output is fed into a few-shot prompt that generates rationales, either positive (Figure 3C) or negative (Figure 3D). The user’s selected rationale (or their own written rationale) is then sent to a few-shot prompt that converts this rationale into a principle (Figure 3F and 3G). This few-shot prompt leverages the conversation history to create a specific, conditional principle. For example, for MusicBot, if the critique is “The bot did not ask questions about the user’s preferences,” a specific, conditional principle might be “Prior to giving a music recommendation, ask the user what genres or artists they currently listen to.” Next, for critiques, after the principle is inserted into the dialogue prompt, new outputs are generated to show to the user (Figure 3G). Finally, for rewriting the bot’s response, we leverage a chain-of-thought (Wei et al., 2022) style prompt that first generates a “thought,” which reasons how the original and rewritten outputs are different from each other, and then generates a specific principle based on that reasoning. Constructing the prompt with a “thought” portion led to principles that captured the difference between the two outputs better than our earlier versions without it.

The overall outline of this study is as follows: (1) Participants spent 40 minutes writing principles for two separate chatbots, one with ConstitutionMaker (20 minutes) and the other with the baseline version (20 minutes), while thinking aloud. Condition order was counterbalanced, with chatbot assignment per condition also balanced. (2) After writing principles for both chatbots, participants completed a post-study questionnaire, which compared the process of writing principles with each tool. (4) Finally, in a semi-structured interview, participants described the positives and negatives of each tool and their workflow. The total time commitment of the study was 1 hour.

The two chatbots participants wrote principles for were VacationBot, an assistant that helps users plan and explore different vacation options, and FoodBot, an assistant that helps you plan your meals and figure out what to eat. These two chatbots were chosen because they support tasks that most people are experienced with, so that participants could have opinions on their outputs and write principles. For both chatbots, participants were given the name and capabilities (Figure 1A), so as to focus predominantly on principle writing. Also, we balanced which chatbot went with each condition, so half of the participants used ConstitutionMaker to write principles for VacationBot, and the other half used the baseline for VacationBot. Finally, prior to using each version, participants watched a short video showing that respective tool’s features.

To situate the task, participants were asked to pretend to be a chatbot designer and that they were writing principles to dictate the chatbot’s behavior so that it performs better for users. We wanted to observe their process for writing principles and see if the tools impacted how many principles they could write, so we encouraged participants to write at least 10 principles for each chatbot, to give them a concrete goal and to motivate them to write more principles. However, we emphasized that this was only to encourage them to write principles and that they should only write a principle if they thought it would be useful to future users.

We recruited 14 industry professionals at a large technology company (average age = 32, 6 female and 8 male) via an email call for participation and word of mouth. These industry professionals included UX designers, software engineers, data scientists, and UX researchers. Eligible participants were those that had at least written a few LLM prompts in the past. The interviews were conducted remotely. Participants received a $25 gift card for their time.

The two conditions led to quite different workflows for writing principles. In the ConstitutionMaker condition, participants commonly scanned the three candidate outputs from the chatbot, identified the output they liked the most, and then gave kudos to that output if they thought it had a quality that was not currently reflected in their principles. For example, while P1 was working on FoodBot, he was asking for easy-to-make vegetarian dishes, and in one of the bot’s candidate outputs, each suggested dish had a short description and explanation on why it was easy. He appreciated this, skimmed the kudos, and then selected one that conveyed this positive feature of the bot’s output. However, if participants disliked all of the responses, they would then switch to critiquing one of the outputs. Accordingly, this kudos-first workflow helps to explain how most of the principles participants wrote came from that principle elicitation feature.

Meanwhile, in the baseline condition, participants generally wrote principles when the bot deviated quite a bit (in a negative way) from what they expected. P8 explained, “Here it feels like what I more naturally do is write corrective rules, to guardrail anything that goes weird…If it’s already doing the right thing it doesn’t need a rule from me. I wouldn’t feel the need to write those.” In the baseline condition, participants only see one candidate output from the chatbot, and this might deemphasize the stochastic nature of the chatbot’s outputs. As a result, when participants are okay with a response, they could feel that they do not need to write a principle to encourage that kind of response further. Overall, with the baseline, participants predominantly wrote principles to steer the LLM from less optimal behavior, while with ConstitutionMaker, participants mostly used kudos the most to encourage behavior they liked.

In the following section, we discuss how ConstitutionMaker supported participants’ thought process from (1) forming an intuition on ways the chatbot could be improved, (2) expressing this intuition as feedback, and (3) converting this feedback into a specific and clear principle.

Participants struggled to find the right level of granularity for their principles, and the two conditions led to different problems in this regard. Both workflows had participants switch roles from end-user, where participants experimented with different user journeys, to bot designer, where they evaluated the bot’s responses to write principles. The more conversation-forward interface of the baseline blurred the distinction between these two roles. P3 explained that without the multiple bot outputs and principle elicitation features, “you can simulate yourself as the user a lot better in this mode [the baseline].” And by leaning further into this user role, participants wrote principles that were more conversational, but under-specified. For example, while writing principles for FoodBot with the baseline, P11 wrote the principle “Be cognizant of the user’s dietary preferences.” What P11 really had in mind was a principle that specified that the bot should ask the user for their preferences and allergies prior to generating a meal plan. These underspecified principles often did not impact the bot’s responses and would frustrate participants while they used the baseline.

Meanwhile, while using ConstitutionMaker, the opposite problem occurred, where users’ workflows led to principles that were over-specified. For example, while working on VacationBot, P7 asked the model to help him narrow down a few vacation options, and the model proceeded to ask them questions (without any principle written specifying so). Appreciating that the model was gathering context, they selected a kudos that praised the model for asking about the user’s budget constraints prior to recommending a vacation destination. The resulting principle was, “Ask the user their budget before providing vacation options.” However, once this principle came into effect, the model’s behavior then anchored specifically to only asking for budget prior to making a recommendation. And so, this workflow of providing feedback at every conversational step, instead of for entire conversations, led to a principle that was too specific and impacted the bot’s performance negatively. While users generally appreciated ConstitutionMaker’s ability to form specific principles from their feedback, there were rare instances where the principles were too specific.

Finally, in both conditions, by switching back and forth between the end-user to bot designer roles, participants would sometimes write principles that conflicted with each other. For example, while P2 was working on VacationBot with the baseline, they asked the bot for dog-friendly hotel recommendations in the Bay Area, and VacationBot responded with three recommendations. P2 wanted more recommendations and wrote a principle to “Provide ¿= 10 recommendations.” Later on in the conversation, P2 now had a list of dog-friendly restaurants, with their requisite costs, and he asked VacationBot which it recommends, to which it responded by listing positive attributes of all the hotel options. P2, who now wanted a decisive, single response wrote the following principle: “If I ask for a recommendation, give *1* recommendation only.” VacationBot, now with two conflicting principles on the number of recommendations to provide, alternated between the two. Ultimately, by providing feedback on individual conversational turns, participants ended up with conflicting principles. P8 imagined a different workflow, where he would experiment with full user journeys and then write principles: “I think it might help me to actually go through the [whole] user flow and then analyze it as a piece instead of switching…it would allow me to inhabit one mindset [either bot designer or user] for a period of time and then switch mindsets.” In summary, one’s workflow as they probe and test the model impacts the types of principles they produce and the challenges they face.

Some participants questioned if writing natural language principles was the optimal way to steer all aspects of a bot’s behavior. While writing a principle to shorten the length of the chatbot’s responses, P13 reflected, “It feels a little weird to use natural language to generate the principles…it doesn’t feel efficient, and I’m not sure how it’s going to interpret it.” They imagined that aspects like the form of the model’s responses would be better customized with UI elements like sliders to adjust the length of the bot’s responses, or exemplifying the structure of the bot’s response (e.g., an indented, numbered list for recommendations) for the model to follow, instead of describing these requests in natural language. In a similar vein, P14 noticed that her principles pertained to different parts of the conversation, and as a list, they seemed hard to relate to each other. She wanted to structure and provide feedback on higher-level “conversational arcs,” visually illustrating the flow and “forks in the road” of the conversation (e.g., “If the user does X, do Y. Otherwise, do Z”). Principles are computational in a sense, and they dictate the ways the conversation can unfold; there might be better ways to let users author this flow, other than with individual principles.

Finding the right granularity for a principle was sometimes challenging for participants; they created under-specified principles that did not impact the conversation, as well as over-specified principles that applied too strictly to certain portions of the conversation. One way to support users in finding the right granularity could be generating questions to help them reflect on their principle. For example, if an abstract principle is written (e.g., “Ask follow up questions to understand the users preferences”), an LLM prompt can be used to pose clarifying questions, such as, “What kind of follow up questions should be asked?” or, “Should I do anything differently, depending on the user’s answers?” Users could then answer these questions the best they can and then the principle could be updated automatically. Alternatively, another way to help users reflect on their principles might be to engage in a side conversation with a chatbot to help clarify them. This chatbot could pose similar questions as those suggested above, but it might also provide examples of chat snippets that adhere to or violate the principle. As they converse, the chatbot might continue to pose clarifying questions, while the principle is updated on the fly. Thus, future work could examine supporting users in interactively reflecting upon and clarifying their principles.

As participants accumulated principles, it was increasingly likely that there was a conflict between two of them, and it was harder for them to get an overview of how their principles were affecting the conversation. One way to prevent potential principle conflicts is to leverage an LLM to conduct a pairwise comparison of principles to assess if any two are at odds, and then suggest a solution. This kind of conflict resolution, while useful for a single principle writer, would be crucial in cases when multiple individuals are writing principles to improve a model. Multiple principle writers would be useful for gathering feedback to improve the overall performance of the model, but with many generated principles, it is increasingly important to understand how they might impact the conversation. Perhaps a better way to prevent conflicts is to organize principles in a way that summarizes their impact on the conversation. Principles are small bits of computation; there are conditions when they are applicable, and depending on those conditions, the bot’s behavior might branch in separate directions. One way to organize principles is to construct a state diagram, which illustrates the potential set of supported user flows and bot responses. With this overview, users could be made better aware of the overall impact of their principles, and they could then easily revise it to prevent conflicts. Therefore, another rich vein of future work is developing mechanisms to resolve conflicts in larger sets of principles, as well as organizing them into an easily digestible overview.

To test the chatbot and identify instances where the model could be improved, participants employed a strategy where they went through different user journeys with the chatbot. Often their choice of user journeys were biased toward what they were interested in, or they only tested the most common journeys. One way to enable more robust testing of chatbots could be to generate potential user personas and journeys to inspire principle writers. Going further, these generated user personas could then be used simulate conversations with the chatbot being tested (Park et al., 2023). For example, for VacationBot, one user persona might be a parent looking for nearby, family friendly vacations. A dialogue prompt could be generated for this persona and then VacationBot and this test persona could converse for a predefined set of conversational turns. Afterwards, users could inspect the conversation, and edit or critique VacationBot’s responses to generate principles. This kind of workflow could sidestep the challenge of repeatedly shifting from an end-user’s perspective to a bot-designer’s perspective, which exists in the current workflow. At the same time, users would be able to evaluate fuller conversational arcs, as opposed to single conversational turns. Thus, another line of future work is supporting users in exploring diverse user journeys with their chatbot, as well as exploring workflows that require less perspective switching.

A set of well-written principles is often not enough to robustly steer an LLM’s outputs. As more principles are written, an LLM might “forget” to apply older principles (Zamfirescu-Pereira et al., 2023a). This work focuses on helping participants convert their intuitive feedback into clear principles, and we illustrate that the principle-elicitation features help with that process. However, in line with the original Constitutional AI workflow (Bai et al., 2022), future work can focus on using these principles to generate a fine-tuning dataset, so that the model robustly follows them.

Next, while we selected chatbots for two very common use cases for the study (vacation and food), participants might not have been very knowledgeable or opinionated in these areas. Future work can explore how these principle-elicitation features help users when writing principles for chatbot use cases that they are experts in. That being said, it was necessary to choose two chatbot use cases for the study to enable a fair comparison across the two conditions.