A Comparison of LLM Fine-tuning Methods and Evaluation Metrics with Travel Chatbot Use Case

The tourism and travel industry took a nosedive during the COVID-19 pandemic, which was declared by the World Health Organization (WHO) on March, 11, 2020, due to the severe lockdown measures restricting travel.[1] Now, post-COVID, travelers are coming back with a vengeance and so was the “revenge travel” phenomena named to reflect the negative aspects of tourists exceeding the carrying capacity of destinations and business and governments desperately scrambling to recover income and grow the economy once again at the cost of tourism quality and sustainability.[2] The travel and tourism industry is projected to grow by 5.8% per year for the next decade, which outpaces the overall economy at 2.7%.[3]

The industry has faced challenges with widespread labor shortages due in short to poor working conditions, which along with the rise of large language model (LLM) applications presents a unique opportunity to incorporate technology into the travel and tourism industry.[3] Already, travelers overwhelmingly prefer to utilize technology for the full stack of travel from planning to booking to implementation. 74% of travelers use the internet for travel planning, 45% specifically use smartphones, 36% are willing to pay more for an interactive and smooth booking experience, and 80% prefer to use self-service to find information.[4] Technology is institutionalized with the global distribution system (GDS), an international reservation system for travel agents and suppliers, which is essential to online travel booking and customer service applications.[4]

Large language models (LLMs) were made possible with the 2017 “Attention is All You Need”[5] transformer architecture without recurrent networks or convolutions, and work by predicting masked words or upcoming words.[6] With LLMs, more data results in better predictions, most models have at least one billion parameters.[6] They have been used for many purposes recently including carrying out task-based instructions, multilingual processing, medical and clinical applications, and programming.[6] According to the literature, LLMs are typically trained on a huge corpus of text data with hundreds of billions of hyperparameters.[7] Just in this year, 2023, we have seen several incredible applications utilize powerful LLMs, including in the booming travel industry. With the projected growth of the travel industry post-COVID and recent technological advances, there are potent opportunities for groundbreaking innovation with tangible effects on tourism income. Current artificial intelligence (AI) applications in travel focus on booking, however, there is a gap when it comes to artificial intelligence (AI) assisted travel planning and recommendations (as of May 2024). While not specifically tuned for marketing, ChatGPT remains a competitive chatbot for this application with its high quality conversational ability, however, to get customized travel recommendations, it requires very specific and detailed prompts, and it has a knowledge cut off of September 2021.

Using the travel use case, this research compared two fine-tuning methods: 1) Quantized Low Rank Adapters (QLoRA) and 2) Retrieval-Augmented fine-tuning (RAFT). Two pretrained 7B models, LLaMa 2 and Mistral, are fine-tuned with these two methods, resulting in four models, then their inferences are evaluated against an extensive set of metrics using GPT as a baseline for comparison. The best model is fine-tuned with the third method, 3) Reinforcement Learning from Human Feedback (RLHF), resulting in 5 total models (see Figure 1). The evaluation metrics include: End to End (E2E) benchmark method of “Golden Answers”, traditional natural language processing (NLP) metrics, RAG Assessment (Ragas), OpenAI GPT-4 evaluation metrics, and human evaluation.

A highly effective fine-tuning technique called QLoRA was introduced in 2023.[8] Given the significant computational resources required for fine-tuning Large Language Models (LLM), QLoRA implements several innovative approaches to save memory without compromising performances. It applies gradient backpropagation through a frozen 4-bit quantized pretrained language model into Low Rank Adapters (LoRA). Additionally, it employs double quantization to lower the average memory demand. Furthermore, it incorporates paged optimizers to handle memory spikes. The second method implemented was a novel method called Retrieval Augmented Fine Tuning (RAFT), a training procedure for domain-specific Retrieval Augmented Generation (RAG), which can adapt pretrained LLMs like LLaMa 2 and Mistral for RAG in specific domains, such as ours in travel.[9] RAG is a text generation method for outsourcing relevant information, from a knowledge base, or a large corpus of relevant, factual and quality information, to supply an LLM with contextual clues for producing factual responses. Then, an RLHF training pipeline will be done for domain-specific LLM curation, aligned with human preferences, with reward model training.[10]

Through a comprehensive review of literature and existing research papers, we build an understanding for state-or-the-art techniques and approaches aimed to achieve competitive performances. New innovations are expressed in different variations for enhancing large language models. Table II summarizes the selection of models, objective addressed, unique approach, performance results and ultimate findings. A survey of existing solutions show that initiatives have been developed to overcome shortcoming identified with public general-purpose and pretrained LLM. Some of the feature performance issues garnered from literature review are potential risks of hallucination, underdeveloped retrieval approaches, and inefficiencies in the use of computational resources.

Zhang et. al introduced Retrieval Augmented fine-tuning (RAFT) as a novel training strategy for fine-tuning LLMs to better perform on RAG tasks. The key concept is data augmentation to generate ”question, answer, document” triplets before fine-tuning. This is done by generating realistic questions paired with elaborate chain of thought answering scheme and purposefully including relevant and irrelevant context documents. Through a chain of thought with the distractor documents, the model learns to extract the correct information from the entire chunk of context through reasoning, ignoring the distractors. RAFT operates by training the model to disregard any retrieved documents that do not contribute to answering a given question, thereby eliminating distractions. The optimal ratio of oracle to distractor documents during training varies across datasets, but including some distractors improves generalization. Finally, during RAG, RAFT retrieves the top-k documents from the database. With RAFT, Zhang et. al addresses the limitations of fine-tuning with RAG, and evaluate their new methods on datasets including: PubMed, HotpotQA, and code documentation. RAFT consistently outperforms standard fine-tuning or RAG alone.[9]

With the following literature survey on the existing research, we tried to gain an understanding on the current state of one of the most recent and fastest growing technologies (see Figure 2). We attempted to get the basic understanding of the LLMs, different types of LLM chatbots, technologies, approach used, learnt different types of approaches works better in certain types of research works, and the performance of these may vary depending on the tasks. Some of the research studies shows, some chatbots perform well without the fine-tuning as well. Below is the summary of different research papers the team has gone through, some of the papers are summarized in the tabular form as well (see Table II), showcasing the brief on the goal, approach used and its conclusion. These papers helped the team to have a better understanding on the topic.

Gudibande et al. (2023) warns against the temptation of open source language models trained on imitation data to mimic ChatGPT’s proprietary abilities.[15] These models use ChatGPT conversations with real users collected through ShareGPT, Human-ChatGPT Comparison Corpus, r/ChatGPT subreddit, and the Discord ChatGPT bot to train.[15] For reference, ChatGPT uses between 1.5 and 3 billion parameters depending on the model, while these imitation models use only 0.3-150 million parameters.[15] Such imitation models include Alpaca, Vicuna, Koala, and GPT-4ALL.[15] Despite comparable evaluation on crowd rating and canonical NLP benchmarks and even better performance on simple instructions, the imitation models simply excel at imitating ChatGPT deceiving crowd raters, but actually there is a big gap when it comes to accuracy and breaking down complex tasks.[15]

Zhou et al. (2023) introduces LIMA, a 65B parameter language model fine-tuned on 1,000 curated prompt-response pairs.[16] The model was highly generalizable, and specifically mentions having trip planning ability. Ablation experiment showed diminishing returns with scaling quantity of data, just 30 dialogue examples dramatically improved multi-turn conversation ability, which was measured by the rate of excellent responses going from 45% to 75%.

Lin et al. (2023) propose LLM-EVAL, a unified automatics evaluation method for conversations using LLMs, which is much more simplified and efficient in comparison to current methods using human annotation, ground truth responses, and multiple LLM prompts. LLM-EVAL is a single prompt based evaluation method that uses a unified schema to evaluate conversational quality. Lin claimed that traditional evaluation metrics like BLEU and ROGUE are insufficient for natural conversations. LLM-EVAL outperformed other supervised, unsupervised, and LLM-based evaluation metrics. Supervised techniques include dialogue quality (fluency, relevance, knowledge) and topic transition known as GRADE. Unsupervised techniques include DEB, which is BERT fine-tuned with relevant and adversarial irrelevant responses, and FED, which uses embedding features and probabilities to determine dialogue quality. LLM based methods include GPTScore to assign quality ratings, using InstructGPT, and G-EVAL which uses chain of thought and a form filling paradigm. They use several chat datasets like DSTC10 Hidden Set, including JSALT, NVM, ESL, Topical-DSTC10, and Persona-DSTC10. They took Spearman correlation coefficient to compare the difference between human ratings and automatic metrics across appropriateness, content, grammar, and relevance, and additionally tested and compared the use of LLM-EVAL on different LLMs including Anthropic Claude and OpenAI ChatGPT. Claude and ChatGPT are optimized for chat, and performed better with LLM-EVAL.[17]

Zhao et al. (2023) surfaces the observation that as multimodal capabilities of generative algorithms become world apparent, their explicit knowledge is subjected to hallucination. World knowledge exists in the forms of images, visuals, and audio formats. Each format would require its own domain-specific retrieval-augmented generation method. Specifically for tasks that query for knowledge, there exists 2 main open-ended challenges. Conventionally, one of the challenges involves storing training data within the parameters of a pretrained model. Limitations to this approach include the size constraint of the parameters and the need to update the parameters with new knowledge. The second challenge is pairing a generative model with a retrieval step to achieve better responses from the input data or external references. A remedy for reducing hallucinations is a foundation of standardized knowledge index. However it is structured, this foundation can be transformed into a dense representation that is optimal for efficient storage and search.The recommendation is to integrate retrieval-based practices into the pre-training process so that the generative model can better generalize against new knowledge and adapt accordingly.[7]

Banerjee et al. (2023) found that LLM-powered chatbots are effective platforms to provide services to customers, but there is an important need to assess its performance in efficient and measurable ways. In order to benchmark a chatbot’s performance, accuracy and usefulness will serve as the two main characteristics for evaluation. The first being how accurate the chatbot is able to complete a set of tasks and the second being how useful it is able to fulfill a user’s needs. The proposed metrics is the End to End (E2E) benchmark method using cosine similarity given a set of predefined answers called the “Golden Answers.” The E2E metric compares the chatbot’s output results to an expert human answer, or the “Golden Answer.” They used the Universal Sentence Encoder (USE) and Sentence Transformer (ST) models to create vectorized embeddings and obtain the underlying context and meaning of the texts[18]. Banerjee et al. and Lin et al. argues that traditional metrics like Recall-Oriented Understudy for Gisting Evaluation (ROUGE), which uses n-grams overlaps, are insufficient to capture the deep complexity and semantic meaning of a conversational chatbot.[18][17] E2E is user centric, considers semantic meaning, and improved with advanced prompt engineering alongside human evaluation metrics, unlike ROUGE.[18]

Maroengsit et al. (2019) discusses the difficulties to evaluate and compare different chatbot systems based on their effectiveness, efficiency, ability to satisfy users, and achieve their specific goal or desired prompt. Chatbots have been around since the 1960’s and there has been many technological changes to advance chatbot development. However, there has not been an updated holistic study on evaluation methods of chatbots and there’s a huge potential for chatbot developers to apply these evaluation techniques to their own work. There are two types of chatbots, rule-based and AI-based. Depending on the chatbot type there will be different evaluation methods used to asses those architectures. The main steps of architecture processes in chatbot systems are Natural Language Processing (NLP), Natural Language Understanding (NLU), and Natural Language Generation (NLG). While NLP focuses on text preprocessing via pattern matching, parsing, Term Frequency-Inverse Document Frequency (TF-IDF), or Word2Vec, NLU is processing by obtaining semantic understanding of texts. This is accomplished by having the model understand the conversation between the user and the model itself using the following techniques commonly found in the literature: intent classification, dialogue planning, name entity recognition, vector recognition using cosine similarity, lexicon, or long short-term memory. NLG is response generation, where the point is to gain information from a user’s response by asking them a question and seeing how they reply. Essentially, NLG determines how the system, or agent, responds to the user base from the information gathered in each conversation. There are three main evaluation methods: content evaluation, user satisfaction, and functional evaluation. Content evaluation uses automatic evaluation metrics such as precision and recall using BLEU or ROUGE. This is beneficial because even though having human judges for evaluation is accurate, it takes a long time and is expensive. Thus, it is more efficient and faster evaluation time with lower costs. User satisfaction is based on a Likert five-point scale, which uses human users’ satisfaction surveys to evaluate complex chatbot systems where there is not just one correct answer. However, the issue is the presence of bias. Future chatbot developers can use these results as a guideline when deciding the most well-suited evaluation methods based on their uses cases.[19]

Svikhnushina et al. (2023) proposed a Dialog system Evaluation framework using Prompting (DEP) to specifically evaluate social chatbots with prompt engineering, where the prompts guide the process of how a chatbot generates a response. The issue is that prompt-based learning hasn’t been studied enough when evaluating dialog systems, so they focused on a comprehensive evaluation of dialog chatbot systems using prompting. There are three types of prompting, zero-shot, one-shot, and few-shot prompting. Zero-shot prompting of LLMs do not need previously labeled data to do new tasks, while few-shot prompting requires the LLMs to have some labeled prompts to conduct new tasks. The three evaluation scores they used were “Bad,” “Okay,” and “Good.” The DEP method essentially collects chat logs between the LLM and chatbot system and then prompts the LLM to generate scores based on the chat logs. However, there are several problems such as being unable to “capture subjective perceptions”.[20] when the user chats with the system, and likely is not able to generalize to real-world settings. Therefore, there is a need and importance to optimize social chatbots by streamlining the evaluation process in a more efficient manner. Their solution to streamline the evaluation process was to removing human involvement during each step, saving time and cost. Instead of using LLMs that use prompting, which tend to create misinformation, they gave instructions and prompted LLMs to conduct a specific behavior using the proposed DEP framework, which they found yielded better results. This indicates how you can leverage LLMs to create realistic conversations. Thus, they found that the most well-performing prompts contain few-shot learning and instructions, which show its ability to generalize on a corpora of data.[20]

Irvine et al. (2023) focuses on utilizing human feedback and intuitive metrics to create engaging social chatbots to improve user retention and engagement based on human feedback. In order to aid chatbots in understanding the intentions of a user’s query, feedback was collected automatically generated pseudo-labels during every interaction with the user. These serve as user engagement proxies to train a reward model, so when the chatbot generates a poor sample response during inference time, or the time it takes to complete the forward propagation, then it will be given a low score. These are basically the worst responses that will be rejected because they do not make sense, are not appropriate responses, etc. Four types of metrics were used, such as mean conversation length (MCL) to measure the chatbot’s engagement level with a user, retry rate, user star rating, and retention rate. The retry rate is how often user requests to regenerate a response and the user star rating is essentially user satisfaction feedback based on a rating scale. On the other hand, retention rate is the most challenging and expensive evaluation rate. The problem is that it is manually extensive, costly, and time consuming to have any human involvement such as expert annotators rank responses. Although human experts can improve the responses that LLMs generate, these disadvantages may likely limit the chatbot development and evaluation process instead. Therefore, there’s a lack of engaging chatbots that retains users to come back and talk, so instead of purely fine-tuning LLMs on conversational data, one may incorporate human feedback during development to collect more data for the human reference or knowledge base, which has shown improved retention scores of over 30%.[21] Thus, the solution is to use human feedback to create engaging chatbots and improved engagement levels. This assumes that the longer a user chats with the system, the more likely they are to return and chat in the future, indicating higher retention. It was found that chatbots have longer average user interactions and user retention when grouped training reward models with the human feedback methodology.[21]

Wong et al. (2023) demonstrate how LLMs, particularly ChatGPT, have the ability to drastically change the tourism industry, such as improving customer experience in 3 travel stages: before the trip, en-route, and post-trip encounters. Unlike previous AI, it is able to improve trip planning efficiency, create more personalized recommendations by having it act as a tour guide or local, and having a readily available and streamlined personal guide in your pocket during the trip in cases of emergency. ChatGPT provides efficient travel solutions by looking for travel information, filtering out irrelevant travel information, and then makes a comparison of different options in a neat fashion that is human-readable. The problem is when travelers research where to go using the internet, through travel agencies, or from word of mouth, there is a huge abundance of information. This makes it difficult, overwhelming, and time-consuming to search for valuable information and make decisions on where and what to go or do. There are a lot of communication problems when traveling such as a lack of cultural understanding, language barriers, etcetera. The largest limitation of utilizing LLMs such as ChatGPT for travel, is that it is still not entirely accurate since sources can be misleading. ChatGPT is unable to fully explain their answers in a rational way, bringing up transparency and accountability issues. The responses can be biased bringing up ethical issues and spreading disinformation. This is because there could be prejudiced information in the training data itself and there is limited domain knowledge as it is not able to be trained on certain semantics and ways of human expression, which could cause the LLM to misinterpret the user query. Also, the data is limited to the year 2021 so any information after that date is not in the training set at all. However, despite these limitations, one can utilize the benefits of using LLMs such as ChatGPT across the three stages for a tourist, before, during, and after traveling. These advantages and disadvantages of LLMs in the travel and tourism industry are important takeaways to take into consideration while developing our travel domain-specific chatbot.[22]

Table III provides a high level overview and Figure 3 provides a visualization of the data structure needed for each model. The travel dataset was collected entirely from Reddit, and specifically pulled from travel domain subreddits. There are a plethora of travel subreddits available, with r/travel being the largest and most active with 8.9 million subscribers. It is one of the most popular communities in the top 1% of subreddits as of December 2023.[30] Calls made to the Reddit API to curate a corpus of Question-and-Answer (Q&A) formatted data, collected conversational data from 201 subreddits.[31] This consists of 27 travel-related subreddits, 30 country subreddits, and 144 city subreddits which include r/solotravel, r/travelhacks, r/roadtrip, and so on so the data can capture the context of conversations. The data collected for this project was for the purpose of fine-tuning the chatbot, providing domain specific knowledge, and to provide current updated travel information to address LLM knowledge cutoffs. Due to the Large Language Model (LLM) capabilities of generating recommendations due to its pretraining[32], the Reddit post would serve as the question and the compiled comments as the answer. Tools used for data collection and processing include: Python, PRAW, Pandas, NLTK, Regex, BERTopic, and Ollama.

When choosing the model, LLaMa 2 is one of the models available in different variants, and is having the accuracy of about 68% while the GPT 3.5 has the accuracy of 70%, which is the slight difference, but considering the fact, the size of the model is also comparatively small, as the models has been trained with the huge difference number of parameters. It has performed fairly well in the multiple benchmarks, reference LLaMa 2-Chat 70B passed the helpfulness evaluation on par with GPT-3.5 — precisely, with a 36% win rate and 31.5% tie rate.[36] Hence, given its open source availability, size, less complex, we chose LLaMa 2 7B. Mistral 7B was selected for the same factors, and additionally due to the novelty of being a newly launched model. It had been trained on fewer parameters yet it has performed at par compared to LLaMa1 and LLaMa 2 13 B in most of the benchmarks like Multi-task Language Understanding (MMLU), TrivialQA, etc.[37] Table IV gives a quick look at the comparison of LLM models.

The following techniques, Quantized Low Rank Adapter (QLoRA), Retrieval-Augmented Fine-tuning (RAFT) were passed through LLaMa 2 and Mistral pretrained language models to assess their performance under each proposed approach, resulting to 4 candidate models. Table V outlines the comparisons between the four selected models for this project.

Quantized Low Rank Adapter (QLoRA) Comprehensively tuning a LLM for a particular domain is impractical for individuals due to financial and resource constraints. Therefore, the alternative fine-tuning methodology, QLoRA is employed. Instead of training every single parameter from scratch, LoRA (Low Rank Adaptation), strategically updates a smaller subset of the model’s parameters. And Q focuses on the quantized LLMs that are most important for the model. So together, QLoRA, is an efficient fine-tuning method tailored for quantized LLMs that is more resource-efficient to maintain high accuracy and response quality while reducing the computational and financial cost associated with traditional LLM training. The final input from the training dataset is formatted as a question-and-answer (Q&A) pair structure.

The Figure 18 shows the architecture and data flow for the fine-tuning method, QLoRA.[38] Singh (2023) found that if there is a huge corpus of task-specified datasets that have been labeled, then fine-tuning is preferable over the retrieval method, RAG, especially for domain-specific tasks. For example, specialized topics in the travel domain are extensively lacking labeled data.

Note: Figure is from an article from Keshav Singh, a Data ML Platform Architect Engineer at Microsoft.[38] Curated datasets are ingested by a baselined LLM which contains quantized 8-bit weights from LoRA (or 4-bit weights in QLoRA). When a user makes a query then prompt engineering, or grounding, is done in a cycle between the user and LLM to fine-tune for new data that comes in.[8]

Retrieval Augment Fine-tuning (RAFT) is a novel approach proposed by researchers at UC Berkeley in March 2024. RAFT is a way to combine fine tuning, which memorizes knowledge before running inference like a closed book exam, and RAG, which is like an open book exam that has no prior knowledge or context and just retrieves information. RAFT is a way to prepare for open book exams, helping distinguish and prepare the LLM for both open and closed book exams. Thus, it will be able to answer questions outside the training document domain since it is baked in knowledge from model weights and reads information from retrieved results. RAFT is a specific domain of RAG that can help train and prepare LLMs to perform particularly well in a specified domain. The authors state the following, “RAFT is a training procedure for RAG. If you are doing RAG you should be doing RAFT”.[9]

Currently there are two main approaches to inject new knowledge to LLMs, which are RAG and fine tuning. However, current optimal methodologies for the model to gain new knowledge is still an open question to be solved. The proposed RAFT to domain specific RAG addresses the naive nature of LLMs that are not able to distinguish between context and noise. The goal is to adapt pretrained LLMs for RAG in specific domains, such as ours in travel. Since RAG is highly dependent on the quality of retrieval and the knowledge base may contain irrelevant documents, in RAFT the LLM is fine-tuned to a specific domain (like travel) where a dataset with questions and answers are curated in training. Then, in the inference stage (zero-shot) there are questions (the query, Q) for the trained model and an answer (A) is generated. In the RAG phase, there is the query Q plus the retrieved documents D which will be fed to the LLM, then an answer A is generated.[9]

RAFT takes the whole dataset to train, providing the model with a question, context, and verified answers (instructions), which is then asked to reason using CoT, which then provides the CoT answers. By removing the oracle documents in some instances during training, this helps train the model to memorize domain-specific answers and knowledge instead of deriving them from the context directly. For each sample in training data all there is the question, answer (context), oracle documents (verified, relevant documents), and distractor documents. With this training dataset, they fine-tune LLM and use it to generate answers for given queries in a specific domain. RAFT inference method is similar to the RAG pipeline where it retrieves the top-k documents in the database, however, unlike RAG, RAFT validates the retrieved documents. Zhang (2024) purports the training recipe for RAFT is quite flexible for every enterprise as it can fine-tune RAG models without needing to use GPT-4 all the time, which can be costly and takes time. The dataset generated from training can be used for fine-tuning.[9]

Reinforcement Learning from Human Feedback (RLHF) An RLHF model is trained and fine tuned through human-labeled data as “good” or “bad” responses generated on the chat-model so that the model’s responses can be aligned with the human preference. An RLHF training pipeline for LLMs is done in three main parts: domain-specific fine-tuning, supervised instruction fine-tuning (SFT), and reward modeling in RLHF. Domain-specific fine-tuning contains domain-specific data only, thus, the input is domain knowledge. Then, the second part of the RLHF training pipeline is supervised fine tuning where there is a target label and we fine-tune the travel-domain LLM that is specified on certain tasks and domain-specific pairs such as Q&A pairs, prompts and instructions, and responses curated for our travel domain. The output of domain specific pretraining is a model that is able to recognize the context from input and predict the next words/sentences. The next step is to perform SFT with prompt-text pairs (Q&A pairs) to provide the pretrained LLM with knowledge, specifically travel-domain specific tasks. Thus, it will be able to respond to context-specific questions curated to our specified travel domain. The output of this second phase is a LLM that mimics a conversational agent. The final phase is training to perform RLHF utilizing human feedback on a LLM, where reward model training teaches the LLM to classify responses as good or bad based on a thumbs-up or thumbs-down icon with a binary range of 0 or 1, or smiley faces with a range from 0.0, 0.25, 0.50, 0.75, to 1.0.[29] According to Iyer (2023), the RLHF training pipeline is particularly useful to prevent biased responses from responses generated from training and at inference time, thus, this pipeline can obtain desirable and accurate responses from LLMs and helps with domain-specific LLMs as well.[10] However, it is important to note that human involvement is needed to manually score and screen for responses.

As mentioned in the previous section, researchers took an expedited approach with RLHF data, but the typical process is outlined here. With each question input and response output, ratings are recorded and stored as training sets for the downstream reward model responsible with monitoring the performance of fine-tuned models. After the LLM produces responses to the user, a built-in feedback system for rating the model’s output is provided. Whether accepted or rejected, the feedback is collected to contribute to the Reinforcement Learning from Human Feedback (RLHF) process.[39] Due to observable model or data drift, model prototypes can be trained periodically to update their parameterized knowledge and capabilities. As part of the evaluation process, user feedback and ratings were collected by selecting the top 10 and bottom 10 percent data,with highest and the worst comments to differentiate among the good and bad response to support the classification reward model for RLHF that labels responses as good or bad.Note: We have had collected the user ratings from the interface as well, which is getting saved in the background, this would be used in future studies for the same. Now, The final model that was recognized as optimal is one that maximizes the output of the adversarial reward model.

Model validation is used to evaluate the model’s performance on data it has not been trained on. Specifically, the validation set that was set apart from the training and testing set is used to assess that performance during the training process and tune the hyperparameters of the models. The validation process helps avoid overfitting by ensuring that the model is generalizing well when exposed to new data. The dataset initially had 16,000 rows, but Figure 22 shows this raw dataset was further filtered using the dot-score on question and summaries to get the more relevant information, resulting in about 10,000 rows. Figure 21 shows the dataset split used in Mistral QLoRA with data filtered on greater than 0.65 dot-score. The dataset was split into a 80:20 ratio, where 20% was further split into 50% to produce the validation and testing split. Along with this, another set of Q, A, D using the same dataset as the knowledge corpus, acts as the Oracle Context and creates the distractor context using data augmentation through GPT-4. Figure 23 shows the training testing split of 90:10 with approximately 420 rows. so that the model can learn about the bad data as well, this format of data was fed to a newly proposed model called RAFT. [41]

LLaMa 2 QLoRA: For this model, various experiments were carried out with the dot score threshold, which affects the quality alignment with Question and the cohesive answers like dot score values ranging from: 0.5, 0.6, 0.65, 0.7, 0.75, and picked the better performing model amongst all, with dot score: 0.65. This model has the following hyperparameters:

• •

  r=64: rank, improves the performance of the model as increases the trainable parameters, but it also increases the computational complexity and training time

• •

  alpha=16

• •

  Used all the target layers : q_proj, v_proj, o_proj,gate_proj,up_proj, down_proj

Mistral QLoRA: For this model, the same dot score experimentation was conducted, and the better performing model amongst all was dot score: 0.75. This model has the following hyperparameters, same as prior model:

• •

  r=64

• •

  alpha=16

• •

  Used all the target layers : q_proj, v_proj, o_proj,gate_proj,up_proj, down_proj

The above hyperparameters were used to train the Mistral QLoRA see figure 26 but this time, tried with the cleaner version of the data (filtered data greater than the dot score 0.75) to have a trade-off between data quality versus data quantity; the model trains and performs better if we have clean data. Data quality has more impact over quantity.

LLaMa 2 RAFT: this model has used the following hyperparameters while training:

• •

  r=64

• •

  alpha=16

These parameters were set while training the LLaMa 2 RAFT, results are discussed for this model in the next subsection (see Figure 24). While training the model, as discussed in previous section the dataset filtered with the dot-score: 0.80 (Reddit).

Mistral RAFT: The model was set for the 50 epoch run and has the following hyperparameters, and the model loss is visualized in Figure 28.

• •

  r=64

• •

  alpha=16

• •

  Used all the target layers : q_proj, v_proj, o_proj,gate_proj,up_proj, down_proj

Figure 25 shows the first iteration model’s performance there is no significant loss in the training, the line flattens after step 300, after which the loss is not that significant though it is reducing, the training loss started at step 25 was 2.86 and by the end of the step 200, the total training and validation loss was very insignificant with just 0.4.

Figure 26 shows the final Mistral model is picked from various other iterations runs (with different dot score values like 0.65, 0.7, 0.75), here the model was trained again on better quality dataset, with the dot score 0.75, and the training of the model is comparatively better here from the first iteration, total training loss 1̃.33 and validation loss 1̃.06 there is slight improvement in the validation loss, but this is again leveling off towards the end.

In earlier trials, the loss was approximately 0.7, which shows that there is certain improvement in the performance of the model.

Figure 27 shows more loss than the previous attempt as can be seen here, the change was the cleaner version of the dataset, created the augmented data again on the data where dot score was about 0.80, this shows our hypothesis and the trade off with the cleaner data gives the good loss in training and validation. With the number of epochs being set to 50 and a step size of 25, the training loss at step 25 was 1.44 and validation loss was 1.129. At step 200, the training loss was reduced to 0.15 and validation loss was 0.14, i.e. total loss 1̃.29. This is real-world data with a lot of variability in user comments that do not directly answer the questions in Reddit posts, but since this time data was more aligned to the question and answer present in the Reddit dataset by calculating the dot score, this improved the models’ training from the previous attempts.

In Figure 28, there is a good loss between the train and validation losses. With the number of epochs being set to 50 and a step size of 10, the training loss at step 10 was 2.57 and validation loss was 1.97. At step 110, step the train loss was 0.126 and evaluation loss was 0.196, i.e. total loss 2̃.44 and validation loss 1̃.78. In addition to being real-world data as aforementioned, this can be further explained by users who may have ironically upvoted a comment for being funny, but does not directly answer the question being asked. But while training this model, we have used the highest dot score on the question and the summarized by Falcon to get the cohesive answer from the Reddit comments by the users. Hence, the training on this model was good. This aligns with our evaluation results where RAFT performs the best out of all the models, which will be discussed in subsequent sections.

Upon completion for the assessment of model performance, the aforementioned Mistral model that underwent RAFT training was selected for Reinforcement Learning with Human feedback. A preference dataset curated from pruning Reddit conversations was formatted with chosen and rejected fields as inputs for training. Given input questions, the Mistral RAFT model should be guided to output the preferred answer. Considering the size of the training set and the scope of the training process in terms of expected outputs, the training and evaluation losses converged to zero fairly quickly (see Figure 29).

Using the DPO Trainer module for fine-tuning LLMs with reinforcement learning, there exists a set of metrics including training loss, validation loss, rewards/chosen, rewards/rejected, rewards/accuracies, rewards/margins. While the rewards/chosen fluctuates around zero across 80 steps or 3 epochs, the rewards/rejected decreases and the rewards/margins increases, monotonically. Rewards/accuracies quickly reaches 1 by the 30th step.[42]

Due to the format of Reddit data, the prompts are longer than the generic question and answer pair format. Thus, even with 10,000 rows, QLoRA models took over 30 hours to train and fine-tune. However, with hyperparameter tuning this overhead can be reduced by reducing the size of the data and parameters set such as the step size. For Mistral 7B QLoRA fine-tuning, it took around 8 hours for 3 epochs in the high power computing lab and 33 hours for 67 epochs in the lab.

LLMs have given a new dimension to artificial intelligence and evaluating LLMs. Since we have so many developments in this field every single day, it is important to have good metrics with which to evaluate these models. It is critical to evaluate the LLMs, as these are important to uncover its performance, its challenges, strengths and limitations, and help the developer to work in the certain areas to get better results and performance as per the requirement of the client. Model variants are evaluated with a standardized evaluation process that is defined by consistent prompt templates. A set of specific templates are designed to help capture any risk of hallucinations, noise or inaccurate responses from the two open-source LLM options and processed implementations. In the event that retrieved context is limited and parameters have not been updated to address unknown and niche domain topics, the prompt templates can help ensure that the model is able to admit naivety where appropriate. Evaluation of an LLM is different from machine learning (ML) model evaluation due to the complexity of the task involved, unlike ML models, which have more structured prediction tasks. The evaluation of LLMs is still an ongoing research topic. For the project, as mentioned earlier in the chapter, implemented the following different approaches: QLoRA and RAFT on LLaMa 2 and Mistral, and RLHF on the best model. In addition to evaluating our models, we will also evaluate against pretrained LLaMa 2, Mistral, and ChatGPT, considered to be a state-of-the-art (SOTA), to establish a baseline.

We have the following 7 candidate models on which we have generated inferences for evaluation:

• •

  Mistral: pretrained Mistral, Mistral QLoRA, Mistral RAFT

• •

  LLaMa 2: pretrained LLaMa 2, LLaMa 2 QLoRA, LLaMa 2 RAFT

• •

  Baseline: GPT-4

• •

  RLHF on best model: Mistral RAFT RLHF

We have 4 processed models, and a final 5th model, where RLHF was applied to the best model picked from the 4 processed models.

Appendix shows the overall results of metric evaluations for all models. There are 27 total metrics: 17 quantitative and 15 qualitative. Quantitative metrics are around golden question and answer dot score, cosine similarity, and embedding distances, as well as traditional natural language processing metrics like ROUGE and BLEU, and finally inference generation times. Qualitative metrics include Ragas, and a select subset of OpenAI’s GPT-4 evaluations: coherence, conciseness, helpfulness, and relevance. Each metric has the variance, and was zero for nearly all quantitative metrics. This validates our hypothesis that traditional NLP metrics are insufficient for the complexity of LLMs. For example, although this was a qualitative metric, context recall’s best was 0.219 and worst was 0.113 for a difference of 0.105, out of a 0 to 1 scale. Comparing models on metrics with no variance is not useful. Due to the number of evaluation metrics and the lack of variance in so many of them, this chapter will focus on analyzing the nonzero variance metrics (see Figure 31).

Figure 31 indicates the best model, and next best model if the best is a baseline model) and worst performing models highlighted in green and red respectively. Mistral RAFT was selected as the best model, then further fine-tuned with RLHF, so then Mistral RAFT RLHF is the final best model. Mistral RAFT was the best model by human evaluation of the fine-tuned models, but did not outperform GPT-4 as a baseline. However, Mistral RAFT RLHF did then outperform GPT-4. RLHF results will be discussed in more detail later on. Mistral RAFT was often next best of the fine-tuned models, including for the following metrics: human evaluation, conciseness, helpfulness, and golden answer dot score, and was selected as the worst for faithfulness and context precision. And yet, human evaluation still determined this model to be the best fine-tuned model, which calls into question the validity of the Ragas and quantitative metrics and highlights the importance of keeping humans in the loop when it comes to evaluation.

Human evaluation was carried out by the researchers, ranking inference on a scale of 1 (worst) to 5 (best), normalized to 0 to 1. Figure 31 shows Mistral RAFT RLHF was the best, Mistral RAFT next best, and Mistral QLoRA the worst. QLoRA models were very repetitive and required post processing, RAFT models produced the best answers, but also produced multiple answer options and sometimes instructions to a travel agent, which also needed to be removed with post processing.

Figure 32 shows the Ragas metric against 3 baseline models on the bottom and 5 processed models on the top. For faithfulness, pretrained LLaMa was the best, GPT-4 next best, and Mistral RAFT the worst. It is not surprising that baseline models are the best, but Mistral RAFT was the highest rating by human evaluation. Outside baseline models, Mistral RAFT RLHF was the best. Answer relevancy had pretrained Mistral as the best, LLaMa QLoRA and Mistral QLoRA tied for second, and GPT-4 as worst. Context precision has pretrained LLaMa as best with pretrained Mistral and LLaMa QLoRA as second, and Mistral RAFT as worst, again at odds with human evaluation. Answer correctness has LLaMa QLoRA as best, Mistral RAFT RLHF as next best, and LLaMa RAFT as worst. The final Ragas metric, coherence, has pretrained LLaMa and Mistral RAFT RLHF tied for best and Mistral QLoRA for worst. In general, context recall is low across models, answer relevancy very high across models, correctness and precision are of similar values for each respective model, and visually see the largest variance with faithfulness. Across the Ragas metrics, Mistral RAFT performed the worst, but is the best according to human evaluation, indicating that Ragas metrics are not aligned with human evaluation.

For OpenAI GPT-4 evaluations, we selected a subset: coherence, conciseness, helpfulness, and relevance. Figure 31 shows coherence had pretrained LLaMa and Mistral RAFT RLHF tied for best, and LLaMa QLoRA as worst. Conciseness had GPT-4 as the best, Mistral RAFT next best of experimentation models, and LLaMa QLoRA as the worst. Helpfulness had a simmelian tie amongst pretrained LLaMa, GPT-4, and Mistral RAFT RLHF, and then LLaMa RAFT, Mistral QLoRA, and LLaMa QLoRA as the worst. Relevance has Mistral RAFT RLHF as the best, and pretrained LLaMa next best amongst experimentation models, and LLaMa QLoRA as the worst. OpenAI GPT-4 metrics most aligned with human evaluation rating Mistral RAFT RLHF and Mistral RAFTas best or next best, and LLaMa QLoRA as the worst, unlike Ragas, which did the opposite.

Figure 33 shows across the board, coherence and helpfulness are high while conciseness and relevance are generally low. Given the lengthy answers typically generated by LLMs and the manual post processing, which was performed before human evaluation, but not before the OpenAI evaluation, the low values for conciseness make sense. Given the use case of travel recommendation, relevance is arguably the most important metrics, with Mistral RAFT RLHF having the highest value.

Of quantitative metrics, Figure 31 shows only two quantitative metrics had variance: GPT-4 dot score and golden answer dot score. GPT-4 dot score found Mistral RAFT RLHF as best and LLaMa QLoRA as the worst. Golden answer dot score was most aligned with pretrained LLaMa, Mistral RAFT as next best, LLaMa QLoRA as the worst. There were more dot score comparisons that did not have variance, shown in Figure 34. Figure 35 shows a negative correlation, as expected, with dot score and embedding distance on a scatter plot. The traditional NLP metrics had no variance, but are visualized in Figure 36. The bar graph distribution is nearly the same across all models.

Figure 37 shows all the evaluation metrics as a correlation matrix heatmap with red representing a positive correlation and blue representing a negative correlation. Given the disparity of Ragas metrics with human evaluation, we are particularly interested in analyzing the correlations with human evaluation, which has a positive correlation with the following metrics, OpenAI GPT-4 (coherence, consciousness, helpfulness, relevance), and quantitative metrics with no variances: dot scores, ROUGE1, ROUGE2, BLEU, BERT_R, BERT_F1, cosine similarity, and embedding distance. This indicates that OpenAI GPT-4 metrics are most aligned with human evaluation, whereas Ragas metrics are not. While there were many positive correlations with quantitative metrics, there was no variance, indicating these metrics are too simplistic to fully evaluate the complexity of LLMs inference.

The travel industry currently lacks chatbots that can deliver real-time insights for all questions and concerns related to itinerary planning, safety recommendations, and tips for navigating as a tourist. With the onset of LLMs, the time required to construct an artificial assistant has been accelerated and the potential for building tailored chatbots is becoming more apparent. However, deriving value from new technological advancements with LLMs requires thorough research into proposed techniques and comprehensive evaluations to gauge their utility and usefulness as viable fit-for-purpose tools. To meet the demand for a fit-for-purpose chatbot, various implementation strategies for fine-tuning foundational models and evaluation metrics are explored and evaluated through a series of experimentation.

This research compared two fine-tuning methods (QLoRA and RAFT) on foundational LLMs (Mistral and LLaMa 2 7B), finding that Mistral RAFT outperformed the other three combinations of those models: 1) Mistral QLoRA, 2) LLaMa RAFT, and 3) LLaMa QLoRA. Based on this initial assessment, a final fine-tuning process with Reinforcement Learning with Human Feedback is adapted to further upskill models of interest. Mistral RAFT was further fine-tuned with RLHF and outperformed all models including baseline models (GPT-4, pretrained LLaMa, pretrained Mistral). This refining process yielded an optimal model trained to recognize inputs and contexts in order to deliver inferences aligning with our expectations. Promoting our fine-tuned model that was trained on the RAFT dataset and human feedback as an inference endpoint.

This research also compared LLM evaluation metrics such as the End-to-End (E2E) benchmark method of ”Golden Answers,” traditional natural language processing (NLP) metrics, RAG Assessment (Ragas), OpenAI GPT-4 evaluation metrics, and human evaluation. An evaluation set with 37 questions and “Golden Answers” was formulated synthetically and manually to ensure a diverse set of performances against 4 of our models and 3 baseline models (see Appendix ). While Mistral RAFT RLHF performed best on human evaluation, it performed the worst for some Ragas metrics. Mistral QLoRA performed the worst on most of the quantitative metrics, best on most of the Ragas metrics.

Here are the key findings from our research:

• •

  Quantitative and Ragas metrics do not align with human evaluation.

• •

  OpenAI GPT-4 evaluation metrics most closely align with human evaluation.

• •

  It is essential to keep humans in the loop for evaluation, as traditional NLP metrics are insufficient.

• •

  Mistral generally outperforms LLaMa.

• •

  RAFT outperforms QLoRA but still requires post-processing.

• •

  RLHF improves model performance significantly.

To conduct this research, a travel dataset was sourced from the Reddit API by querying travel-related subreddits to gather travel conversation prompts and personalized travel experiences, and then augmented for each fine-tuning method including formats like Q&A format, RAFT, and RLHF. These datasets are available for further research. As part of the data cleaning process, a dot score, typically used for quantifying semantic similarity, was employed to potentially reduce the amount of noise in raw datasets. By iteratively transforming for higher quality inputs, the model’s ability to learn and generalize improved. Much of the experimentation involved fine-tuning with QLoRA where preferred examples for probable questions and responses were provided to the model. Ultimately through successive iterations, 5 fine-tuned models: 1) LLaMa 2 QLoRA, 2) Mistral QLoRA, 3) LLaMa 2 RAFT, 4) Mistral RAFT, 5) Mistral RAFT RLHF were publicly deployed on HuggingFace (see Figure 40).