Identifying Key Terms in Prompts for Relevance Evaluation with GPT Models

The relevance evaluation between a query and a passage has been a fundamental task since the inception of ranking systems. This assessment has historically been conducted in a binary manner, categorizing results as either relevant or non-relevant, but has evolved to include graded relevance scales offering more detailed evaluations.

In the realm of traditional IR, the reliance on human assessors for relevance judgment has been extensively documented [1, 3]. Despite their ability to provide nuanced evaluations, this approach has been criticized for its time and cost inefficiencies, as well as the subjective variability in results it can produce [20, 21].

The advancement of machine learning and its integration into IR has marked a transition towards automated relevance judgment. This area, particularly the use of transformer-based models like BERT, has been the focus of recent research [7]. The challenge, however, lies in achieving a balance between the precision offered by human assessment and the scalability of automated methods.

The introduction of LLMs, especially GPT-3 and GPT-4, has further transformed the landscape of relevance judgment. Initial studies, such as those by [32] and [8], explored the use of GPT-3 in annotation tasks, including relevance judgment. Sun et al. [30]’s research extends this to examining GPT-3’s broader capabilities in data annotation. In a distinct approach, MacAvaney and Soldaini [19] investigated the use of LLMs for evaluating unassessed documents, aiming to improve the consistency and trustworthiness of these evaluations. Complementing this, Thomas et al. [31] delved into the integration of LLMs for comprehensive relevance tagging, highlighting their comparable precision to human annotators. On the contrary, Faggioli et al. [10] has presented theoretical concerns regarding the exclusive use of GPT models for independent relevance judgment.

While extensive research has been conducted in this field, the specific influence of terms within a prompt on relevance evaluation remains unexplored. This study seeks to bridge this gap by investigating the impact of individual terms used in prompts.

Recent advancements in LLMs have emphasized their capability for in-context learning, classified as either few-shot or zero-shot based on the presence of in-context examples. Few-shot learning, where a model is given a limited set of examples, has historically shown superior performance over zero-shot learning, which relies on instructions without examples, as highlighted by Brown et al. [4].

The “pre-train and prompt” paradigm emphasizes the distinction between few-shot prompts (conditioned on task examples) and zero-shot prompts (template-only). While few-shot learning was traditionally favored, recent studies, including those on GPT-4, suggest that zero-shot approaches can sometimes outperform few-shot methods, particularly in specific domains [13, 22].

In our study, to investigate the terms in prompts, we conduct experiments using both few-shot and zero-shot settings and compare their outcomes.

Prompt engineering has emerged as a critical factor in harnessing the full potential of LLMs across various natural language processing applications. The formulation of a prompt is instrumental in guiding an LLM’s output, significantly influencing its performance in diverse tasks [27, 4]. The art of crafting effective prompts involves meticulous design and strategic engineering, ensuring that prompts are precise and contextually relevant [25, 11, 28].

The increasing complexity of LLMs has spurred interest in developing sophisticated prompt tuning methods. These methods often utilize gradient-based approaches to optimize prompts over a continuous space, aiming for maximal efficiency and efficacy [17, 24]. However, the practical application of these methods can be limited due to constraints such as restricted access to the models’ gradients, particularly when using API-based models. This challenge has led to the exploration of discrete prompt search techniques, including prompt generation [2], scoring [33], and paraphrasing [12].

In the broader context of prompt-learning, or “prompting,” the approach is increasingly recognized as a frontier in natural language processing, seamlessly bridging the gap between the pre-training and fine-tuning phases of model development [14, 9]. This technique is particularly valuable in low-data environments, where conventional training methods may be less effective [26, 15, 23].

Within the realm of prompt-learning, two primary strategies are employed: few-shot and zero-shot learning. Liang et al. [16] demonstrated a few-shot technique for generating relevance, while studies like those by Sun et al. [30] and Dai et al. [6] have successfully applied few-shot learning in various scenarios. Conversely, Ding et al. [9] suggested that with an appropriate template, zero-shot prompt-learning could yield results surpassing those of extensive fine-tuning, emphasizing the power and flexibility of well-engineered prompts.

So far, there has been little focus on the terms within a prompt in existing research. This study is important because even small changes in a prompt can lead to different results. Our research, which concentrates on word terms, can be considered a form of micro-level prompt engineering.

To evaluate the effectiveness of each prompt in the relevance evaluation task, an objective metric is required. For this purpose, we decided to use the similarity between the evaluations conducted by humans and those conducted by the LLM using the prompt. To measure the similarity between the two sets of evaluations, we utilize Cohen’s kappa ($\kappa$) coefficient, a statistical measure for inter-rater reliability that accounts for chance agreement. This measure compares the agreement between relevance labels generated by the LLM and human judgments, reflecting the quality of the prompt. Higher kappa values indicate a stronger alignment between the LLM and human evaluations. The Cohen’s kappa coefficient is calculated using the following formula:

In this equation, $P_{o}$ represents the observed agreement between the two sets of evaluations, and $P_{e}$ is the expected agreement by chance. The kappa value ranges from -1 to 1, where 1 indicates perfect agreement, 0 no agreement other than what would be expected by chance, and -1 indicates total disagreement. A higher kappa value suggests that the LLM’s relevance evaluations are more closely aligned with human assessments, indicating a higher quality of the prompt in guiding the LLM to make evaluations similar to those of human judges.

We utilize two types of prompts, as shown in Table 7 of Appendix B. The first type consists of prompts named with an ‘M’, sourced from previous research [16, 30, 10]. The second type includes prompts generated using the template in Table 1, which are named with a ‘G’. After assessing the performance of both prompt types, we aim to determine which prompts perform better. Following the experiments, we will analyze whether there are any terms common to the more effective prompts. If common terms are identified, it would suggest that these terms play a crucial role in the effectiveness of the prompt.

We conduct the experiments under both zero-shot and few-shot settings. Few-shot examples, derived from  [10], are illustrated in Table 0.A of Appendix A. These few-shot examples consist of four instances: two are positive examples, and the other two are negative ones. To ensure a fair comparison, we apply the same set of few-shot examples across all prompts.

We analyze which terms are beneficial for relevance evaluation. Initially, we compare the performance of the prompts illustrated in Table 7. We then categorize the prompts into those with high performance and those with lower performance and look for distinguishing characteristics in each group. To identify the specific terms that play a role, we utilize the analysis prompts provided in Table 1. Furthermore, we compare how the results of each group vary depending on the presence or absence of few-shot examples.

We advance our analysis by constructing confusion matrices for the prompts, allowing for a more in-depth evaluation of their impact. Through the examination of precision and recall values derived from these matrices, we gain insights into the roles played by different terms within the context of relevance evaluation.

The evaluation of prompt efficacy in relevance assessments, as outlined in Table 3, reveals notable trends. A key observation is the significant performance variation among semantically similar prompts, highlighting the impact of subtle differences in prompt design on evaluation outcomes. For example, although M3 and G3 are similar prompts asking if the query and passage are ‘relevant,’ they yield different results. Moreover, despite all prompts addressing the relevance between the query and passage, their outcomes vary substantially.

When comparing results between GPT-3.5 and GPT-4 across both few-shot and zero-shot settings, Prompts M1, G7, G8, and G9 consistently rank in the top five across both GPT-3.5-turbo and GPT-4, indicating their inherent effectiveness. Conversely, certain prompts consistently underperform in both models. Specifically, prompts M4, G1, and G3 are found in the bottom five, underscoring elements that may detract from the efficacy of relevance evaluations.

Examining the performance of individual models reveals distinct characteristics in response to the prompts. Each model demonstrates unique preferences in prompt efficacy, illustrating that LLMs may respond differently to the same prompt structures. Certain prompts show high efficacy in GPT-3.5-turbo, while others perform better in GPT-4. Notably, GPT-4 generally exhibits superior performance compared to GPT-3.5-turbo across a range of prompts. A particular case of interest is prompt G1 in the zero-shot setting, where GPT-4’s performance is the only instance of falling behind GPT-3.5-turbo. Aside from this case, GPT-4’s performance is generally superior to that of GPT-3.5-turbo.

Further statistical analysis, involving a paired t-test on the averages of the top five and bottom five prompts, reinforces these findings. Specifically, the top five prompts in GPT-3.5-turbo had an average performance of 0.386, while in GPT-4, this average was higher at 0.452. Conversely, the bottom five prompts averaged 0.274 in GPT-3.5-turbo and 0.351 in GPT-4. These results indicate a statistically significant difference in performance at a 95% confidence level, emphasizing the pivotal role of prompt design in influencing the effectiveness of relevance evaluations in LLMs.

In our analysis, we utilized the template from Table 1 to identify key terms in prompts that play a significant role in relevance evaluation using LLMs. The findings are summarized in Table 4.

We observed that prompts demonstrating top performance commonly used the term ‘answer’ or its variations. For instance, in M1, the prompt asks if the passage ‘answers’ the query. Similarly, G7 and G9 emphasize whether the passage contains or directly ‘answers’ the query. This pattern is also evident in G10, where the prompt focuses on whether the passage ‘correctly answers’ the query.

On the other hand, prompts associated with lower performance frequently included the term ‘relevant’ or related terms. For example, M3’s prompt requires indicating if the passage is ‘relevant’ for the query, while G1 asks if the query and passage ‘relate’ to the same topic. This trend continues in M4 and G3, where the term ‘relevant’ is central to the prompt’s structure.

These findings indicate that the choice of key terms in prompts significantly impacts the performance of LLMs in relevance evaluation tasks. Terms like ‘answer’ seem to guide the LLM towards more effective evaluation, while the use of ‘relevant’ appears to be less conducive for this purpose.

The differences in performance between zero-shot and few-shot models for GPT-3.5-turbo and GPT-4 are illustrated in Figure 2, which presents the average results for each approach. From this analysis, we can discern two interesting observations.

Firstly, there is a notable variation in performance across the top and bottom five performers between the two model versions. In the case of GPT-3.5-turbo, while there is an improvement in the performance of the bottom five prompts (from an average of 0.274 in zero-shot to 0.291 in few-shot), the top five prompts exhibit a decrease in performance (from 0.386 in zero-shot to 0.352 in few-shot). This indicates that while few-shot examples enhance GPT’s ability to handle previously lower-performing prompts, they might detrimentally affect the performance of the highest-performing prompts.

In contrast, GPT-4 shows a consistent improvement in both the top and bottom performers with few-shot examples. The top five prompts improve from an average of 0.452 in zero-shot to 0.468 in few-shot, and the bottom five improve from 0.351 to 0.425. This shows that few-shot examples enhance the overall performance in evaluation tasks with GPT-4.

Secondly, both models demonstrate a reduction in the performance gap between the top and bottom five prompts with few-shot learning. This convergence is more pronounced in GPT-4, which sees a more significant increase in performance for the bottom five prompts. It suggests that few-shot examples is particularly effective in refining the model’s responses to less optimal prompts, leading to a more consistent performance across different types of prompts.

Given the role of few-shot examples in providing clearer instructions and context, these results suggest that GPT-4 is more adept at adapting to varied prompt structures and content than GPT-3.5-turbo.

The analysis of confusion matrices in Table 5 provides key insights into the effectiveness of different prompt types in relevance evaluation. This analysis highlights G6, which had the highest performance, G1 with the lowest performance, and G10, known for its use of the term ‘correctly.’

G6, achieving the highest performance, questions if the passage provides ‘an answer’ to the query. This prompt led to a significant agreement between LLM predictions and human assessors, as evident by a high Cohen’s kappa value of 0.528, along with strong precision and recall. The high number of true positives (43) and true negatives (116) in G6’s matrix suggests that focusing on ‘answering’ is highly effective in evaluating the relevance of the passage to the query.

Conversely, G1, which demonstrated the lowest performance, focuses on whether the query and passage ‘relate’ to the same topic. Despite its high recall, this prompt yielded a lower Cohen’s kappa value of 0.275. The comparatively fewer true negatives (56) against G6 indicate that a broader ‘relevance’ focus may lead to less precise evaluations.

G10, with its emphasis on whether the passage ‘correctly answers’ the query, shows a distinct performance, marked by a Cohen’s kappa value of 0.495. Its precision is notably high, but the recall is somewhat limited, suggesting that while it is effective in identifying specific relevant answers, it may overlook some broader aspects of relevance.

This comparison underlines the varying effectiveness of prompts based on their focus in the context of information retrieval. Prompts like G6, with an ‘answering’ focus, tend to lead to more accurate and precise evaluations, while ‘relevance’-focused prompts like G1 might not capture the entire scope of the query-passage relationship. G10’s specific focus on ‘correctly answering’ demonstrates a particular effectiveness in identifying precise answers but at the potential expense of broader relevance. Therefore, the choice of key terms and their emphasis is crucial in designing prompts for efficient retrieval and ranking in LLMs.

As discussed in the previous section, defining ‘relevance’ in the context of LLM prompts varies significantly in its scope. G10’s approach, using the term ‘correctly answers’, tends to give a slightly narrow definition in relevance evaluation. It focuses on whether the passage precisely addresses the query, potentially overlooking broader aspects of relevance.

On the other hand, we explored a more balanced approach with G6’s prompt. This prompt, focusing on whether the passage provides ‘an answer’ to the query, strikes a middle ground. It covers not just the direct answer but also the broader context, leading to a more comprehensive consideration of relevance.

Conversely, G1’s prompt offers the broadest definition of relevance by asking if the query and passage ‘relate’ to the same topic. This wide approach, while inclusive, risks being too expansive. As reflected in the confusion matrix for G1 in Table 5, this broad definition results in high recall but at the cost of lower precision, as it captures a wide net of potentially relevant information, including false positives.

This analysis highlights the need for a balanced definition of relevance in prompt design. While G1’s broad approach increases recall, its precision suffers. G10’s narrow focus may miss broader relevance aspects. In contrast, G6’s approach appears to offer a more optimal balance. It captures a wide array of relevant information without being overly narrow or inclusive, leading to more accurate and balanced performance in relevance evaluations. These findings are pivotal for crafting prompts that precisely measure the relevance of information in LLM-based retrieval and ranking systems.

As can be seen in Figure 2, in GPT-3.5-turbo, the performance of zero-shot is slightly higher than that of few-shot. In contrast, in GPT-4, the performance of few-shot exceeds that of zero-shot. This variation indicates that a conclusive determination of the relative impacts of few-shot and zero-shot approaches is complex and model-dependent.

However, there is a characteristic that appears consistently in both models: the use of few-shot examples reduces the performance gap between the top-5 and bottom-5 groups. In GPT-3.5-turbo, the gap decreased from 0.112 to 0.061, and in GPT-4, it nearly halved from 0.101 to 0.043. These results suggest that few-shot examples help in defining unclear aspects in the bottom-5 instructions. For instance, consider the case of the G1 prompt. In the zero-shot setting, GPT-4 shows a low performance of 0.209, but when few-shot examples are used, the performance dramatically increases to 0.409. This could indicate that while the term ‘relate’ in G1 has a broad meaning, the use of few-shot examples helps in clarifying its interpretation.

An emerging area of interest is the potential for using LLMs directly as rankers in IR, rather than just for generating relevance judgments. However, the practical application of LLMs as direct rankers faces significant challenges, primarily due to efficiency concerns. Directly ranking with LLMs, especially when reliant on API calls, can be slow and costly, as it requires repeated, resource-intensive interactions with the model for each ranking task. This approach, therefore, becomes impractical for large-scale or real-time ranking applications.

Given these constraints, future research in this domain should consider the development and utilization of downloadable, standalone LLMs. Such models, once sufficiently advanced, could potentially be integrated directly into ranking systems, offering a more efficient and cost-effective solution compared to API-dependent models. This shift would allow for the direct application of LLMs in ranking tasks, potentially overcoming the limitations currently posed by API reliance. However, this path also necessitates further advancements in LLM technology to ensure these models can operate effectively and reliably in a standalone capacity.