Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression

We use $P$ and $C$ to represent the original and compressed prompt respectively. Likewise, we use $N$ and $\widetilde{N}$ to represent the length of the original and compressed prompt. We use $\eta=\widetilde{N}/N$ to represent the compression rate and $1/\eta$ as the compression ratio. $\eta^{*}$ is used to represent the target compression rate. The graph is represented by $\mathcal{G}=\{(e_{i},r_{i},e_{i}^{{}^{\prime}})\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}\}$, where $e_{i}$, $r_{i}$ and $e_{i}^{{}^{\prime}}$ represent the subject entity, relation and object entity in the graph respectively; $\mathcal{E}=\{e_{1},e_{2},\cdots,e_{m}\}$ and $\mathcal{R}=\{r_{1},r_{2},\cdots,r_{n}\}$ denote the set of entities and relations in $\mathcal{G}$. $g_{i}=(e_{i},r_{i},e_{i}^{{}^{\prime}})$ is used to represent small-scale information elements in $\mathcal{G}$, equivalent to graph triplet. $M$ represents auxiliary models used for graph construction. $E$ is used to represent the encoder network. $\delta$ represents the similarity threshold used for sub-graph construction.

In this work, we aim to design and develop an effective prompt compression strategy that can cut down the prompt text by only preserving the requisite information content while at the same time maintaining the semantics and end performance of the prompt to the best possible extent.

Formally, we aim to generate a compressed prompt $C=\{c_{i}\}_{i=1}^{\widetilde{N}}$ given the original prompt $P=(p^{\mathrm{ins}},p^{\mathrm{info}},p^{\mathrm{que}})$, where $p^{\mathrm{ins}}=\{p_{i}^{\mathrm{ins}}\}_{i=1}^{N^{\mathrm{ins}}}$, $p^{\mathrm{info}}=\{p_{i}^{\mathrm{info}}\}_{i=1}^{N^{\mathrm{info}}}$, and $p^{\mathrm{que}}=\{p_{i}^{\mathrm{que}}\}_{i=1}^{N^{\mathrm{que}}}$, denote the prompt instruction, information and question, respectively; $\widetilde{N}$, $N^{\mathrm{ins}}$, $N^{\mathrm{info}}$ and $N^{\mathrm{que}}$ represent the number of tokens in $C$, $p^{\mathrm{ins}}$, $p^{\mathrm{info}}$, $p^{\mathrm{que}}$ and respectively. We denote $N=N^{\mathrm{ins}}$ + $N^{\mathrm{info}}$ + $N^{\mathrm{que}}$ as the length of the original prompt.

Prompt-SAW is motivated by the observation that the key information within the prompt text could be inferred as a set of entities and relations, which can also be organized into a graph structure, commonly known as a knowledge graph in literature.

Formally, given a prompt text $P$, we claim it encompasses a set of entities $\mathcal{E}$, i.e., names of persons, locations, organizations, miscellaneous elements, etc., (Ali et al., 2020). These entities serve as the key elements of the prompt structure. In addition to the entities, we can also infer some relations $\mathcal{R}$ in $P$ that may be used to describe the connections between the entities. Prompt-SAW re-organizes these key elements of the prompt (i.e., entities and their relations) in a graph structure, represented by $\mathcal{G}=\{(e_{i},r_{i},e_{i}^{{}^{\prime}})\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}\}$. We use $g_{i}=(e_{i},r_{i},e_{i}^{{}^{\prime}})$ to represent the $i$-th information element of $\mathcal{G}$, i.e., a fact stating that $e_{i}$ has $r_{i}$-$th$ relation with $e_{i}^{{}^{\prime}}$.

We argue this transformation of text information to graph is a more reasonable and natural approach as: (i) It helps in highlighting the key information elements in the prompt. (ii) Later, analyzing these key entities in combination with underlying relations helps in filtering/digging out the salient content within the prompt to come up with a compressed prompt.

The workflow of Prompt-SAW consists of two parts. First, it uses the information in prompt $P$ to construct a graph $\mathcal{G}$. Then, based on the specific scenario, we proceed as follows:

Further details about the model components of Prompt-SAW are provided in the following subsections.

Results for Task-agnostic Settings. For task-agnostic settings, we report the results of Prompt-SAW for GSM8K-aug in Table 1. Note, unlike existing research that reports their performance for one fixed setting, we report these results for $i$-shot settings, where $i$ indicates the number of prompts have been employed by Prompt-SAW, i.e., {1, 2, 4 and 8}-shots.

Comparing these results against the baseline models, we can observe that Prompt-SAW outperforms the previous state-of-the-art by a significant margin. For instance, compared to the best performing baselines, Prompt-SAW improves the EM score by up to 7.3%, 10.1%, 5.5% and 4.2% under 1-shot, 2-shot, 4-shot and 8-shot settings, respectively. Correspondingly reduction in the prompt size is 32.3%, 34.9%, 33.0% and 32.9%. We attribute such drastic performance improvement to the following factors: (1) Prompt-SAW retains the logical integrity of the prompts by sub-dividing the original prompts into smaller comprehensive information elements; (2)Prompt-SAW benefits from the workflow that allows selecting and omitting individual information elements for prompt compression without destroying the overall information structure of the compressed prompt. These help Prompt-SAW to ensure the utility of the compressed prompt for the end task.

The results of Prompt-SAW on ShareGPT (in Table 3) show for GPT3.5-turbo as target LLM, Prompt-SAW improves the Rouge-1 score by up to 29.3%, 34.9% and 38.6%. The performance for Rouge-2 and Rouge-L exhibit a similar behavior. For these results, we also observe for LLaMA2-7B-chat, the improvement in performance is relatively lower compared to that of GPT3.5-turbo. A probable justification in this regard is the fact that LLaMA2-7B-chat is more influenced by the change in context for the compressed prompt. Whereas, higher performance on GPT3.5-turbo indicates that Prompt-SAW preserves the critical information in the prompt.

Correlating the results for both settings, we observe that compared to the task-agnostic scenarios, Prompt-SAW yields better performance for task-aware settings, especially for NaturalQuestions. This is due to the fact that it is more difficult to dig out the latent correlation between information elements within the prompt’s internal structure rather than explicit task-aware correlation extraction.

From Table 2 and Figure 3 we can also find that the performance of Prompt-SAW drops when the value for the target compression rate $(\eta^{*})$ decreases from 0.5 to 0.1. A probable justification that the actual compression ratio of Prompt-SAW is significantly higher than the target compression ratio when the target compression ratio is 10, i.e., $\eta^{*}=0.1$. This is owing to the fact that Prompt-SAW only retains the information elements in $\mathcal{G}$ as the key/basic information units for the compression process. It will delete some entities and relations that may be highly similar to the problem but their overall structure is too long, leading to relatively poor performance for a higher compression ratio.

In this section, we perform an in-depth analysis of the performance of Prompt-SAW.

For NaturalQuestions, GPT3.5-turbo achieves up to 19.8% higher Acc scores compared to that of LLaMA2-7B-chat. Likewise for ShareGPT, it achieve up to 47.55% higher value for Rouge-1. We attribute this result to GPT3.5’s stronger ability to understand and comprehend context, resulting in generating higher quality response for the prompts compressed by Prompt-SAW.

For instance, as shown in Example D.1 (Appendix D.1), the prompt compressed by LLMLingua encompasses grammatical incoherent text, such as: {“List of Nobelates in The first Prize1 Wilhelmrad, of who received82 in en prize”}. Lack of grammatical coherence significantly undermines the readability and/or interpretability of the compressed prompt, thus impacting its end-utility. Whereas, the prompt compressed by Prompt-SAW, relying on knowledge graph triples exhibits a consistent grammatical sentence structure, as shown in the lower half of Example D.1.

To further support our claims, we also conducted a quantitative comparison. Specifically, we assess the fluency of the compressed prompts through the computation of a weighted mean of bi-gram and tri-gram entropies (Meng et al., 2022). Its computational details are given in Appendix C.3, and result is reported in Table 4. These results show that Prompt-SAW yields relatively higher fluency scores than the baseline models. A lower score for baseline models, e.g., LLMLingua, is attributable to loss of intrinsic semantic relationship between the tokens for the compressed prompt.

In order to compute the computational overhead of Prompt-SAW, we assume embedding and computing similarity takes a constant amount of time and its cost may be ignored because it is much smaller than that of LLM. Specifically, we use the following formulation to study the computational efficiency of Prompt-SAW:

where $c_{\text{graph}}$ represents the per-token computation load to generate knowledge triples and $c_{\text{LLMs}}$ is per-token computation for target LLMs to generate final output respectively. $L$ represents length of the original prompt, $\eta^{*}$ is the target compress rate, and $c$ represents the total computational overhead for compression.

Following the assumption of Jiang et al. (2023a), we estimate $c_{\text{graph}}$ and $c_{\text{LLMs}}$ based on model parameters, as: $c_{\text{graph}}\approx 0.3/175c_{\text{LLMs}}\approx 0.0017\cdot c_{\text{LLMs}}$ using OpenIE toolkit as the knowledge extraction approach. When $\eta^{*}=0.2$, we have $c\approx 0.2017L\cdot c_{\text{LLMs}}$. This means we can achieve nearly 5x savings in terms of computational resources. For the cases employing a small LLM for graph construction, $c_{\text{graph}}\approx 3.8/175c_{\text{LLMs}}\approx 0.02\cdot c_{\text{LLMs}}$, the computational overhead is $c\approx 0.22\cdot c_{\text{LLMs}}$, which is only slightly higher than the first case.

Overall, these results show the computation efficiency of our model is comparable with that of Jiang et al. (2023a). On the other hand, Prompt-SAW offers much higher benefits, i.e., compressing prompts without distorting their readability for end-users, while at the same time preserving their end-utility to the best possible extent.