Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach

Given a pair of texts (condition-text, generated-text) that represent the text used to condition the LLM to its generation. We want to detect if a given generated-text is a hallucination.

Given a set of pairs of texts of the type (condition-text, generated-text) from an LLM (we will call it the LLM-Generator ($LLM_{G}$)), we extract four numerical features based on the generated tokens and vocabulary tokens probabilities from another LLM that we call the LLM-Evaluator ($LLM_{E}$).¹¹1Which could be the same as $LLM_{G}$.

Then, using these four features, we trained two different classifiers: a Logistic Regression (LR) and a Simple Neural Network (SNN). Finally, we evaluate these classifiers on a test set they did not see before. Figure 1 illustrates the process.

This section provides more details on each feature extracted. Every feature is computed using token probabilities and the vocabulary probability distribution corresponding to each token on the generated-text. These can be formally defined as follows: (i) The token probability of each token of the Vocabulary of the $LLM_{E}$ corresponding to $t_{i}$ as $P_{LLM_{E_{i}}}(v_{k})=(v_{k}|t_{i-1},...,t_{1},c_{m},...,c_{1};\theta)$ for every $k$. (ii) The token with the highest probability at position $i$ in the generated-text according to $LLM_{E}$ as the $v^{*}=\arg\max_{k}P_{LLM_{E_{i}}}(v_{k})$. (iii) The token with the lowest probability at position $i$ according to $LLM_{E}$ as the $v^{-}=\arg\min_{k}P_{LLM_{E_{i}}}(v_{k})$.

Next is a natural language description of the four features and, the mathematical definition:

• •

  Minimum Token Probability (mtp): Minimum of the probabilities that the $LLM_{E}$ gives to the tokens on the generated-text.

• •

  Average Token Probability (avgtp): Average of the probabilities that the $LLM_{E}$ gives to the tokens on the generated-text.

• •

  Maximum $LLM_{E}$ Probability Deviation (Mpd): Maximum from all the differences between the token with the highest probability according to $LLM_{E}$ at position $i$ and the assigned probability from $LLM_{E}$ to $t_{i}$ which is the token generated by $LLM_{G}$.

• •

  Minimum $LLM_{E}$ Probability Spread (mps): Maximum from all the differences between the token with the highest probability according to $LLM_{E}$ at position $i$ ($v^{*}$) and the token with the lowest probability according to $LLM_{E}$ at position $i$ ($v^{-}$).

Formally, these features can be defined as:

These four numerical features are inspired by the mathematical investigation of the GPT model in [13], and recent results in [10, 15], suggesting there is a correlation between the minimum token probability on the generation, the average of the token probabilities, and the average and maximum entropy.

Lee et al. [13] proposes that a reliable indicator of hallucination during GPT model generation is the low probability of a token being generated. This is based on the assumption that forcing the model to generate such a low-probability token occurs when the difference between the token with the highest probability and all other tokens is less than a small constant $\delta$. Here, $mps$ is an estimator to avoid the cost of calculating differences across a large vocabulary and generated text.

Additionally, Azaria et al. [14] trained a simple classifier called Statement Accuracy Prediction, based on Language Model Activations (SAPLMA) that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement. The authors showed that SAPLMA, which leverages the LLM’s internal states, performs better than prompting the LLM to state explicitly whether a statement is true or false. Different from [14], Su et al. [15] introduced MIND, an unsupervised training framework that leverages the internal states of LLMs for hallucination detection requiring no manual annotations.

Diverging from these three previous papers [14, 13, 10, 15], the approach here differs in several aspects. This is an empirical paper, not a theoretical one like [13]. We do not use Self-Consistency as [10] does, and also, our approach is not zero-shot or few-shot learning. Our approach follows supervised learning like [14]. However, instead of using the contextual embeddings and hidden layers, we only use four features that result from aggregations of the token and vocabulary probabilities. We also test a simpler Logistic Regression classifier besides a Simple Dense Neural network.

Moreover, instead of using only the LLM generating the text ($LLM_{G}$), the argument is made that depending on the task and model type, different LLM-Evaluators ($LLM_{E}$) can provide consistent but quantitatively different results than using probabilities from $LLM_{G}$. The belief is that probabilities from a different model, varying in architecture, size, parameters, context length, and training data, can also serve as reliable indicators of hallucinations in the text generated by $LLM_{G}$. Since $LLM_{E}$ and $LLM_{G}$ are not always the same, an additional numerical feature, $Mpd$ (Maximum $LLM_{E}$ Probability Deviation), is introduced. This feature indicates the difference between the maximum probability token in the vocabulary of $LLM_{E}$ and the token generated by $LLM_{G}$.

Using different LLMs as evaluators takes advantage of the diversity of training data among different language models (LLMs) and captures various linguistic patterns and styles. Detecting hallucinations in LLM outputs, such as in $LLM_{G}$, might be possible through analyzing probability distributions. Yet, specific patterns may remain undetectable, potentially addressed by other models specialized in particular topics. Evaluations from multiple models enhance robustness by mitigating biases inherent in individual models’ training data.

In the previous section, we described the numerical features selected, but the process of extracting these features is still ongoing. To extract the features, we used $LLM_{E}$ models that can be used for the Conditional Generation Task. Notably, in our case, it is a force decoding since the tokens of generated-text were potentially generated by a different LLM ($LLM_{G}$). Instead of letting the model generate the answer token-by-token from the condition-text alone, we provide it with the token predicted by $LLM_{G}$ at each step. This way, $LLM_{E}$ is forced to follow the path to generate the generated-text and, from there, extract the token probabilities from $LLM_{E}$ if it would generate that sequence itself. Then, using these token probabilities, we compute the four numerical features previously described.

The classifiers used are a Logistic Regression [28] (LR) and a Simple Neural Network (SNN). Both for a data point of the type (condition-text, generated-text) only use the four numerical features extracted. We selected the LR for its simplicity, fast training, and effectiveness in binary classification tasks. However, we implemented a SNN to explore complex non-linear relationships in the data. The SNN architecture consists of an input layer with four neurons representing each features, followed by two hidden layers, each comprising 512 neurons with the ReLU activation function. Followed by an output layer, containing a single neuron activated by a sigmoid function, suitable for binary classification tasks.

Here, we list the three datasets we are using for experimentation and comparison.

• •

  HaluEval: Hallucination Evaluation for Large Language Models (HaluEval) benchmark is a collection of generated and human-annotated hallucinated samples for evaluating the performance of LLMs in recognizing hallucinations. HaluEval includes 5,000 general user queries with ChatGPT responses and 30,000 task-specific examples (10,000 per task) from three tasks: question answering, knowledge-grounded dialogue, and text summarization [12].

• •

  HELM: Hallucination detection Evaluation for multiple LLMs (HELM) benchmark is a list of 3582 sentences from randomly sampling 50,000 articles from WikiText-103 [29] where the selected LLMs were tasked with prompt-based continuation writing. The resulting sentences were annotated as hallucination or not [15].

• •

  True-False: Comprises 6,084 sentences divided into the topics of “Cities,” “Inventions,” “Chemical Elements,” “Animals,” “Companies,” and “Scientific Facts.” All sentences in each category are conformed of true statements and false statements [14]. Unlike HaluEval, this dataset only has generated-text and does not include any condition-text.

The LLMs selected as evaluators to study the impact of factors such as architecture, training method, and training data include GPT-2, its large version (gpt2-large) [30]; Bidirectional and Auto-Regressive Transformers (BART), its CNN-Large version (bart-large-cnn) [31]; Longformer Encoder-Decoder (LED) [32], with the version fine-tuned on the arXiv dataset (led-large-16384-arxiv). Also, we used four bigger LLMs like OPT-6.7B (OPT) [33], GPT-J-6.7B (GPT-J) [34], LLaMA-2-Chat-7B (LLC-7b) [16] and Gemma-7b (Gemma) [35]. We used the known transformers library.²²2https://huggingface.co/ In most cases, we utilized the Conditional Generation setup. For GPT-2, we employed the GPT2LMHeadModel setup. Additionally, when forwarding inputs to these models with a pair of (condition-text, generated-text), we encountered the challenge of context limitation, which varied depending on the LLM. To address this issue, we did not truncate the generated-text if possible. Instead, if truncation was necessary (with a truncation length of truncate_len), we removed the excess from the condition-text. If additional knowledge was included, we evenly split the truncation between the knowledge and the condition-text.

To train LR, we used the sklearn library³³3https://pypi.org/project/sklearn/ with the Limited-memory Broyden-Fletcher-Goldfarb-Shanno Algorithm solver [37] and default parameters. The SNN was trained during $10^{4}$ epochs. We used the Adam [38] optimizer with learning rate of $10^{-3}$. All experiments, including feature extraction, training, and evaluation of the classifiers, were conducted on an NVIDIA L40S GPU core with 48GB of memory. Given a dataset and a single $LLM_{E}$, training takes anywhere from 30 minutes (HELM) to 4.5 hours (HaluEval), depending on the size of the training data and the length of the condition-text and generated-text.

We performed experiments using single numerical feature to determine which features were significant or not to the results. Table V showed for three tasks in HaluEval and three $LLM_{E}$ how the results in accuracy were affected by which features were used or not. In most cases, the use of all features provides the best results. Then, when each feature was used alone, we discovered that the most meaningful features in our approach were mtp and avgtp, especially in Summarization and QA. However, in the KGD task, it was more important for bigger models like GPT-J and LLC-7b, the feature introduced by this paper: $Mpd$.

Additionally, in the Appendix, we conduct a feature analysis using the coefficients obtained from the LR classifier.