How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis

To study the causal effect of the different input words, we build a Structured Causal Model (SCM) for the missing words generation process and apply interventions on some input words to estimate their effect quantitatively. We consider the missing words as outcome words and the remaining words that hold a certain association (e.g. positionally close) with the outcome words as treatment words. Then, we can formally represent the word generation process with SCM, as the following structural equations:

$$\begin{split}&w_{c}=f(\mathbb{I}),w_{t}={\rm PLM}(w_{c})\\ &w_{o}={\rm PLM}(w_{c},w_{t}).\end{split}$$

(1)

Separate the words in a sentence into three groups: treatment words $W_{t}$, outcome words $W_{o}$, and context words $W_{c}$ (specified by $w_{t}$, $w_{o}$, and $w_{c}$ respectively). Equation 1 formulates the following data generation process: (1) Sample a sentence from the natural text space $\mathbb{I}$ and get the context words $w_{c}$ using function $f$. (2) Generate the treatment words $w_{t}$ by the PLM based on $w_{c}$ only. (3) Generate outcome words $w_{o}$ based on both the $w_{c}$ and $w_{t}$.

To obtain the quantitative causal effect of treatment words $W_{t}$ on outcome words $W_{o}$, we apply the do-calculus $do()$ Pearl (2009) on treatment words $W_{t}$ to introduce interventions for estimating the causal effect. $do()$ denotes the operation of forcibly setting the value of $W_{t}$. Then the causal effect of $W_{t}$ on $W_{o}$ can be estimated by the Average Treatment Effect (ATE) Rubin (1974):

$$\begin{split}&\mathbb{E}\left[P(W_{o}|do(W_{t}=\hat{w}_{t}))\right]\\ -&\mathbb{E}\left[P(W_{o}|do(W_{t}=w_{m}))\right].\end{split}$$

(2)

Accordingly, we define ATE for PLMs as:

$$\begin{split}\tau&=\sum_{\mathbb{I}}{\rm PLM}\left(do(W_{t}=\hat{w}_{t}),w_{c}\right)P(s)\\ &-\sum_{\mathbb{I}}{\rm PLM}\left(do(W_{t}=w_{m}),w_{c}\right)P(s),\end{split}$$

(3)

where $\hat{w}_{t}$ is the ground truth of the treatment words $W_{t}$ (the original value of $W_{t}$ without intervention). $w_{m}$ is the intervention value (several [mask]s) for $W_{t}$, and we use it to simulate removing the ground-truth value $\hat{w}_{t}$ from the input. $P(s)$ denotes the probability of selecting the sample $s$ that consists of $w_{t}$, $w_{o}$, and $w_{c}$ from $\mathbb{I}$. $\rm PLM(\cdot,\cdot)$ denotes the output of PLMs with certain input. Table 1 illustrates the interventions on the SCM for different associations.

The raw output of PLMs is a probability distribution over fixed vocabulary. We transform the output into reciprocal rank to quantify the differences:

$${\rm PLM}_{k}{(w_{t},w_{c})}=\begin{cases}\frac{1}{rank_{\hat{w}_{o}}},&\text{if }rank_{\hat{w}_{o}}\leq k\\ 0,&\text{otherwise}\end{cases}.$$

(4)

$\hat{w}_{o}$ is the ground-truth outcome words. $rank_{\hat{w}_{o}}$ is the rank position of $\hat{w}_{o}$ according to the generation probability of $\hat{w}_{o}$ output by ${\rm PLM}(w_{c},w_{t})$. We set $k$ to 100 and use ${\rm PLM_{100}}$ to replace ${\rm PLM}$ in Equation 3 to calculate ATE. The ATE reflects the effect of $W_{t}$ on the prediction of $W_{o}$, it can be regarded as a quantitative estimation of how much PLMs depend on $W_{t}$ when generating $W_{o}$.

Wikipedia is a rich source of knowledge Thom et al. (2007); Hassanzadeh (2021), and most of the PLMs nowadays have been pre-trained on Wikipedia Devlin et al. (2019); Liu et al. (2019); Lan et al. (2019), so we take Wikipedia sentences as pre-training samples to construct the probing samples for dependence measure. We probe the mask-filling on these sentences to analyze what PLMs based on when capturing factual knowledge in pre-training.

The outcome we want to observe is the predictions of factual words in the sentences. In order to locate the factual words, we align each sentence with a triplet (subject, predicate, object) in the KB. The words that correspond to the object serve as outcome words $W_{o}$ for observation, and the remaining words that hold an explicit association with $W_{o}$ are marked as treatment words $W_{t}$ for intervention. For different associations, the $W_{t}$ is identified as:

1. 1.

   Knowledge-Dependent: all the remaining words correspond to the predicate and object in the same triplet with the $W_{t}$.

2. 2.

   Positionally Close: the remaining words closest to $W_{o}$.

3. 3.

   Highly Co-occurred: the remaining words that have higher Pointwise Mutual Information (PMI) Church and Hanks (1990) with $W_{o}$. The PMI is calculated over all the Wikipedia sentences using the following equation:

   $${\rm PMI}(w_{i};\hat{w}_{o})=\frac{P(w_{i}|\hat{w}_{o})}{P(w_{i})},$$

   (5)

   where $\hat{w}_{o}$ is a group of words (a span) and $w_{i}$ is a single word.

4. 4.

   We further define a Random (R) association, where the $W_{t}$ are randomly selected remaining words. It provides some empirical support for how much the modifications in the context affect the mask-filling output.

Accordingly, one sentence yields four probing samples for the four associations, respectively. The four probing samples share the same $W_{o}$ but use different words as $W_{t}$ to show the dependence on different associations when predicting the same $W_{o}$. We preserve that the number of words in $W_{t}$ is the same among different associations. For example, if there are two words used as $W_{t}$ by the KD association, we select the top two closest words with $W_{o}$ as $W_{t}$ for PC, and the words have the top two PMI with $W_{o}$ for HC. We can obtain a set of probing samples for each association. The sample sets for different associations source from the same set of sentences and have the same size.

Section 2.2 uses the original Wikipedia sentence as pre-training samples to quantify the dependence PLMs used to capture the corresponding fact in pre-training. The performance of capturing the corresponding fact is probed by having PLMs fill masks on crafted quires. We construct these queries by instantiating templates on triplets Petroni et al. (2019). $T_{i}(s)$ denotes the $i$-th query for the fact corresponds to $s$. The accuracy ${\rm mrr}$ of capturing this fact is obtained by averaging over all the predictions obtained with different queries:

$${\rm mrr}\left(s\right)=\frac{1}{n}\sum_{i}^{n}{\rm PLM}_{k}({T_{i}(s)}),$$

(6)

${\rm PLM}_{k}({T_{i}(s)})$ denotes the reciprocal rank of the ground truth in the PLM’s output for query $T_{i}(s)$, it is defined in Equation 4.

The consistency of the capture is indicated by the percentage of the pairs of queries that have the same result Elazar et al. (2021a):

$${\rm con}\left(s\right)=\frac{\sum_{i\neq j}\mathds{1}_{{\rm PLM}({T_{i}(s)})={\rm PLM}(T_{j}(s))}}{n(n-1)}.$$

(7)

There are $n$ different queries on every fact, and we can get $\binom{n}{2}=n(n-1)$ pairs of predictions in total. ${\rm PLM}({T_{i}(s)})$ denotes the top-1 output for the query $T_{i}(s)$. The value of $\mathds{1}_{{\rm PLM}({T_{i}(s)})={\rm PLM}(T_{j}(s))}$ is an indicator function that takes the value 1 if the PLMs returns identical at top-1 for ${T_{i}(s)}$ and ${T_{j}(s)}$ and 0 otherwise. The PLMs are better on the consistency metric if they keep the predictions consistent when queries only vary on the surface forms. E.g., the two queries “Dante was born in [mask]” and “The birthday of Dante is [mask]” should return the same results.

Finally, we evaluate the factual knowledge capture performance by jointly examining the accuracy and consistency Elazar et al. (2021a):

$${\rm test}(s)={\rm mrr}\left(s\right)\cdot{\rm con}\left(s\right)$$

(8)

${\rm test}(s)$ measures the probing performance on template-based queries. We also define a metric to measure how well the PLMs memorize the missing words in pre-training samples (Wikipedia sentences):

$${\rm train}(s)={\rm PLM}_{k}(s).$$

(9)

We have quantified the dependence on each association and defined the metrics for probing performance in the above sections. We then calculate the Pearson correlation coefficient Kirch (2008) between dependence and probing performance to reveal the effectiveness of different associations. An association is considered more effective if the probing performance positively correlates with its dependence more.

Because only part of the facts has available templates, the samples in the dependence measure without templates are ignored in the calculation. The factual knowledge captured by different PLMs may vary significantly due to the differences in model scale, pre-training data, or other settings. To make the correlation coefficient comparable between different PLMs, we calculate the correlation only on the factual knowledge gathered correctly by the PLM. I.e., only the pre-training samples with ${\rm train}(s)={\rm PLM}_{k}(s)=1$ are involved.