Epistemic confidence, the Dutch Book and relevant subsets

The idea of relevant subset appeared in Fisher’s writings on the nature of probability (Fisher, 1958). He considered probability meaningful provided there is no relevant subset. However, he treated the condition as an axiom – appealing to our intuition as to what ‘meaningful’ means – and did not establish what conditions are needed to guarantee no relevant subset. To avoid unnecessary philosophical discussions, we’re limiting the idea of relevant subset to the confidence interval procedures but not to the probability concept in general.

Intuitively, we could use the coverage probability $\gamma$ as a betting price if there is no better price given the data at hand. So the question is, are there any features of the data that can be used to improve the price? Mathematically, these ‘features’ are some observed statistics that can be used to help predict the true coverage status. Given an arbitrary statistic $S(y)$, the conditional coverage probability $P_{\theta}(\theta\in\mbox{CI}|S(y))$ will in general be biased, i.e., different from the marginal coverage. However, the bias as a function of the unknown $\theta$ is generally not going to be consistently in one direction. For example, trivially, if we use the full data $S(y)=y$ itself, i.e., fixing the data as observed, then the conditional coverage is either one or zero depending on the true status of the CI, hence completely non-informative. In terms of betting, this means we cannot exploit an arbitrary feature of the data as a way to construct a Dutch Book against someone who sets the price at $\gamma$. The betting motivation also appeared in Buehler (1959) and Robinson (1979), though they only assumed two people betting each other repeatedly, but not the existence of the betting market. As we shall discuss in Example 1 and after Theorem 1, this has a significant impact on the interpretation of epistemic confidence.

A statistic $R(y)$ is defined to be relevant (cf. Buehler, 1959) if the conditional coverage is non-trivially and consistently biased in one direction. That is, for a positive bias, there is $\epsilon>0$ free of $\theta$, such that

Now, potentially the feature $R(y)$ can be used to construct a Dutch Book: Suppose you and I are betting, and I notice that the event $[R(y)]$ occurs. If you set the price at $\gamma$, then I would buy the bet from you and then sell it in the betting market at $\gamma+\epsilon$. So I make a risk-free profit of $\epsilon$. (We have assumed that the market contains intelligent players, so they would also have noticed the relevant statistic and set the price accordingly.) Similarly, for the negative bias, the relevant $R(y)$ has the property

Technically, $R(y)$ induces subsets of the sample space, known as the ‘relevant subset’; for convenience, we use the terms ’relevant statistic’ and ’relevant subset’ interchangeably. So, if there is a relevant subset, the confidence level $\gamma$ is not epistemic. Conversely, if there are no relevant subsets, the betting price determined by the confidence level is protected from the Dutch Book. So, mathematically, we establish epistemic confidence by showing that it corresponds to a coverage probability that has no relevant subsets.

Here the range $R\equiv R(y)\equiv y_{(2)}-y_{(1)}$ is relevant. If $R=2$ we know for sure that $\theta$ is equal to the midpoint of the interval, so the CI will always be correct. But if $R=0$, the CI is equal to the point $y_{1}$, and it falls with equal probability at the integers $\{\theta-1,\theta,\theta+1\}$. So, for all $\theta$, we have

In the betting market, the range information will be used by the intelligent players to settle prices at these conditional probabilities. We can be sure, for example, that if $y_{1}=3$ and $y_{2}=5$, the intelligent players will not use 7/9 as the price and will instead use 1.00. So, the information can be used to construct a Dutch Book against anyone who ignores $R$ and unwittingly uses the unconditional coverage. How do we know that there is a relevant subset in this case? Moreover, given $R$, how do we know if there is no further relevant subset?

To contrast with the classical Ramsey-de Finetti Dutch Book argument, suppose $y_{1}=y_{2}=3$. If for whatever subjective reasons, you set the price 7/9 for $[\theta\in\mbox{CI}]$, you are being internally consistent as long as you set the price 2/9 for $[\theta\not\in\mbox{CI}]$, since the two numbers constitute a valid probability measure. Internal consistency means that I cannot make a risk-free profit from you based on this single realization of $y$. Even if I know based on the conditional coverage that 1/3 is a better price, I cannot take any advantage of you because there is no betting market. So 7/9 is a valid subjective probability.

Now consider Buehler-Robinson’s setup, again assuming no betting market and supposing $y_{1}=y_{2}=3$. They would say the marginal price 7/9 is a bad idea, because there is a relevant subset giving a conditional probability 1/3. In a series of independent repeated bets, if you set the price 7/9 whenever $y_{1}=y_{2}=3$, I will be happy to ‘sell’ you the bet and be guaranteed to win in long term. This is the usual frequentist interpretation; in any single bet I am not guaranteed a risk-free money. As previously described, the presence of the betting market allows me make free-money from a single realization of $y$. So, the threat of the market-based Dutch Book is more potent. The exact technical difference between Buehler-Robinson’s and our setup will be discussed below after Theorem 1, where the former allows one to choose an arbitrary prior distribution. $\Box$

It turns out that establishing a no-relevant-subset condition relies on the concept of a confidence. Let $t\equiv T(y)$ be a statistic for $\theta$, and define the right-side P-value function

Assuming that, for each $t$, it behaves formally like a proper cumulative distribution function, $C_{m}(\theta;t)$ is called the confidence distribution of $\theta$. The subscript $m$ is used to indicate that it is a ‘marginal’ confidence, as it depends on the marginal distribution of $T$. For continuous $T$, at the true parameter, the random variable $C_{m}(\theta;T)$ is standard uniform. For continuous $\theta$, the corresponding confidence density is

The functions $C_{m}(\theta;t)$ and $c_{m}(\theta)$ across $\theta$ are realized statistics, which depend on both the data and the model, but not on the true unknown parameter $\theta_{0}$. We can view the confidence distribution simply as the collection of P-values or CIs. Suggestively, and with a slight abuse of notation, we define

to convey the ‘confidence of $\theta$ belonging in the CI’.

We assume a regularity condition, called R1 below, that for any $\alpha\in(0,1)$, the quantile function $q_{\alpha}(\theta)$ of $T$ is a strictly increasing function of $\theta$. Then the frequentist procedure based on $T$ gives a $\gamma$-level CI defined by

for some $\gamma_{2}>\gamma_{1}>0$ with $\gamma_{2}-\gamma_{1}=\gamma$, to have a coverage probability

Here the coverage probability is a frequentist probability based on the distribution of $T$, whereas the confidence is for the observed interval $\mbox{CI}(t)$ based on the confidence density of $\theta$. The confidence becomes

Thus, we have the following lemma.

Fisher (1950) was against the idea of interpreting the level of significance as a long-term frequency in repeated samples from the same population. For $i=1,\cdots,n,$ suppose $T_{i}$’s are estimates for $\theta_{i}$’s from different populations. Let $X_{i}=I(\theta_{i}\in CI(T_{i}))$ such that $\gamma=C_{m}(\theta_{i}\in CI(t_{i}))$ and let $\bar{X}=\sum X_{i}/n.$ Then, $\bar{X}=\gamma+O_{p}(1/\sqrt{n}),$ so that $\gamma$ can be a long-term frequency of true coverage from different populations or experiments.

Fisher (1930, 1933) called $C_{m}(\theta;t)$ the fiducial distribution of $\theta$, but he required $T$ to be sufficient. However, the recent definition of the confidence distribution (e.g. Schweder and Hjort, 2016, p.58) requires only $C_{m}(\theta;T)$ to be uniform at the true parameter, thus guaranteeing a correct coverage probability. Lemma 1 states when Fisher’s fiducial probability $C_{m}(\theta;t)$ becomes a frequentist probability, which requires $T$ to be continuous. When $T$ is discrete, the equality is only achieved asymptotically; see Appendix A3 for an example.

However, as shown in Example 1, a correct coverage probability does not rule out relevant subsets. This means that the current definition of the confidence distribution does not guarantee epistemic confidence. The key step is to define a confidence distribution that uses the full information. Motivated by the Bayesian formulation and Efron (1993), first define the implied prior as

where $m(t)$ cancels out all the terms not involving $\theta$ in ${c_{m}(\theta;t)}/{L(\theta;t)}$. In this paper, the full confidence density is defined by

The subscript $f$ is now used to indicate that it is associated with the full likelihood based on the whole data. When necessary for clarity, the dependence of the confidence density and the likelihood on $t$ and on the whole data $y$ will be made explicit. $c_{f}(\theta)$ is defined only up to a constant term to allow it to integrate to one. Obviously, if $T$ is sufficient, then $c_{m}(\theta)=c_{f}(\theta)$, but in general they are not equal. In Section 3, we show a more convenient way to construct $c_{f}(\theta)$. The confidence parallel to (5) can be denoted by $C_{f}(\cdot)$. Thus, the full confidence density looks like a Bayesian posterior. However, the implied prior is not subjectively selected, and can be improper and data-dependent.

The full confidence density $c_{f}(\theta)$ is used in general to compute the degree of confidence $\gamma$ to any observed $\mbox{CI}(y)$ as

The CI has a coverage probability, which may or may not be equal to $\gamma$. We say that $c_{f}(\theta)$ has no relevant subsets, if there is no $R(y)$ such that the conditional coverage probability is biased in one direction according to (1) or (2).

For our main theorem, we assume the following regularity conditions, the proof is given in Appendix A1. For completeness and easy access, R1 is restated in full here.

• R1.

  $T=T(Y)$ is a continuous scalar statistic whose quantile function $q_{\alpha}(\theta)$, defined by

  $$P_{\theta}(T\leq q_{\alpha}(\theta))=\alpha,$$

  is strictly increasing function of $\theta$ for any $\alpha\in(0,1)$.

• R2.

  $c_{0}(\theta)$ is positive and locally integrable on the parameter space $\Theta$ such that

  $$\int_{J}c_{0}(\theta)d\theta<\infty,\ \ \text{for any compact subsets}\ \ J\subseteq\Theta.$$

• R3.

  $\log c_{0}(\theta)$ is uniformly continuous in $y$.

• R4.

  The confidence interval $\mbox{CI}(y)=(b_{L}(y),b_{U}(y))$ is locally bounded, i.e., for any compact set $K$ in the sample space of $y$, there exist $M_{1}$ and $M_{2}$ such that

  $$|b_{L}(y)|\leq M_{1}\ \text{and}\ |b_{U}(y)|\leq M_{2}\quad\text{for any }y\in K$$

Note we have two ways of computing the price of an observed CI: using $C_{f}(\theta\in\mbox{CI})$ or using $P_{\theta}(\theta\in\mbox{CI})$. The latter is not guaranteed to be free of relevant subsets, while the former is not guaranteed to match the coverage probability. If the two are equal, we have a confidence that corresponds to a coverage probability that has no relevant subsets, hence epistemic confidence. If $T$ is sufficient and satisfies R1, Lemma 1 implies that the frequentist CI satisfies

Thus, we summarize the first key result in the following corollary:

In Example 2, on observing $t$ from $N(\theta,1)$, $c_{m}(\theta)=c_{f}(\theta)=\phi(t-\theta).$ Furthermore, we also have $L(\theta)=\phi(t-\theta)$, so the implied prior $c_{0}(\theta)=1$. The coverage probabilities match the full confidence, and by Corollary 1, the confidence is epistemic.

We note that $P_{\theta}(\theta\in\mbox{CI}(Y))=C_{f}(\theta\in\mbox{CI}(y))$ holds asymptotically, regardless whether $y$ is continuous or discrete. Corollary 1 specifies the conditions where it is true in finite samples.

For more generality, it is actually more convenient to prove the theorem using $c_{0}(\theta)\equiv c_{0}(\theta;y)$ an arbitrary function that satisfies R2 and R3, as long as it leads to a proper $c(\theta;y)$. In particular, it does not have to be an implied prior (8) that depends on the statistic $T$. If $c_{0}(\theta)$ is a proper probability density that does not depend on $y$, then $c_{f}(\theta)$ is a Bayesian posterior density, shown already by Robinson’s (1979) Proposition 7.4 not to have relevant subsets. For proper priors, $\log c_{0}(\theta)$ is trivially uniformly continuous in $y$, so the theorem extends his result to improper and data-dependent priors.

However, there is a significant impact on the interpretation. If you use an arbitrary $c_{0}(\theta;y)$ that is not the same as the implied prior, and there is a betting market, your price $\gamma$ will differ from the market price. So, as illustrated in the Introduction and in Example 1, I can construct a Dutch Book against you. This means that, in this case, the theorem is meaningful only for two people betting repeatedly against each other, with gains or losses expressed in terms of expected value or long-term average. It is exactly the setting described by Buehler (1959) and Robinson (1979). Crucially, in such a setting, the presence of relevant subsets does not guarantee an external agent to make a risk-free profit from a single bet. In this sense, it does not satisfy our original definition of epistemic confidence.

Lindley (1958) showed that, assuming $T$ is sufficient, Fisher’s fiducial probability – hence the marginal confidence – is equal to the Bayesian posterior if and only if the family $p_{\theta}(y)$ is transformable to a location family. However, his proof assumed $c_{0}(\theta)$ to be free of $y$. Condition R3 of the theorem allows $c_{0}(\theta)$ to depend on the data, so our result is not limited to the location family.

The current definition of confidence (e.g. Schweder and Hjort, 2016, p.58) only requires $C_{m}(\theta;T)$ to follow uniform distribution. However, if $T$ is not sufficient, the marginal confidence is not epistemic, because it does not use the full likelihood, so it is not guaranteed free of relevant subsets. Limiting ourselves to models with sufficient statistics to get epistemic confidence is overly restrictive, since sufficient statistics exist at arbitrary sample sizes in the full exponential family only (Pawitan, 2001, Section 4.9). Using non-sufficient statistics implies a potential loss of efficiency and epistemic property. Further progress depends on the ancillary statistic, a feature or a function of the data whose distribution is free of the unknown parameter. As reviewed by Ghosh et al. (2010), it is one of Fisher’s great insights from the 1920s. We first have a parallel development for the conditional confidence distribution given the ancillary $A(y)=a$:

As for the marginal case, we have the following corollary from Lemma 1. Condition R1 needs a little modification, where it refers to the conditional statistic $T|a$ for each $a$.

Furthermore, define the implied prior as

where $m(t,a)$ cancels out all the terms not involving $\theta$ in ${c_{c}(\theta;t|a)}/{L(\theta;t|a)}$. As before, the full confidence is $c_{f}(\theta)\propto c_{0}(\theta)L(\theta;y).$

Suppose $T(y)=t$ is not sufficient but $(t,a)$ is, where $a$ is an ancillary statistic. In this case, $a$ is called an ancillary complement, and in a qualitative sense it is a maximal ancillary, because

Thus, conditioning a non-sufficient statistic by a maximal ancillary has recovered the lost information and restored the full-data likelihood. In particular, the conditional confidence becomes the full confidence: $c_{c}(\theta;t|a)=c_{f}(\theta)$. Note that (12) holds for any maximal ancillary, so if a maximal ancillary exists, then the full likelihood is automatically equal to the conditional likelihood given any maximal ancillary statistic. In its sampling theory form, when $t$ is the maximum likelihood estimator (MLE) $\hat{\theta},$ full information can be recovered from $p_{\theta}(\hat{\theta}|a),$ whose approximation has been studied by Barndorff-Nielsen (1983).

In conditional inference (Reid 1995), it is commonly stated that we condition on the ancillary to make our inference more ‘relevant’ to the data at hand, in other words, more epistemic. But this is typically stated on an intuitive basis; the following corollary provides a mathematical justification. Since we already condition on $A(y)$, a further relevant subset $R(y)$ is such that the conditional probability $P_{\theta}(\theta\in\mbox{CI}|A(y),R(y))$ is non-trivially and consistently biased in one direction from $P_{\theta}(\theta\in\mbox{CI}|A(y))$ in the same manner as (1). As we describe following Theorem 1 above, the result holds for an arbitrary $c_{0}(\theta)$ that satisfies R2-R3 and leads to a valid confidence density. So it applies to $c_{0}(\theta)$ defined by (11). Following a similar reasoning as for the previous corollary, we can state our second key result:

Remark: In view of (12), the confidence is epistemic for any choice of the maximal ancillary. Basu (1959) showed under mild conditions that maximal ancillaries exist. However, they may not be unique; this is an issue traditionally considered most problematic in conditional inference. If the maximal ancillary is not unique, then the conditional coverage probability might depend upon the choice. However, this does not affect the absence of relevant subset guaranteed by the corollary. We discuss this further in Section 4 and illustrate with an example in Appendix A4.

The MLE of $\theta$ is not unique, but we can choose $\widehat{\theta}=\bar{y}$ as the MLE. It is not sufficient, but $(\bar{y},R)$ is, so $R$ is a maximal ancillary. Indeed the full confidence values match the conditional probabilities given the range $R$ as previously given. Furthermore, according to Corollary 3, there is no further relevant subset, so the confidence is epistemic. $\Box$

Now let’s consider trying to bet the value $\lfloor\theta\rfloor$, the largest integer smaller than $\theta$, based on observing $y$. What price would you give to the bet that $\lfloor\theta\rfloor=\lfloor y\rfloor=1$? According to the confidence distribution, it should be 0.9. But we can show that the random variable $\lfloor y\rfloor$ is Bernoulli shifted by $\lfloor\theta\rfloor$ and with success probability $\langle\theta\rangle\equiv\theta-\lfloor\theta\rfloor$, the fractional part of $\theta$. For example, if $\theta=1.6$ then $\lfloor\theta\rfloor=1$ and $\langle 1.6\rangle=0.6$, so $\lfloor y\rfloor$ is equal to $1+0=1$ or $1+1=2$ with probabilities 0.4 and 0.6, respectively. This means that, in general, using $\lfloor y\rfloor$ as a guess, the ‘coverage’ probability of being correct is

This probability varies from 0 to 1 across $\theta$, not matching the specific confidence – such as 0.9 above – derived from the confidence density.

The problem is that $\lfloor y\rfloor$ is no longer a sufficient statistic, so its marginal distribution is not fully informative. Now, the fractional part $\langle y\rangle$ is uniform between 0 and 1 for any $\theta$, so it is an ancillary statistic. We can show that, conditional on $\langle y\rangle$, the distribution of $\lfloor y\rfloor$ is degenerate: with probability 1, it is equal to $\lfloor\theta\rfloor+1$ if $\langle y\rangle<\langle\theta\rangle$; and equal to $\lfloor\theta\rfloor$ if $\langle y\rangle>\langle\theta\rangle$. This conditional distribution is distinct from the unconditional version, so $\langle y\rangle$ is relevant. Basu (1964) and Ghosh et al. (2010) used this example as a counter-example, where conditioning by an ancillary leads to a puzzling degenerate distribution. But actually, it is not so puzzling: The conditional likelihood is the same as the full likelihood, so $\langle y\rangle$ is maximal ancillary. This is of course as we should expect, since $\langle y\rangle$ together with $\lfloor y\rfloor$ form the full data $y$.

To illustrate with real numbers, for example, on observing $y=1.9$ we have $\langle y\rangle=0.9$, so $\lfloor\theta\rfloor=\lfloor y\rfloor=1$ if the unknown $\langle\theta\rangle<0.9$. Now, your betting situation is much clearer: you will bet that $\lfloor\theta\rfloor=1$ if you believe that $\langle\theta\rangle<0.9$, i.e. $1<\theta<1.9$. This is exactly the same logical situation you faced before with the full likelihood and confidence density. $\Box$

Suppose $y_{1},\ldots,y_{n}$ are an iid sample from the location family with density

where $f(\cdot)$ is an arbitrary but known density function, for example the Cauchy or normal densities. Immediately, based on $y_{1}$ alone, the confidence density is

so the implied prior $c_{0}(\theta)=1$. So, again using formula (13), the full confidence density is

This is a remarkably simple way to arrive at the confidence density of $\theta$ without having to find the MLE and its distribution.

Without further specifications, the MLE $T\equiv\widehat{\theta}$ is not sufficient, so the marginal P-value $P_{\theta}(T>t)$ will not yield the full confidence. The distribution of the residuals $(y_{i}-\theta)$ are free of $\theta$, so the set of differences $(y_{i}-y_{j})$’s are ancillary. In his classic paper, Fisher (1934) showed that

where $a$ is the set of differences from the order statistics $y_{(1)},\ldots,y_{(n)}$. This means that the conditional likelihood based on $\widehat{\theta}|a$ matches the full likelihood (14), and the confidence of CIs based on (14) will match the conditional coverage probability. Indeed, here $(\widehat{\theta},a)$ is sufficient and $a$ is maximal ancillary. Overall, the confidence of CIs based on (14) is epistemic.

Suppose the dataset $y$ is an iid sample from the exponential family model with log-density of the form

The MLE is sufficient if $J=1$, but not if $J>1$. In the latter case, the family is called the curved exponential family. By Theorem 1, when $J=1$ confidence statements based on the MLE will be epistemic. (Our theory covers the continuous case in order to get exact coverage probabilities. Many important members are discrete, which is more complicated because the definition of the P-value is not unique, and the coverage probability function is guaranteed not to match any chosen confidence level. We discuss an example in Appendix A3.)

The standard evaluation of the confidence requires the tail probability of the distribution of the MLE, which in general has no closed form formula. Barndorff-Nielsen’s (1983) approximate conditional density of the MLE $\widehat{\theta}$ is given by

where the MLE is the solution of $A^{\prime}(\theta)=\sum_{i}\sum_{j}h^{\prime}_{j}(\theta)t_{j}(y_{i})$, $a$ is the maximal ancillary and $k$ is a normalizing constant that is free of $\theta$. For $J=1$ and the canonical parameter $h_{1}(\theta)=\theta$, the ancillary is null, and the approximation leads to the right-side P-value

where $Z$ is the standard normal variate and

with $w=2\log\{L(\widehat{\theta})/L(\theta)\}$ and $I(\widehat{\theta})$ the observed Fisher information.

To get the marginal confidence density based on the P-value formula (17), we need

where $\widehat{\theta}$ is the solution of

with $\psi(\theta)\equiv\partial\log\Gamma(\theta)/\partial\theta$, and the observed Fisher information is

The circle points in Figure 1(b) are the marginal confidence density based on the same sample above. As expected, this tracks almost exactly the one given by formula (13). The corresponding implied prior based on $c_{m}(\theta)/L(\theta)$ is given in Figure 1(a), also closely matching the implied prior based on $n=1$. $\Box$

Here $(\sum y_{i}^{2},\sum y_{i})$ is minimal sufficient, and the likelihood function is

The MLE is given by

with a maximal ancillary