Robust Bayesian methods using amortized simulation-based inference

To make our discussion easier we define some notation. For a model with parameter $\theta\in\Theta$ for data $y$, let $0\leq l(\theta)\leq u(\theta)$ be two functions, which we call lower and upper bound functions, and assume that

$$\int l(\theta)\,d\theta>0\;\;\;\text{and}\;\;\;\int u(\theta)\,d\theta<\infty.$$

Writing $\pi(\theta)$ for a possibly unnormalized density (i.e. a density that does not integrate to one), we follow Rinderknecht et al., (2014) and write $\hat{\pi}(\theta)$ for the normalized verison of $\pi(\theta)$ when this exists. The density ratio class with lower bound $l(\theta)$ and upper bound $u(\theta)$ is

$\displaystyle\psi_{l,u}:=\left\{\hat{\pi}(\theta)=\frac{\pi(\theta)}{\int\pi(\theta)\,d\theta};l(\theta)\leq\pi(\theta)\leq u(\theta)\right\}.$

(1)

The functions $l(\theta)$ and $u(\theta)$ are bounds on the shape of a density in $\psi_{l,u}$. If an unnormalized density can fit between the bounds, its normalized version is in $\psi_{l,u}$. In the case where $l(\theta)>0$ for all $\theta\in\Theta$, an equivalent definition of $\psi_{l,u}$ is

$\displaystyle\psi_{l,u}$

$\displaystyle=\left\{\hat{\pi}(\theta):\frac{l(\theta)}{u(\theta^{\prime})}\leq\frac{\pi(\theta)}{\pi(\theta^{\prime})}\leq\frac{u(\theta)}{l(\theta^{\prime})}\mbox{ for all $\theta,\theta^{\prime}\in\Theta$}\right\}.$

(2)

In (2) the left-most inequality is equivalent to the right-most one by inverting ratios. However, including the redundancy makes the implications of the definition clearer. The definition (2) explains the name “density ratio class” first used in Berger, (1990). The equivalence of (1) and (2) is demonstrated in Appendix B.

From (1), it is immediate that $\psi_{l,u}=\psi_{kl,ku}$, for any constant $k>0$. Hence we could take either the lower or upper bound function to be a normalized density. If we take $u(\theta)=\hat{u}(\theta)$, and write

$\displaystyle r$

$\displaystyle=\frac{\int u(\theta)\,d\theta}{\int l(\theta)\,d\theta},$

(3)

then

$\displaystyle\psi_{l,u}=\psi_{r^{-1}\hat{l},\hat{u}}.$

(4)

In visualizing density ratio classes later, we will normalize the upper bound function.

A density ratio class implies a range of probabilities for any event. Suppose we are interested in the event $E\subseteq\Theta$, and for some density ratio class $\psi_{l,u}$ we want the lower and upper probabilities $\underline{P}(E)$, $\overline{P}(E)$ for $E$, defined by

$$(\underline{P}(E),\overline{P}(E)):=\left(\inf_{\hat{\pi}(\theta)\in\psi_{l,u}}\int_{E}\hat{\pi}(\theta)\,d\theta,\sup_{\hat{\pi}(\theta)\in\psi_{l,u}}\int_{E}\hat{\pi}(\theta)\,d\theta\right).$$

It is easily shown (e.g. Rinderknecht et al., 2014, Section 2.1) that

$\displaystyle(\underline{P}(E),\overline{P}(E))$

$\displaystyle=\left(\frac{\int_{E}l(\theta)\,d\theta}{\int_{E}l(\theta)\,d\theta+\int_{E^{c}}u(\theta)\,d\theta},\frac{\int_{E}u(\theta)\,d\theta}{\int_{E}u(\theta)\,d\theta+\int_{E^{c}}l(\theta)\,d\theta}\right),$

where $E^{c}$ denotes the complement of $E$. In terms of the normalized densities $\hat{l}(\theta)$ and $\hat{u}(\theta)$ and with $r$ defined in (3), we can write

$\displaystyle(\underline{P}(E),\overline{P}(E))$

$\displaystyle=\left(\frac{\int_{E}\hat{l}(\theta)\,d\theta}{\int_{E}\hat{l}(\theta)\,d\theta+r\int_{E^{c}}\hat{u}(\theta)\,d\theta},\frac{\int_{E}\hat{u}(\theta)\,d\theta}{\int_{E}\hat{u}(\theta)\,d\theta+r^{-1}\int_{E^{c}}\hat{l}(\theta)\,d\theta}\right).$

(5)

Two important properties of density ratio classes are closure under Bayesian updating and marginalization. Wasserman, (1992) demonstrated that the density ratio class is the only class of densities possessing these properties. Invariance under Bayesian updating means the following. If a set of prior densities is a density ratio class, $\psi_{l,u}$ say, and if we update each prior in $\psi_{l,u}$ to its posterior density using a likelihood function $p(y|\theta)$, then the set of posterior densities is also a density ratio class, which we write as $\psi_{l(\theta;y),u(\theta;y)}$, where $l(\theta;y)=l(\theta)p(y|\theta)$ and $u(\theta;y)=u(\theta)p(y|\theta)$. Later we will make use of the ratio of areas under the upper and lower bound functions for the posterior density ratio class as a discrepancy in a Bayesian predictive check, and it will be useful to have some notation for this. We write

$\displaystyle r(y)$

$\displaystyle:=\frac{\int u(\theta;y)\,d\theta}{\int l(\theta;y)\,d\theta}=r\frac{\int\hat{u}(\theta)p(y|\theta)\,d\theta}{\int\hat{l}(\theta)p(y|\theta)\,d\theta},$

(6)

with the last equality following from (4). The quantity $r(y)$ is a measure of how large the class of posterior densities is. Multiplying the lower and upper bound functions by an arbitrary positive constant does not change $r(y)$.

Next we describe invariance of a density ratio class under marginalization. Suppose we partition $\theta$ into two subvectors $\theta=(\theta_{A}^{\top},\theta_{B}^{\top})^{\top}$ and for $\hat{\pi}(\theta)\in\psi_{l,u}$, write $\hat{\pi}(\theta_{A})$ for its $\theta_{A}$ marginal:

$$\hat{\pi}(\theta_{A})=\int\hat{\pi}(\theta)\,d\theta_{B}.$$

The set of all densities $\hat{\pi}(\theta_{A})$ for $\hat{\pi}(\theta)\in\psi_{l,u}$ is a density ratio class, $\psi_{l(\theta_{A}),u(\theta_{A})}$, where

$$l(\theta_{A})=\int l(\theta)\,d\theta_{B},\;\;\;\;u(\theta_{A})=\int u(\theta)\,d\theta_{B}.$$

Rinderknecht et al., (2014) considered how a density ratio class defined on parameters propagates in Bayesian predictive inference for data $y^{\prime}$. Rinderknecht et al., (2014) consider both deterministic and stochastic prediction. For the case where predictions are deterministic given $\theta$, the predictive densities propagated from a density ratio class on the parameter space are a density ratio class on predictive space. This follows from the closure under marginalization property discussed above.

Next consider stochastic prediction, where given $\theta$ the data to be predicted has density $p(y^{\prime}|\theta)$ given $\theta$. Suppose we have a density ratio class $\psi_{l,u}$ of densities defined on the parameter space $\Theta$. For $\hat{\pi}(\theta)\in\psi_{l,u}$, consider the prior predictive density for $y^{\prime}$ defined as

$\displaystyle p(y^{\prime};\hat{\pi})$

$\displaystyle=\int p(y^{\prime}|\theta)\hat{\pi}(\theta)\;d\theta.$

(7)

Later we will also consider extensions to settings where there is previously observed data $y$ say and $y^{\prime}$ is not conditionally independent of it given $\theta$, and we define

$\displaystyle p(y^{\prime};y,\hat{\pi})$

$\displaystyle:=\frac{p(y^{\prime},y;\hat{\pi})}{p(y;\hat{\pi})}.$

The set of $p(y^{\prime};\hat{\pi})$ for all $\hat{\pi}(\theta)\in\psi_{l,u}$ is not a density ratio class, but it is contained in the density ratio class

$\displaystyle\psi_{l(y^{\prime}),u(y^{\prime})},$

(8)

where

$$l(y^{\prime})=\int l(\theta)p(y^{\prime};\theta)\,d\theta\;\;\;\;u(y^{\prime})=\int u(\theta)p(y^{\prime};\theta)\,d\theta.$$

There can be predictive densities in (8) that may not be obtained from (7) for some $\hat{\pi}\in\psi_{l,u}$; Rinderknecht et al., (2014, Section 2.4.1) give an example. We can use the density ratio class (8) to give conservative upper and lower predictive probabilities. It is immediate that

$$\psi_{l(y^{\prime}),u(y^{\prime})}=\psi_{(r)^{-1}\hat{l}(y^{\prime}),\hat{u}(y^{\prime})}.$$

Later we will consider density ratio classes of prior predictive densities based on the observed $y$, as given by $\psi_{l(y),u(y)}$. We will also consider prior predictive densities of data summary statistics, $S=S(y)$, and in this case we write $\psi_{l(S),u(S)}$.

We use similar notation to previous sections, with $\theta$ a parameter in a statistical model for data $y$. The density of $y$ is $p(y|\theta)$, and we consider in this subsection conventional Bayesian inference with a prior density $p(\theta)$ for $\theta$. The observed value of $y$ is $y_{\text{obs}}$. The posterior density is $p(\theta|y_{\text{obs}})\propto p(\theta)p(y_{\text{obs}}|\theta)$. Bayesian model checking is usually performed through Bayesian predictive checks, which require the choice of a statistic $D(y)$, and a reference density $m(y)$ for the data. The check examines whether the observed discrepancy $D(y_{\text{obs}})$ is surprising or not under the assumed model. To measure surprise, the observed discrepancy is calibrated by computing a tail probability, sometimes referred to as a Bayesian predictive $p$-value,

$\displaystyle p$

$\displaystyle=P(D(y)\geq D(y_{\text{obs}})),$

(16)

for $y\sim m(y)$, where it is assumed that $D(y)$ has been defined in such a way that a larger value is considered more surprising.

The test statistic and reference distribution used should depend on what aspect of the model we wish to check. When the goal is to check the likelihood component of the model, it is popular to choose $m(y)$ as the posterior predictive density of a replicate observation (Guttman,, 1967; Gelman et al.,, 1996):

$$p(y|y_{\text{obs}})=\int p(\theta|y_{\text{obs}})p(y|\theta)\,d\theta.$$

Although easy to use, posterior predictive $p$-values are not necessarily uniform or otherwise having a known distribution when the model is correct, and this lack of calibration means it can be hard to identify when such a check has produced a surprising result. Lack of calibration is often symptomatic of an insufficiently thoughtful choice of $D(y)$, but alternatives to the posterior predictive $p$-value with better calibration properties have been explored (Bayarri and Berger,, 2000; Moran et al.,, 2023).

Most relevant to the present work is checking for prior-data conflict, where we consider checking the prior $p(\theta)$ rather than the likelihood $p(y|\theta)$. By “checking the prior” here, we mean considering a Bayesian predictive check which will alert us if the information in the prior and the likelihood are contradictory. This happens if $p(\theta)$ puts all its mass in the tails of the likelihood. These conflicts are important to detect, because they can result in prior sensitivity for inferences of interest (Al Labadi and Evans,, 2017), and may alert us to an inadequate understanding of the model and its parametrization. For prior-data conflict checking, an appropriate reference density $m(y)$ is the prior predictive density (Box,, 1980),

$$p(y)=\int p(\theta)p(y|\theta),$$

since we wish to see if the observed likelihood is unusual compared to the likelihood for data generated using the prior density for the parameters. Box, (1980) suggested prior predictive checking for criticism of both likelihood and prior, but Evans and Moshonov, (2006) argued that prior predictive checks based on a minimal sufficient statistic are appropriate for checking for prior-data conflict, while alternative methods are more appropriate for criticizing the likelihood. Various choices of $D(y)$ can be considered for summarizing the likelihood in a prior predictive check, such as prior predictive density values for exactly or approximately sufficient statistics, or prior-to-posterior divergences (Evans and Moshonov,, 2006; Nott et al.,, 2020). Extensions to hierarchical conflict checking methods are also discussed by these and other authors (e.g. Marshall and Spiegelhalter, 2007; Bayarri and Castellanos, 2007; Steinbakk and Storvik, 2009; Scheel et al., 2011), where an attempt is made to perform model checking informative about different levels of a hierarchical prior. A discussion of checking for conflicting sources of information in a Bayesian model in a broad sense is given in Presanis et al., (2013).

There has been much discussion of the connection between prior-data conflict and imprecise probability models. For example, Walter and Coolen, (2016) consider the phenomenon of prior-data conflict insensitivity for exponential family models and a precise conjugate prior, and suggest imprecise probability methods where the range of posterior inferences is reflective of prior-data conflicts or prior-data agreement. For earlier related work see Pericchi and Walley, (1991), Coolen, (1994), Walter and Augustin, (2009) and Walter and Augustin, (2010) for example. This work is insightful about how various imprecise probability formulations deal with conflicts in simple settings where computations are tractable, such as for exponential families. However, to the best of our knowledge, existing approaches have not been extended to prior-data conflict checking for density ratio classes, or to models with intractable likelihood. We consider this now.

A first simple approach to conflict checking for density ratio classes is to apply conventional prior-data conflict checks to the priors specified by the bounds, $\hat{l}(\theta)$ and $\hat{u}(\theta)$. If the prior class represents ambiguity in our prior knowledge, it may be felt that this prior information should be consistent with the information in the likelihood, and that none of the priors in the prior class should have conflict with the likelihood. If we take that view, it can be interesting to conduct a conventional Bayesian check on the bounds. If there is a conflict, this suggests there is a problem with the elicitation of the bounds. In the next subsection we will consider a different approach, where the goal is to determine whether every prior in the prior class is in conflict with the likelihood.

In implementing a conventional prior-data conflict check for bounds, we will consider the approach of Evans and Moshonov, (2006). Consider a conventional Bayesian analysis with prior $\hat{\eta}(\theta)$. Evans and Moshonov, (2006) suggest a prior-data conflict check using the discrepancy

$\displaystyle D(T)$

$\displaystyle=-\log p(T;\hat{\eta}),$

(17)

where $T$ is a minimal sufficient statistic. They calibrate the observed value $D(T_{\text{obs}})$ of $D(T)$ using a prior predictive tail probability

$$P(D(T)\geq D(T_{\text{obs}})),\;\;\;T\sim p(T;\hat{\eta}).$$

The discrepancy used by Evans and Moshonov, (2006) is a function of a minimal sufficient statistic, which is a desirable feature for a prior-data conflict check. If a proposed discrepancy depends on aspects of the data not captured by a minimal sufficient statistic, then these aspects are irrelevant to the likelihood, and can have nothing to do with whether the likelihood and a prior conflict.

In SBI, choosing the summary statistic $S$ as a minimal sufficient statistic is an ideal that is often not attainable, but it is desirable to choose an $S$ which is “near minimal sufficient”. Near sufficiency reduces information loss, and having a statistic of minimal dimension aids computation. Hence if a good summary statistic choice has been made for an SBI analysis, it is natural to use $T=S$ in implementing the prior-data conflict check of Evans and Moshonov, (2006). Computation of discrepancies for the bound checks requires computation of $p(S;\hat{l})$ and $p(S;\hat{u})$. These calculations can be done approximately using (13), and the calibration tail probability can be estimated by Monte Carlo. Considering the check for $\hat{l}(\theta)$, and substituting a normalizing flow approximations $\widetilde{l}(\theta|S)$ and $\widetilde{l}(S|\theta)$ for $\hat{l}(\theta|S)$ and $\hat{l}(S|\theta)$ in (13) we obtain an approximate discrepancy $\widetilde{D}(S)$. A Monte Carlo approximation of a Bayesian predictive $p$-value is obtained as

$$\frac{1}{V}\sum_{v=1}^{V}I\left\{\widetilde{D}(S_{v})\geq\widetilde{D}(S_{\text{obs}})\right\},\;\;\;\mbox{for $S_{v}\sim p(S;\hat{l})$, $v=1,\dots,V$}.$$

The check for $\hat{u}(\theta)$ is similar.

A possible problem with the check of Evans and Moshonov, (2006) is that the check is not invariant to the particular form used for the minimal sufficient statistic $T$. So, for example, if we make an invertible transformation of $T$ to $T^{\prime}$ say, the result of the check may differ. Although there are ways to address this issue (Evans and Jang,, 2011) the checks for density ratio classes in the next subsection do possess a property of invariance to invertible transformations of summary statistics.

Next we consider checks for whether all the priors in a prior density ratio class are in conflict with the likelihood. If the “size” of the prior density ratio class is very large, then the checks we discuss here are unlikely to produce evidence of a conflict. This is natural, because when the degree of prior ambiguity is large, there are many different priors compatible with the prior information. Hence the checks discussed here are useful mostly when the elicited density ratio class is highly informative.

Here we will consider $r(S)$ defined at (12) as the discrepancy for a check,

$$r(S)=r\frac{p(S;\hat{u})}{p(S;\hat{l})}.$$

Intuitively, $r(S)$ will be very large if the prior predictive density value for $\hat{u}$ is much larger than for $\hat{l}$. This often happens in the case of a prior-data conflict, where prior predictive density values are very sensitive to the prior, and the usually more diffuse $\hat{u}$ places mass closer to parameter values receiving support from the likelihood. We will compare $r(S_{\text{obs}})$ to what is expected under the model to check for conflict. It is possible to have density ratio classes where $\hat{l}(\theta)=\hat{u}(\theta)$, and in this case $r(S)$ is not a suitable discrepancy to use for conflict checking, since $r(S)$ is then a constant not depending on the data. Alternative discrepancies that could be used in such cases are discussed below.

We calibrate our check using the upper probability of the event $\{r(S)\geq r(S_{\text{obs}})\}$ for the prior predictive density ratio class $\psi_{l(S),u(S)}$,

$\displaystyle\overline{P}(r(S)\geq r(S_{\text{obs}})).$

(18)

As discussed in Section 2.4, the density ratio class $\psi_{l(S),u(S)}$ is larger than the set of predictive densities $\{p(S;\hat{\pi}):\hat{\pi}\in\psi_{l,u}\}$, leading to conservative Bayesian predictive $p$-values. The upper probability $\overline{P}(r(S)\geq r(S_{\text{obs}})$ is

$\displaystyle\overline{P}(r(S)\geq r(S_{\text{obs}}))$

$\displaystyle=\frac{\int_{\{r(S)\geq r(S_{\text{obs}})\}}p(S;\hat{u})\,dS}{\int_{\{r(S)\geq r(S_{\text{obs}})\}}p(S;\hat{u})\,dS+r^{-1}\int_{\{r(S)<r(S_{\text{obs}})\}}p(S;\hat{l})\,dS}.$

(19)

We use $\widetilde{r}(S)$ at (15) to approximate $r(S)$, and we can approximate the probabilities given by the integrals in (19) from samples $S_{v}^{u}\sim p(S;\hat{u})$, $v=1,\dots,V$ and $S_{v}^{l}\sim p(S;\hat{l})$, $v=1,\dots,V$.

The conflict checks considered here are related to the checks of Evans and Moshonov, (2006) with $T=S$, since both are based on prior predictive densities for $S$. The statistic $r(S)$ looks at a ratio of prior predictive density values at $S$ for $\hat{u}(\theta)$ and $\hat{l}(\theta)$. For our robust Bayes conflict check, if we make an invertible transformation of $S$ to another statistic $S^{\prime}$, it is easy to see that $r(S)=r(S^{\prime})$, because $r(S^{\prime})$ is a ratio of densities where the Jacobian term for the transformation cancels out. As mentioned above, if we choose $\hat{u}(\theta)=\hat{l}(\theta)$ in defining the density ratio class, $r(S)$ cannot be used as a discrepancy for checking for conflict. In that case, we could use use the statistic of Evans and Moshonov (17) with $T=S$ and $\hat{\eta}=\hat{u}$, or $\hat{\eta}=\hat{l}$. Then we can calibrate the check of the density ratio class with $\overline{P}(D(S)\geq D(S_{\text{obs}}))$. As $r\rightarrow 1$, and the density ratio class shrinks towards a single precise prior, we would recover the check of Evans and Moshonov, (2006) for this prior.

In Mao et al., (2021), the authors consider a way of checking for conflict between different components of a summary statistic vector in a conventional SBI analysis with summary statistics. They consider dividing $S$ into subvectors $S=(S_{A}^{\top},S_{B}^{\top})^{\top}$, with observed value $S_{\text{obs}}=(S_{\text{obs},A}^{\top},S_{\text{obs},B}^{\top})^{\top}$, and writing

$$p(\theta|S_{\text{obs}})\propto p(\theta|S_{\text{obs},A})p(S_{\text{obs},B}|S_{\text{obs},A},\theta),$$

we can consider $p(\theta|S_{\text{obs},A})$ as the prior after observing $S_{A}=S_{\text{obs,A}}$ to be updated by the likelihood term $p(S_{\text{obs},B}|S_{\text{obs},A},\theta)$. If we have checked the adequacy of the model $p(S_{A}|\theta)$ for $S_{A}$, and if there is no prior data conflict between the prior $p(\theta)$ and $p(S_{A}|\theta)$, then conflict between $p(\theta|S_{\text{obs},A})$ and $p(S_{\text{obs},B}|S_{\text{obs},A},\theta)$ indicates conflict between the different subvectors of the summary statistics. To get some intuition, a conflict here would suggest that the values of $\theta$ that give a good fit to $S_{\text{obs},A}$ are not values that give a good fit to $S_{\text{obs},B}$, which may indicate model misspecification. The definition of the check of Mao et al., (2021) is not invariant to swapping $S_{A}$ and $S_{B}$ in the definition; the order matters. If $S_{A}$ were a sufficient statistic, this check would not be useful, since in that case $p(S_{B}|S_{\text{obs},A},\theta)$ does not depend on $\theta$, and hence this likelihood term cannot conflict with $p(\theta|S_{\text{obs},A})$ no matter what $S_{B}$ is observed.

It is possible to construct a similar check for conflict between summary statistics for density ratio classes. Follow the discussion of Section 5.3, the prior-data conflict discrepancy for the case where $S_{A}$ is observed but before updating by $S_{B}$, is

$\displaystyle r(S_{B}|S_{\text{obs},A})$

$\displaystyle=r(S_{\text{obs},A})\frac{p(S_{B}|S_{\text{obs},A},\hat{u})}{p(S_{B}|S_{\text{obs},A},\hat{l})},$

(20)

where

$\displaystyle p(S_{B}|S_{\text{obs},A},\hat{u})$

$\displaystyle:=\frac{p(S_{\text{obs},A},S_{B};\hat{u})}{p(S_{\text{obs},A};\hat{u})},$

(21)

$\displaystyle p(S_{B}|S_{\text{obs},A},\hat{l})$

$\displaystyle:=\frac{p(S_{\text{obs},A},S_{B};\hat{l})}{p(S_{\text{obs},A}.\hat{l})},$

(22)

and

$\displaystyle r(S_{\text{obs},A})=r\frac{p(S_{\text{obs},A};\hat{u})}{p(S_{\text{obs},A};\hat{l})}$

(23)

Write $A$ for the event $A=\{r(S_{B}|S_{\text{obs},A})\geq r(S_{\text{obs},B}|S_{\text{obs},A})\}$. A calibration calibration tail probability for the discrepancy (20) is

$\displaystyle\overline{P}(A)$

$\displaystyle=\frac{\int_{A}p(S_{B}|S_{\text{obs},A};\hat{u})\,dS_{B}}{\int_{A}p(S_{B}|S_{\text{obs},A};\hat{u})\,dS_{B}+r(S_{\text{obs},A})^{-1}\int_{A^{c}}p(S_{B}|S_{\text{obs},A};\hat{l})\,dS_{B}}.$

(24)

Further details about approximation of the discrepancy and computation are given in Appendix C.

This check may not be very helpful in the case where the prior density ratio class is large, leading to very conservative results. For checking conflicts between summary statistics, we find it more useful to conduct the corresponding checks based on the bounds, and we discuss this now. For checking for conflict between summaries, which is a check of the likelihood, it is not necessary to consider prior ambiguity, since the prior has nothing to do with correct specification of the likelihood.

In the check for conflict between summary statistics based on the bounds, we can use check of Evans and Moshonov, (2006). If it is observed that $S_{A}=S_{\text{obs},A}$, then the discrepancy for the check based on e.g. the lower bound, is

$$D(S_{B}|S_{\text{obs},A})=-\log p(S_{B}|S_{\text{obs},A};\hat{l})=-\log\frac{p(S_{\text{obs},A},S_{B};\hat{l})}{p(S_{\text{obs},A};\hat{l})}.$$

Estimating the prior predictive density values in the expression on the right can be done using the methods of Section 4 to obtain an approximate discrepancy $\widetilde{D}(S_{B}|S_{\text{obs},A})$. A Monte Carlo approximation of a Bayesian predictive $p$-value is obtained as

$$\frac{1}{V}\sum_{v=1}^{V}I\left\{\widetilde{D}(S_{v,B}|S_{\text{obs},A})\geq\widetilde{D}(S_{\text{obs},B}|S_{\text{obs},A})\right\},\;\;\;\mbox{for $S_{v,B}\sim p(S_{B}|S_{\text{obs},A};\hat{l})$}.$$

The check for for the upper bound of the posterior density ratio class given $S_{A}=S_{\text{obs},A}$ is similar.

We discuss a simple normal example first, in which calculations can be done analytically. The example comes from the literature on Bayesian modular inference (Liu et al.,, 2009). There are two data sources, $z=(z_{1},\dots,z_{n_{1}})^{\top}$ and $w=(w_{1},\dots,w_{n_{2}})^{\top}$. There are scalar parameters $\varphi$ and $\eta$. Given $\varphi$, the elements of $z$ are conditionally independent, $z_{i}|\varphi\sim N(\varphi,1)$, $i=1,\dots,n_{1}$. Give $\varphi$ and $\eta$, the elements of $w$ are conditionally independent, $w_{i}|\varphi,\eta\sim N(\varphi+\eta,1)$, $i=1,\dots,n_{2}$. The parameter of interest is $\varphi$, and $\eta$ is a bias parameter which affects the data $w$. Liu et al., (2009) consider the setting where $n_{1}$ is small, and $n_{2}$ is much larger. So $\varphi$ can be estimated using the data $z$ only. However, one might think that since $n_{1}$ is small, using the biased data $w$ might improve the inference about $\varphi$ substantially. Liu et al., (2009) consider a single prior where $\varphi$ and $\eta$ are independent a priori with $\varphi\sim N(0,\delta_{1}^{-1})$, $\eta\sim N(0,\delta_{2}^{-1})$ where $\delta_{1},\delta_{2}>0$ are known precision parameters. A flat prior on $\varphi$ can be obtained if $\delta_{1}\rightarrow 0$, and if $\delta_{2}$ is large, this would express confidence that the bias is small. Liu et al., (2009) show that using the biased data $w$ in addition to $z$ is not helpful for inference about $\varphi$, with limited improvement for small bias, and very poor inference for large bias.

We consider the checks developed in Sections 5.4 and 5.5 for conflict between summary statistics for this problem. A minimal sufficient statistic for $\theta=(\varphi,\eta)$ is $S=(\bar{z},\bar{w})$, where $\bar{z}$ and $\bar{w}$ are the sample means of $z$ and $w$ respectively. In the notation of Section 5.4, we use $S_{A}=\bar{z}$ and $S_{B}=\bar{w}$. After observing $\bar{z}$, we can think of the posterior given $\bar{z}$ as a prior used for updating by $S_{B}=\bar{w}$, and check its consistency with the likelihood term for $\bar{w}$. A conflict could indicate inconsistency between the prior and likelihood, or a problem with the likelihood; here we will consider the situation where the problem is a prior-data conflict, with a large bias $\eta$ that conflicts with prior information expressing belief in a small bias.

The lower and upper bound functions $l(\theta)$ and $u(\theta)$ are defined as follows. Write $\phi(x;\mu,\sigma^{2})$ for the normal density in $x$ with mean $\mu$ and variance $\sigma^{2}$. Without loss of generality we take the upper bound function to be normalized,

$$u(\theta)=\hat{u}(\theta)=\phi(\varphi;0,\delta_{1}^{-1})\times\phi(\eta;0,\delta_{2}^{-1})$$

and $l(\theta)=r^{-1}\hat{l}(\theta)$ where

$$\hat{l}(\theta)=\phi(\varphi;0,\delta_{1}^{-1})\times\phi(\eta;0,k\delta_{2}^{-1}).$$

We set $k$ to be $0.9$ and then multiply $\hat{l}(\theta)$ by $r^{-1}=0.9$ which ensures that $l(\theta)$ is a lower bound. $\hat{u}(\theta)$ is a joint prior of similar form to the one considered in Liu et al., (2009), and $\hat{l}(\theta)$ is similar but with a smaller prior variance for the bias parameter $\eta$.

Write $\bar{z}_{\text{obs}}$ for the observed value of $\bar{z}$. Again using the notation of Section 5.4, routine manipulations show that

$\displaystyle p(S_{B}|S_{\text{obs},A};\hat{u})$

$\displaystyle=p(\bar{w}|\bar{z}_{\text{obs}};\hat{u})=\phi\left(\bar{w};\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}},\frac{1}{n_{1}+\delta_{1}}+\frac{1}{\delta_{2}}+\frac{1}{n_{2}}\right),$

(25)

and

$\displaystyle p(S_{B}|S_{\text{obs},A};\hat{l})$

$\displaystyle=p(\bar{w}|\bar{z}_{\text{obs}};\hat{l})=\phi\left(\bar{w};\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}},\frac{1}{n_{1}+\delta_{1}}+\frac{k}{\delta_{2}}+\frac{1}{n_{2}}\right).$

(26)

Let’s first consider the checks of Section 5.5 for the bounds using the approach of Evans and Moshonov, (2006). The discrepancy for the check is $-\log p(\bar{w}|\bar{z}_{\text{obs}},\hat{l})$ for the lower bound and $-\log p(\bar{w}|\bar{z}_{\text{obs}},\hat{u})$ for the upper bound. If we take logs in (25) and (26) and ignore irrelevant constants we get equivalent discrepancies, which are the same for both cases and equal to

$\displaystyle\left(\bar{w}-\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}}\right)^{2}.$

(27)

Let

$$A=\left\{\left(\bar{w}-\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}}\right)^{2}\geq\left(\bar{w}_{\text{obs}}-\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}}\right)^{2}\right\}.$$

The probabilities of $A$ under (25) and (26) respectively are

$$P_{A,u}=P\left(W\geq\frac{\left(\bar{w}_{\text{obs}}-\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}}\right)^{2}}{\frac{1}{n_{1}+\delta_{1}}+\frac{1}{\delta_{2}}+\frac{1}{n_{2}}}\right)\hskip 28.45274ptP_{A,l}=P\left(W\geq\frac{\left(\bar{w}_{\text{obs}}-\frac{n_{1}}{n_{1}+\delta_{1}}\bar{z}_{\text{obs}}\right)^{2}}{\frac{1}{n_{1}+\delta_{1}}+\frac{k}{\delta_{2}}+\frac{1}{n_{2}}}\right),$$

where $W\sim\chi^{2}_{1}$. $P_{A,u}$ and $P_{A,l}$ are the exact calibration tail probabilities discussed in Section 5.5 for the priors $\hat{u}$ and $\hat{l}$ respectively.