Objective Priors: An Introduction for Frequentists

Bayesian methods are increasingly used in recent years in the theory and practice of statistics. Their implementation requires specification of both a likelihood and a prior. With enough historical data, it is possible to elicit a prior distribution fairly accurately. However, even in its absence, Bayesian methods, if judiciously used, can produce meaningful inferences based on the so-called “objective” or “default” priors.

The main focus of this article is to introduce certain objective priors which could be potentially useful even for frequentist inference. One such example where frequentists are yet to reach a consensus about an “optimal” approach is the construction of confidence intervals for the ratio of two normal means, the celebrated Fieller–Creasy problem. It is shown in Section 4 of this paper how an “objective” prior produces a credible interval in this case which meets the target coverage probability of a frequentist confidence interval even for small or moderate sample sizes. Another situation, which has often become a real challenge for frequentists, is to find a suitable method for elimination of nuisance parameters when the dimension of the parameter grows in direct proportion to the sample size. This is what is usually referred to as the Neyman–Scott phenomenon. We will illustrate in Section 3 with an example of how an objective prior can sometimes overcome this problem.

Before getting into the main theme of this paper, we recount briefly the early history of objective priors. One of the earliest uses is usually attributed to Bayes (1763) and Laplace (1812) who recommended using a uniform prior for the binomial proportion $p$ in the absence of any other information. While intuitively quite appealing, this prior has often been criticized due to its lack of invariance under one-to-one reparameterization. For example, a uniform prior for $p$ in the binomial case does not result in a uniform prior for $p^{2}$. A more compelling example is that a uniform prior for $\sigma$, the population standard deviation, does not result in a uniform prior for $\sigma^{2}$, and the converse is also true. In a situation like this, it is not at all clear whether there can be any preference to assign a uniform prior to either $\sigma$ or $\sigma^{2}$.

In contrast, Jeffreys’ (1961) general rule prior, namely, the positive square root of the determinant of the Fisher information matrix, is invariant under one-to-one reparameterization of parameters. Wewill motivate this prior from several asymptotic considerations. In particular, for regular models where asymptotic normality holds, Jeffreys’ prior enjoys many optimality properties in the absence of nuisance parameters. In the presence of nuisance parameters, this prior suffers from many problems—marginalization paradox, the Neyman–Scott problem, just to name a few. Indeed, for the location–scale models, Jeffreys himself recommended alternate priors.

There are several criteria for the construction of objective priors. The present article primarily reviews two of these criteria in some detail, namely, “divergence priors” and “probability matching priors,” and finds optimal priors under these criteria. The class of divergence priors includes “reference priors” introduced by Bernardo (1979). The “probablity matching priors” were introduced by Welch and Peers (1963). There are many generalizations of the same in the past two decades. The development of both these priors rely on asymptotic considerations. Somewhat more briefly, I have discussed also a few other priors including the “right” and “left” Haar priors.

The paper does not claim the extensive thorough and comprehensive review of Kass and Wasserman (1996), nor does it aspire to the somewhat narrowly focused, but a very comprehensive review of probability matching priors as given in Ghosh and Mukerjee (1998), Datta and Mukerjee (2004) and Datta and Sweeting (2005). A very comprehensive review of reference priors is now available in Bernardo (2005), and a unified approach is given in the recent article of Berger, Bernardo and Sun (2009).

While primarily a review, the present article has been able to unify as well as generalize some of the previously considered criteria, for example, viewing the reference priors as members of a bigger class of divergence priors. Interestingly, with some of these criteria as presented here, it is possible to construct some alternatives to Jeffreys’ prior even in the absence of nuisance parameters.

The outline of the remaining sections is as follows. In Section 2 we introduce two basic tools to be used repeatedly in the subsequent sections. One such tool involving asymptotic expansion of the posterior density is due to Johnson (1970), and Ghosh, Sinha and Joshi (1982), and is discussed quite extensively in Ghosh, Delampady and Samanta (2006) and Datta and Mukerjee (2004). The second tool involves a shrinkage argument suggested by Dawid and used extensively by J. K. Ghosh and his co-authors. It is shown in Section 3 that this shrinkage argument can also be used in deriving priors with the criterion of maximizing the distance between the prior and the posterior. The distance measure used includes, but is not limited to, the Kullback–Leibler (K–L) distance considered in Bernardo (1979) for constructing two-group “reference priors.” Also, in this section we have considered a new prior different from Jeffreys even in the one-parameter case which is also invariant under one-to-one reparameterization. Section 4 addresses construction of priors under probability matching criteria. Certain other priors are introduced in Section 5, and it is pointed out that some of these priors can often provide exact and not just asymptotic matching. Some final remarks are made in Section 6.

Throughout this paper the results are presented more or less in a heuristic fashion, that is, without paying much attention to the regularity conditions needed to justify these results. More emphasis is placed on the application of these results in the construction of objective priors.

An asymptotic expansion of the posterior density began with Johnson (1970), followed up later by Ghosh, Sinha and Joshi (1982), and many others. The result goes beyond that of the theorem of Bernstein and Von Mises which provides asymptotic normality of the posterior density. Typically, such an expansion is centered around the MLE (and occasionally the posterior mode), and requires only derivatives of the log-likelihood with respect to the parameters, and evaluated at their MLE’s. These expansions are available even for heavy-tailed densities such as Cauchy because finiteness of moments of the distribution is not needed. The result goes a long way in finding asymptotic expansion for the posterior moments of parameters of interest as well as in finding asymptotic posterior predictive distributions.

The asymptotic expansion of the posterior resembles that of an Edgeworth expansion, but, unlike the latter, this approach does not need use of cumulants of the distribution. Finding cumulants, though conceptually easy, can become quite formidable, especially in the presence of multiple parameters, demanding evaluation of mixed cumulants.

We have used this expansion as a first step in the derivation of objective priors under different criteria. Together with the shrinkage argument as mentioned earlier in the Introduction, and to be discussed later in this section, one can easily unify and extend many of the known results on prior selection. In particular, we will see later in this section how some of the reference priors of Bernardo (1979) can be found via application of these two tools. The approach also leads to a somewhat surprising result involving asymptotic expansion of the distribution function of the MLE in a fairly general setup, and is not restricted to any particular family of distributions, for example, the exponential family, or the location–scale family. A detailed exposition is available in Datta and Mukerjee (2004, pages 5–8).

For simplicity of exposition, we consider primarily the one-parameter case. Results needed for the multiparameter case will occasionally be mentioned, and, in most cases, these are straightforward, albeit often cumbersome, extensions of one-parameter results. Moreover, as stated in the Introduction, the results will be given without full rigor, that is, without any specific mention of the needed regularity conditions.

We begin with $X_{1},\dots,X_{n}|\theta$ i.i.d. with common p.d.f. $f(X|\theta)$. Let $\hat{\theta}_{n}$ denote the MLE of $\theta$. The likelihood function is denoted by $L_{n}(\theta)=\prod^{n}_{1}f(X_{i}|\theta)$ and let $\ell_{n}(\theta)=\log L_{n}(\theta)$. Let $a_{i}=n^{-1}[d^{i}\ell_{n}(\theta)/\break d\theta^{i}]_{\theta=\hat{\theta}_{n}}$, $i=1,2,\dots,$ and let $\hat{I}_{n}=-a_{2}$, the observed per unit Fisher information number. Consider a twice differentiable prior $\pi$. Let $T_{n}=\sqrt{n}(\theta-\hat{\theta}_{n})\hat{I}_{n}^{1/2}$, and let $\pi^{*}_{n}(t)$ denote the posterior p.d.f. of $T_{n}$ given$X_{1},\ldots,X_{n}$. Then, under certain regularity conditions, we have the following result.

The proof is given in Ghosh, Delampdy and Samanta (2006, pages 107–108). The statement involves a few minor typos which can be corrected easily. We outline here only a few key steps needed in the proof.

We begin with the posterior p.d.f.,

Substituting $t=\sqrt{n}(\theta-\hat{\theta}_{n})\hat{I}_{n}^{1/2}$, the posterior p.d.f. of $T_{n}$ is given by

The rest of the proof involves a Taylor expansion of $\exp[\ell_{n}\{\hat{\theta}_{n}+t(n\hat{I}_{n})^{-1/2}\}$ and $\pi\{\hat{\theta}_{n}+t(n\hat{I}_{n})^{-1/2}\}$around $\hat{\theta}_{n}$ up to a desired order, and collecting the coefficients of $n^{-1/2}$, $n^{-1}$, etc. The other component is evaluation of $C_{n}$ via momets of the N(0, 1) distribution.

A multiparameter extension of Theorem 1 is as follows. Suppose that $\theta=(\theta_{1},\dots,\theta_{p})^{T}$ is the parameter vector and $\hat{\theta}_{n}$ is the MLE of $\theta$. Let

and $\hat{I}_{n}=((\hat{I}_{njr}))$. Then retaining only up to the $O(n^{-1/2})$ term, the posterior of $W_{n}=\sqrt{n}(\theta-\hat{\theta}_{n})$ is given by

Next we present the basic shrinkage argument of J. K. Ghosh discussed in detail in Datta and Mukherjee (2004). The prime objective here is evaluation of $E[q(X,\theta)|\theta]=\lambda(\theta)$, say, where $X$ and $\theta$ can bereal- or vector-valued. The idea is to find first$\int\lambda(\theta)\bar{\pi}_{m}(\theta)\,d\theta$ through a sequence of priors $\{\bar{\pi}_{m}(\theta)\}$ defined on a compact set, and then shrinking the prior to degeneracy at some interior point, say, $\theta$ of the compact set. The interesting point is that one never needs explicit specification of $\bar{\pi}_{m}(\theta)$ in carrying out this evaluation. We will see several illustrations of this in this article.

First, we present the shrinkage argument in a nutshell. Consider a proper prior $\bar{\pi}(\cdot)$ with a compact rectangle as its support in the parameter space, and $\bar{\pi}(\cdot)$ vanishes on the boundary of support, while remaining positive in the interior. The support of $\bar{\pi}(\cdot)$ is the closure of the set. Consider the posterior of $\theta$ under $\bar{\pi}(\cdot)$ and, hence, obtain $E^{\bar{\pi}}[q(X,\theta)|X]$. Then find $E[\{E^{\bar{\pi}}(q(X,\theta)|X)\}|\theta]=\lambda(\theta)$ for $\theta$ in the interior of the support of $\bar{\pi}(\cdot)$. Finally, integrate $\lambda(\cdot)$ with respect to $\bar{\pi}(\cdot)$, and then allow $\bar{\pi}(\cdot)$ to converge to the degenerate prior at the true value of $\theta$ at an interior point of the support of $\pi(\theta)$. This yields $E[q(X,\theta)|\theta]$. The calculation assumes integrability of $q(X,\theta)$ over the joint distribution of $X$ and $\theta$. Such integrability allows change in the order of integration.

When executed up to the desired order of approximation, under suitable assumptions, these steps can lead to significant reduction in the algebra underlying higher order frequentist asymptotics. The simplification arises from two counts. First, although the Bayesian approach to frequentist asymptotics requires Edgeworth type assumptions, it avoids an explicit Edgeworth expansion involving calculation of approximate cumulants. Second, as we will see, it helps establish the results in an easily interpretable compact form. The following two sections will demonstrate multiple usage of these two basic tools.