Logistic regression models: practical induced prior specification

Regarding the induced priors for $\theta$, for maximum simplicity we begin with a linear model for $\eta$ that is a constant, i.e., $\eta=\beta$:

$\displaystyle\eta=g(\theta)=\ln\left(\frac{\theta}{1-\theta}\right)$

$\displaystyle=\beta.$

Let $\pi(\beta)$ be the prior distribution for $\beta$, which in this simple case is also the prior for $\eta$, i.e., $\pi(\eta)=\pi(\beta)$. The induced prior for $\theta$ can be found by the change of variable theorem (Siegrist, , 2024):

$\displaystyle\pi_{\theta}(\theta)$

$\displaystyle=\pi_{\beta}(h^{-1}(\theta))\left|\frac{dh^{-1}(\theta)}{d\theta}\right|,$

where $h(\beta)=\theta$. We define,

$\displaystyle\theta=h(\beta)$

$\displaystyle=\frac{\exp(\beta)}{1+\exp(\beta)}.$

which leads to the induced pdf for $\theta$:

$\displaystyle\pi_{\theta}(\theta)$

$\displaystyle=\pi_{\beta}\left(\ln\left[\frac{\theta}{1-\theta}\right]\right)\left|\frac{1}{\theta(1-\theta)}\right|.$

Our objective is to determine what the induced prior for $\beta$ is given a desired or target prior for $\theta$. The above process is reversed. For example, if the desired prior for $\theta$ is Uniform$(0,1)$, then the problem is to determine what prior for $\beta$ would yield such a prior for $\theta$. To begin

$\displaystyle\beta=g(\theta)$

$\displaystyle=\ln\left(\frac{\theta}{1-\theta}\right).$

Then the inverse operation, $g^{-1}(\theta)$, is $\theta=\exp(\beta)/(1+\exp(\beta))$ and $\frac{dg^{-1}(\beta)}{d\beta}$ = $\frac{\exp(\beta)}{(1+\exp(\beta))^{2}}$. Consequently the induced pdf for $\beta$ is:

$\displaystyle\pi_{\beta}(\beta)$

$\displaystyle=\frac{\exp(\beta)}{(1+\exp(\beta))^{2}},~{}~{}-\infty<\beta<\infty.$

(2)

Figure 2 is a plot of the induced prior for $\beta$ given a Uniform(0,1) prior for $\theta$.

The pdf for the induced prior for $\beta$ in Equation (2) is a special case of the logistic distribution. The logistic distribution has location parameter $\mu$ and scale parameter $s$:

$\displaystyle\mbox{Logistic}(x;\mu,s)$

$\displaystyle=\frac{\exp\left(-\frac{(x-\mu)}{s}\right)}{s\left(1+\exp\left(-\frac{(x-\mu)}{s}\right)\right)^{2}}.$

The corresponding expectation and variance are as follows:

$\displaystyle E[X]=\mu,~{}$

$\displaystyle~{}V[X]=\frac{s^{2}\pi^{2}}{3}.$

The pdf in Equation (2) is then Logistic($\mu=0$, $s=1$), where due to symmetry in the pdf around 0, the pdf is the same for $\beta$ and $-\beta$. $E[\beta]=0$ (as Figure 2 indicates) and $V[\beta]$=$\frac{\pi^{2}}{3}$. We can conclude from this that if a Logistic$(0,1)$ prior is used for $\beta$, the induced prior for $\theta$ is Uniform$(0,1)$. Depending on the context, the target induced prior for $\theta$ may not be Uniform$(0,1)$, e.g., it might be Beta($\alpha,\beta$) and this is later addressed.

Given the shape of the Logistic(0,1) distribution and its symmetry about zero, a Normal(0, $\frac{\pi^{2}}{3}$) distribution could be used as an approximation. As Figure 3 shows the tails of the normal are heavier.

We begin with an unrealistic setting to make clearer the approach taken by assuming that the linear model for $\eta$ is simply a sum of $p+1$ coefficients, equivalently assume that all $x_{i}$=1:

$\displaystyle\eta=g(\theta)$

$\displaystyle=\beta_{0}+\beta_{1}\times 1+\ldots+\beta_{p}\times 1=\sum_{i=0}^{p}\beta_{i}.$

Hence there is a $p+1$ dimensional prior, $\pi(\beta_{0},\ldots,\beta_{p})$. The many-to-one problem can be bypassed by assuming that the prior distributions are all the same and independent. By moment matching we can determine what the mean, $\mu_{\beta}$, and variance, $\sigma_{\beta}^{2}$, of the in-common prior, denoted $\pi(\beta)$ distribution, should be.

	$\displaystyle E[\eta]=0$	$\displaystyle=E[\beta_{0}]+E[\beta_{1}]+\ldots E[\beta_{p}]=(p+1)\mu_{\beta}$
	$\displaystyle V[\eta]=\frac{\pi^{2}}{3}$	$\displaystyle=V[\beta_{0}]+V[\beta_{1}]\ldots+V[\beta_{p}]=(p+1)\sigma_{\beta}^{2}$

So that $\mu_{\beta}=0$ and $\sigma^{2}_{\beta}$ = $\frac{\pi^{2}}{3(p+1)}$. When $p=0$, we obtain the variance of $\frac{\pi^{2}}{3}$ considered in Section 2.4.

We now examine the more realistic case where there are covariates in the linear model. We will, however, assume that the covariates, $x_{1}$, $\ldots$, $x_{p}$, are standardised, $x_{i}$= $(x_{i}^{*}-\bar{x_{i}^{*}})/s_{x_{i}^{*}}$, where $x_{i}^{*}$ are the original values, thus $\bar{x_{i}}$=0 and $s_{x_{i}}$= 1. To bypass the many-to-one problem, we assume “some” in-common, and independent prior distributions. To begin we will assume that all the coefficients $\beta_{0}$, $\ldots$, $\beta_{p}$ are iid with prior $\pi_{\beta}(\beta)$.

We again use moment matching to determine what conditions are necessary for $E[\eta]$=0 and $V[\eta]$=$\pi^{2}/3$ to be satisfied.

	$\displaystyle E[\eta]=0$	$\displaystyle=E[\beta_{0}+\beta_{1}x_{1}+\ldots+\beta_{p}x_{p}]$
		$\displaystyle=E[\beta_{0}]+\sum_{i=1}^{p}E[\beta_{i}]E[X_{i}].$

Given that the covariate values have been standardised, their expected values, $E[X_{i}]$, equal 0, so that the second term above equals 0. This result arises from the expectation being calculated over a finite space with at most $n$ values for each $x_{i}$ and assuming implicitly random draws of each $x_{i}$ independent of $x_{j}$, $i\neq j$, which does not recognize any within sample dependencies. However, to ensure $E[\eta]$=0, $E[\beta_{0}]$ must also equal 0. If the $\beta$’s are iid, their expected values would also equal 0.

We next examine $V[\eta]$. In the following we assume that $V[x_{i}]$=1.

	$\displaystyle V[\eta]=\frac{\pi^{2}}{3}$	$\displaystyle=V\left[\beta_{0}+\sum_{i=1}^{p}\beta_{i}x_{i}\right]$
		$\displaystyle=V_{\beta}E_{\hbox{\boldmath$x$}|\beta}\left[\beta_{0}+\sum_{i=1}^{p}\beta_{i}x_{i}\left|\beta\right.\right]+E_{\beta}V_{\hbox{\boldmath$x$}|\beta}\left[\beta_{0}+\sum_{i=1}^{p}\beta_{i}x_{i}\left|\beta\right.\right]$
		$\displaystyle=V_{\beta}\left[\beta_{0}+\sum_{i=1}^{p}\beta_{i}E[x_{i}]\right]+E_{\beta}\left[\sum_{i=1}^{p}\beta_{i}^{2}V[x_{i}]\right]$
		$\displaystyle=V_{\beta}[\beta_{0}]+0+\left[\sum_{i=1}^{p}E_{\beta}[\beta_{i}^{2}]\times 1\right]$
		$\displaystyle=\sigma^{2}_{\beta}(1+p).$

To match the Logistic$(0,1)$ variance of $\frac{\pi^{2}}{3}$, we require

$\displaystyle\sigma^{2}_{\beta}$

$\displaystyle=\frac{\pi^{2}}{3(p+1)}.$

Therefore, as for the case with $x_{i}$=1, $\mu_{\beta}$=0 and $\sigma^{2}_{\beta}$ = $\frac{\pi^{2}}{3(p+1)}$. The reason for the identical results is that the standardized variance of the $x_{i}$’s, (namely, 1), is the same as the assigned constant of 1 in the previous case. Newman, (2003) proposed the identical normal prior distributions for an analysis of release-recovery data but the derivation details were not shown.

As for the special case of Uniform$(0,1)$, i.e. Beta$(1,1)$, a normal approximation to the distribution for $\eta$ is to moment match $\mu_{\eta}$ and $\sigma^{2}_{\eta}$.

For an intercept only model, $\eta=\beta_{0}$, and the prior for $\beta_{0}$ is

$\displaystyle\beta_{0}$

$\displaystyle\sim\mbox{Normal}\left(\mu_{\eta},\sigma_{\eta}\right).$

Examples of such normal approximations to the pdf for $\eta$ are shown by dashed blue lines in Figure 4, which show the adequacy of the approximation is a function of $\alpha$ and $\beta$.

In the case of one or more covariates, as for the Uniform$(0,1)$ case, there is again the many-to-one problem where there are an infinite number of combinations of normal priors where the expected value and variance for the sum of the priors would equal $\mu_{\eta}$ and $\sigma^{2}_{\eta}$. Furthermore, given that standardizing the covariates to have mean zero implies $E[\beta_{i}X_{i}]=0$ the mean $\mu_{\eta}$ is solely found in the prior mean for the intercept $\beta_{0}$. Again, we can avoid the many-to-one problem by using the $p$ independent and identically distributed normal distributions for the slope coefficients. Namely,

	$\displaystyle\beta_{0}$	$\displaystyle\sim\mbox{Normal}\left(\mu_{\eta},\frac{\sigma^{2}_{\eta}}{p+1}\right);$		(4)
	$\displaystyle\beta_{i}$	$\displaystyle\sim\mbox{Normal}\left(0,\frac{\sigma^{2}_{\eta}}{p+1}\right),\qquad i=1,\ldots,p.$		(5)

Setting the mean of the priors for the $\beta_{i}$ terms equal to 0 is arbitrary and can be any value since $E[\beta_{i}X_{i}]=0$. Then

	$\displaystyle E[\eta]$	$\displaystyle=E[\beta_{0}]+\sum_{i=1}^{p}E[\beta_{i}x_{i}]=\mu_{\eta}+\sum_{i=1}^{p}0=\mu_{\eta};$
	$\displaystyle Var[\eta]$	$\displaystyle=Var[\beta_{0}]+\sum_{i=1}^{p}Var[\beta_{i}x_{i}]=\frac{\sigma^{2}_{\eta}}{p+1}+\sum_{i=1}^{p}\frac{\sigma^{2}_{\eta}}{p+1}=\sigma^{2}_{\eta}.$

As $p$ increases, the variances for both the intercept and the slope coefficients are equally increasingly concentrated around, or shrinking toward, $\mu_{\eta}$ or 0. We next present an alternative which does not cause such shrinkage in the prior for the intercept.

We note that while there are different classes of so-called shrinkage priors , e.g., spike and slab priors, the intent differs somewhat, to shrink small effects to zero while maintaining true large effects (Van Erp et al., , 2019), while the intent is here is to yield a desired induced distribution on a particular parameter.

In some cases there may be a prior opinion regarding the value of $\theta$ even if the covariates have no effect, or for an “average” covariate setting, and a distinct handling of the intercept can be considered. In particular one may not want to have the number of covariates affecting the prior for the intercept, i.e., having the prior for $\beta_{0}$ become increasingly concentrated around zero.

We begin by assuming a more general prior for $\theta$, Beta($\alpha,\beta$), rather than the special case dealt with so far, namely $\alpha=\beta=1$ or Uniform$(0,1)$. Direct specification of the $\alpha$ and $\beta$ parameters could be done as well but thinking in terms of the average value for $\theta$ and its variation may be easier. Specifically, we first specify the expected value of $\theta$ and a measure of its variation, the coefficient of variation (CV), and then determine what values of $\alpha$ and $\beta$ match these specified values. For example, if we specify $E[\theta]=0.7$ and $CV[\theta]=0.30$, then the prior for $\theta$ is Beta($2.633,1.129$). Given the Beta distribution parameters, and referring to Equation (3), the expected value and variance of $\eta$ can be numerically calculated: in this example $\mu_{\eta}$ $\approx$ 1.146 and $\sigma^{2}_{\eta}$ $\approx$ 1.785. Again assuming that the priors for $\beta_{0}$ and $\beta_{i}$, $i=1,\ldots,p$ are Normal distributions, the resulting $E[\eta]$ and $Var[\eta]$ guide the choice of the Normal distribution parameters.

We note that with the exception of the case where $\alpha$=$\beta$, thus $E[\theta]$=0.5, and then $E[\eta]$ = 0, the contribution of $\mu_{\eta}$ to the linear combination (Equation 1) must all come from the mean for the prior for the intercept, $\beta_{0}$. This is because the covariates have been standardized to have mean 0, then $E[\beta_{i}x_{i}]$ = 0 no matter what the value of the expected value for $\beta_{i}$ in the normal distribution.

To bypass the many-to-one issue we again assume that the priors for the slope coefficients are identical. Let $\mu_{0}$ and $\sigma^{2}_{0}$ denote the mean and variance for the normal prior for $\beta_{0}$. For the normal prior for $\beta_{i}$, $\mu_{1}$ denotes the normal mean for the $\beta_{i}$’s, which can be anything given it does not affect the expected value of the linear combination, of $\eta$. Here for simplicity we will set $\mu_{1}$=0, and $\sigma^{2}_{1}$ denotes the variance for prior for $\beta_{1}$. While there might be some argument for $\mu_{1}\neq 0$, we cannot think of one. Subsequently,

	$\displaystyle\beta_{0}$	$\displaystyle\sim\mbox{Normal}\left(\mu_{\eta},\sigma^{2}_{0}\right),$		(6)
	$\displaystyle\beta_{1}$	$\displaystyle\sim\mbox{Normal}\left(0,\sigma^{2}_{1}\right).$		(7)

The constraint for the linear combination is that its expected value equals $\mu_{\eta}$ and its variance equals $\sigma^{2}_{\eta}$. Redisplaying the linear combination for convenience, with a randomly drawn sequence of values for $x_{i,j}$

$\displaystyle\eta_{j}$

$\displaystyle=\beta_{0}+\sum_{i=1}^{p}\beta_{i}x_{i,j},$

with $E[\eta_{j}]$ = $\mu_{\eta}$ and

$\displaystyle Var[\eta_{j}]$

$\displaystyle=\sigma^{2}_{0}+p\sigma^{2}_{1},$

where the latter sum must equal $\sigma^{2}_{\eta}$. Given values for $\mu_{\eta}$ and $\sigma^{2}_{\eta}$, once $\sigma^{2}_{0}$ are specified then the variance for the slope coefficient priors is

$\displaystyle\sigma^{2}_{1}$

$\displaystyle=\frac{\sigma^{2}_{\eta}-\sigma^{2}_{0}}{p}.$

Given the assumption of identical priors for all slope coefficients the only decision is to specify $\sigma^{2}_{0}$. There are numerous ways this can be done and that shown below is just one possibility.

If the aim is that the number of covariates does not affect the prior for $\beta_{0}$, in particular the prior variance for $\beta_{0}$, then $\sigma^{2}_{0}$ should not be a function of $p$. One approach is to define $\sigma^{2}_{0}$ relative to $\sigma^{2}_{\eta}$. For example specify a constant, $k$, $0<k<1$, and set $\sigma^{2}_{0}$ = $k\sigma^{2}_{\eta}$. Then

$\displaystyle\sigma^{2}_{1}$

$\displaystyle=\frac{(1-k)\sigma^{2}_{\eta}}{p}.$

The choice of $k$ is discussed in Section 7.

Multiple data sets of Bernoulli responses were generated from the following model:

	$\displaystyle y_{i}$	$\displaystyle\sim\mbox{Bernoulli}(\theta_{i}),~{}~{}i=1,\ldots,15$		(8)
	$\displaystyle logit(\theta_{i})$	$\displaystyle=-0.5+0.3x_{i},~{}~{}i=1,\ldots,15,$

where the values of $x$ were simulated from a Gamma(3,0.2), and then standardised. In the Appendix, the resulting response and covariate values are shown in Table 10 and the functional relationship between $\theta$ and $x$ is shown in Figure 10.

We compared the following two priors for $\beta_{0}$ and $\beta_{1}$.

	$\displaystyle\mbox{Vague: }\beta_{0},\beta_{1}$	$\displaystyle\sim\mbox{Normal}(0,1000^{2});$		(9)
	$\displaystyle\mbox{Logistic: }\beta_{0},\beta_{1}$	$\displaystyle\sim\mbox{Normal}\left(0,\frac{\pi^{2}}{3\times 2}=1.283^{2}\right).$

The induced priors for $\theta$ are shown in Figure 11 in the Appendix.

We simulated 100 datasets from the model with $n=15$, $n=50$ and $n=100$, and computed the following indexes: the mean squared error from the maximum likelihood estimate ($\text{MSE}^{*}$), the mean squared error from the posterior mean (MSE), and the coverage of the 95% credible interval. The above mean square errors are based on the following formula:

$$\text{MSE}(\hat{\theta})=\frac{1}{n}\sum_{i=1}^{n}(\hat{\theta}_{i}-\theta)^{2},$$

where $\hat{\theta}$ is the point estimate (i.e. the posterior mean or MAP, for example) and $\theta$ is either the MLE (for the $\text{MSE}^{*}$) or the true value of $\theta$ (for the MSE).

For the case $n$=15, from Table 1 we note that the MSE for the Logistic prior is smaller (about one order of magnitude) than the one computed for the Vague normal prior. In addition, while the posterior coverage for the proposed prior is within the expected limits, under the Vague prior we note a slightly lower than expected coverage for the intercept (0.92) and considerably lower coverage for the slope (0.78).

An interesting comparison is between the average MLEs and average posterior means for both parameters. The MLEs are -0.6184 and 0.4203, for $\beta_{0}$ and $\beta_{1}$, respectively. The average posterior mean for the intercept is -0.7775 and -0.5076, for the Vague prior and the Logistic prior, respectively, and for the slope coefficient, we have 0.6065 and 0.2780 for the Vague and the Logistic, respectively. We note that the distance between the MLEs and the posterior means are smaller for the Logistic prior than for the Vague prior.

For the larger sample size $n=50$, the corresponding results are presented in Table 2 where, as expected due to increasing data having more influence on the posterior distributions, the MSE from the MLE and from the posterior mean are lower. Note that, unlike in the case for $n=15$, the coverage of the coefficients under the Vague prior are close to the nominal value of 95%.

In this case, the average MLE for the intercept is -0.5104 and for the slope coefficient is 0.3184. The average posterior means are, for the intercept, -0.5290 (Vague prior) and -0.4950 (Logistic prior) and, for the slope coefficient, 0.3425 (Vague prior) 0.3165 (Logistic prior). Here, while the distances of the MLE and the posterior means for the intercept are virtually the same, the ones for the slope coefficient are smaller for the Logistic prior.

To conclude this experiment, we have run the above simulation with a larger sample size ($n=100$). The results are presented in Table 3, where we note that the priors’ behaviours are virtually the same as the information in the data dominates. Again, the overall performance of the Logistic prior is better than the Vague prior, in terms of distance between the MLE and the posterior mean, the difference diminishes, as one would expect as the sample size increases.

For this final simulation study, the average MLE for the intercept is -0.5265 and for the slope coefficient is 0.2914. The average posterior means are, for the intercept, -0.5355 (Vague prior) and -0.5206 (Logistic prior) and, for the slope coefficient, 0.3013 (Vague prior) and 0.2914 (Logistic prior).

In the second and third scenarios, we drew samples from the below model given in Equation (10), considering $\theta\sim U(0,1)$ (Scenario 2), and its generalisation $\theta\sim\text{Beta}(\alpha,\beta)$ with distinct handling of the intercept (Scenario 3):

$\displaystyle logit(\theta)$

$\displaystyle=1.1+0.3x_{1}-0.6x_{2}+0.02x_{3},$

(10)

where the covariates were simulated independently from three Gamma distributions, Gamma(10,2), Gamma(12,6), and Gamma(3,3), with the resulting $\theta$ values ranged from 0.39 to 0.93.

The prior distributions we compared are the Vague prior and the Logistic prior (see equations (4) and (5)) for the three covariates setting in Scenario 2.

	$\displaystyle\mbox{Vague: }\beta_{0},\beta_{1},\beta_{2},\beta_{3}$	$\displaystyle\sim\mbox{Normal}(0,1000^{2})$		(11)
	$\displaystyle\mbox{Logistic: }\beta_{0},\beta_{1},\beta_{2},\beta_{3}$	$\displaystyle\sim\mbox{Normal}\left(0,\frac{\pi^{2}}{3\times(1+3)}=0.907^{2}\right),$

and, for Scenario 3, the additional prior

	$\displaystyle\beta_{0}$	$\displaystyle\sim\text{Normal}(1.150,0.4\times 1.843=0.7372)$		(12)
	$\displaystyle\beta_{i}$	$\displaystyle\sim\text{Normal}\left(0,\frac{0.6\times 1.843}{3}=0.3686\right),\qquad i=1,2,3.$

In the Appendix, it is possible to find the induced $\theta$ priors, as well as the illustration of the model applied to a single sample.

The frequentist analysis for Scenario 2 was performed on 100 samples of sizes $n=30$, $n=50$ and $n=100$, respectively. We defer the discussion of the results below where the Vague, Logistic, and Weighted priors are shown together.

Similrly to Scenario 2, the frequentist analysis of Scenario 3 was performed on 100 samples of sizes $n=30$, $n=50$ and $n=100$, respectively.

Table 4, for $n$=50, shows the $\text{MSE}^{*}$, the MSE and interval coverage for the three priors. It appears that both the Logistic prior and Weighted prior perform better than the Vague prior; this is obvious by noticing that the MSE is systematically smaller for the two proposed priors for all the parameters. If we compare the coverage, we notice that the Weighted prior tends to have more extreme results, as it is either close to 100% or 90%, suggesting a less precise posterior. Given that the Weighted prior assumes different prior for the intercept and the slope coefficients, an increase in uncertainty is expected.

For the above simulation study, the average MLEs, and the average posterior means (under each prior) are reported in Table 5.

Similarly to the case with one covariate, we examined (and compared) the prior performances for a relatively small sample size. Using the rule of thumb of having ten data points per covariate, we have drawn 100 samples of size $n=30$. The results, in terms of $\text{MSE}^{*}$, MSE and posterior coverage, are reported in Table 6.

The MLE averages and the posterior mean averages, are reported in Table 7.

Finally, in Table 8 we have reported the results for the larger sample size ($n=100$), where the differences between the priors are reduced, although the better performance of the Logistic and Weighted priors remains.

The average summaries are reported in Table 9.

For the Weighted prior, the proposed solution for determining the variance for the intercept was to set it equal to some fraction of $\sigma^{2}_{\eta}$, $\sigma^{2}_{0}$ = $k\sigma^{2}_{\eta}$, $0\leq k\leq 1$. Can some guidance be provided regarding the choice of $k$?

If $k$ is near 0, then the prior for the intercept will be concentrated around $\mu_{\eta}$, and the priors for the slope coefficients will be near their maximum dispersion, $\sigma^{2}_{\eta}/p$. At the extreme of $k$=0, the intercept $\beta_{0}$ is a fixed value. On the other hand if $k$ is near 1, then the variance of the prior for the intercept is near its maximum $\sigma^{2}_{\eta}$ and the priors for the slope coefficients are more concentrated around zero. At the extreme of $k$=1 all slope coefficients disappear.

Therefore $k$ is balancing the effect of the intercept and the covariates. This begs the question of how to choose $k$. One thought is to treat $k$ as a parameter to estimate and then specify a hierarchical prior distribution on it. Another is to view $k$ as a tuning parameter in terms of a goodness-of-fit measure. Further investigation of this issue is a topic of further research.

As the number of covariates increases, the priors for the slope coefficients are increasingly concentrated around 0, i.e., there is shrinkage toward 0. Two statistical procedures that have some similarity with this result is Lasso (Tibshirani, , 1996) and penalized complexity priors (Simpson et al., , 2017). Lasso can cause shrinkage of coefficients to the point that a covariate is completely removed. Penalized complexity priors aim, given a set of nested models, to give highest prior probability to a “base” model, e.g., an intercept only model. Whether or not this shrinkage is desireable and whether or not more direct connections can be made with Lasso or penalized complexity priors are additional areas for research.

The linear combination of random variables, namely the intercept and partial slope coefficients in the linear model, is a convolution. Dropping the intercept and ignoring the effects of the covariate values in the linear combination, assuming all covariates equal 1, if identical distributions are assumed for the slope coefficients, which is a way to address the many-to-one problem mentioned previously, the moment generating function (MGF) for the linear combination is the product of $p$ identical MGFs:

$\displaystyle E[\exp(\eta)]$

$\displaystyle=E[\exp(t\times[\beta_{1}+\ldots+\beta_{p}])]=E[\exp(t\beta)]^{p},$

where $\beta$, $\beta_{1}$, $\ldots$, and $\beta_{p}$ are independent and identically distributed. This results in the MGF for $\beta$,

$\displaystyle E[\exp(t\beta)]$

$\displaystyle=\left(E[\exp(\eta)]\right)^{\frac{1}{p}}.$

If the MGF for $\eta$ is known, the MGF for $\beta$ the $p$th root of that MGF. Inversion of an arbitrary MGF to yield the underlying probability distribution is usually not a simple task let alone inversion of the $p$th root of a MGF (Walker, , 2017). While interesting, this approach seems impractical.

Another topic for research is to extend these results to multi-Bernoulli distributions. For example, an animal is at location $A$ at time $t$ and at time $t+1$ it could remain at $A$ or move to $B$ or move to $C$. Model location at next time step by a multi-Bernoulli:

$\displaystyle y_{t+1}|y_{t}$

$\displaystyle\sim\mbox{Mult-Bernoulli}(p_{1},p_{2},p_{3}).$

The probabilities are modelled by a multi-logistic transformation:

	$\displaystyle p_{1}$	$\displaystyle\propto\exp(\beta_{1,0}+\beta_{1,1}x_{1})$
	$\displaystyle p_{2}$	$\displaystyle\propto\exp(\beta_{2,0}+\beta_{2,1}x_{2})$
	$\displaystyle p_{3}$	$\displaystyle=1-p_{1}-p_{2}.$

If discrete uniform priors for $p_{1}$, $p_{2}$, and $p_{3}$ are desired, what do the priors for the above $\beta$’s need to be?

Here we have only addressed potentially problematic induced priors for Bernoulli parameters. Both Seaman III et al., (2012) and Lele, (2020) discussed this problem for non-Bernoulli settings. In particular Lele, (2020) compared two mathematically equivalent parameterizations for a population dynamics model in a state-space model framework, namely the Ricker model. The two formulations are

	$\displaystyle\mbox{Model A :}N_{t+1}$	$\displaystyle=N_{t}\times\exp(a-b\times N_{t})$		(13)
	$\displaystyle\mbox{Model B :}N_{t+1}$	$\displaystyle=N_{t}\times\exp(a\times(1-N_{t}/K)),$		(14)

where $a$ is a measure of intrinsic population growth rate, $b$ is a measure of density dependence, and $K$ is carrying capacity. $K$ can be calculated for the first model as $a/b$. He showed how apparently non-informative priors for the parameters yielded quite different posterior means for the population growth parameter and population carrying capacity.

One likely explanation for the discrepancy in the posterior means is that the induced priors for the functions of parameters with specified priors are quite different than the explicitly specified priors for those parameters in the other parameterization. Figure 9 shows how the induced distribution for carrying capacity $K$ in Model A is highly skewed compared to specified distribution for $K$ in Model B, with a similar result for the induced prior for the growth parameter $a$ in Model B compare to the specified prior in Model A.

In principle the same concepts used to yield a desired induced prior distribution for the Bernoulli parameter could be applied by specifying desired induced priors for the growth rate and the carrying capacity parameters—using the change-of-variables approach to work backwards to derive the prior probabilities on the lower level parameters that would induce the desired priors. However, the resulting derived prior distributions will typically be quite unique and Monte Carlo procedures will likely be required to generate samples from the probability function.