Distribution-Free Conditional Median Inference

We begin by setting up definitions to formalize the concepts above. Throughout this paper, we assume that any distribution $(X,Y)\sim P$ is over $\mathbb{R}^{d}\times\mathbb{R}$ unless explicitly stated otherwise. Each result in this paper holds true for all values of $d$ and $n$.

Given a feature vector $X_{n+1}$, we let $\hat{C}_{n}(X_{n+1})\subseteq\mathbb{R}$ denote a confidence interval for some functional of the conditional distribution $Y_{n+1}|X_{n+1}$. Note that we use the phrase confidence interval for convenience; in its most general form, $\hat{C}_{n}(X_{n+1})$ is a subset of $\mathbb{R}$. This interval is a function of the point $X_{n+1}$ at which we seek inference as well as of our training data $\mathcal{D}=\{(X_{1},Y_{1}),\ldots,(X_{n},Y_{n})\}$. We write $\hat{C}_{n}$ to refer to the general algorithm that maps $\mathcal{D}$ to the resulting confidence intervals $\hat{C}_{n}(x)$ for each $x\in\mathbb{R}^{d}$.

In order for $\hat{C}_{n}$ to be useful, we want it to capture, or contain, the parameter we care about with high probability. We formalize this as follows:

The probabilities in Definitions 1 and 2 are both taken over the training data $\mathcal{D}=\{(X_{1},Y_{1}),\ldots,(X_{n},Y_{n})\}$ and test point $X_{n+1}$. Thus, satisfying $(1-\alpha)$-Median is equivalent to satisfying $(1-\alpha,0.5)$-Quantile.

These concepts are similar to predictive coverage, with the key difference being that our goal is now to predict a function of $X_{n+1}$ rather than a new datapoint $Y_{n+1}$.

Finally, we define the type of conformity scores that we will use in our general quantile inference algorithm.

We find that there exists a distribution-free predictive inference algorithm that satisfies both $(1-\alpha/2)$-Predictive and $(1-\alpha)$-Median. Moreover, an improved version of this algorithm also satisfies $(1-\alpha,q)$-Quantile. Together, these prove that there exists nontrivial algorithms $\hat{C}_{n}$ that satisfy $(1-\alpha)$-Median and $(1-\alpha,q)$-Quantile for all $0<q<1$.

We go on to show that conditional median inference and predictive inference are nearly equivalent problems. Specifically, we show that any algorithm that contains $\textrm{Median}(Y_{n+1}|X_{n+1})$ with probability $1-\alpha$ must also contain $Y_{n+1}$ with probability at least $1-\alpha$, and that any algorithm that contains $Y_{n+1}$ with probability at least $1-\alpha/2$ must contain $\textrm{Median}(Y_{n+1}|X_{n+1})$ with probability $1-\alpha$.

Taken together, these results give us somewhat conflicting perspectives. On the one hand, there exist distribution-free algorithms that capture the conditional median and conditional quantile with high likelihood; on the other hand, any such algorithm will also capture a large proportion of the distribution itself, putting a hard limit on how well such algorithms can ever perform.

Algorithm 1 below operates by taking the training dataset and separating it into two halves of sizes $n_{1}+n_{2}=n$. Next, a regression algorithm $\hat{\mu}$ is trained on $\mathcal{D}_{1}=\{(X_{1},Y_{1}),\ldots,(X_{n_{1}},Y_{n_{1}})\}$. The residuals $Y_{i}-\hat{\mu}(X_{i})$ are calculated for $n_{1}<i\leq n$, and the $1-\alpha/2$ quantile of the absolute value of these residuals is then used to create a confidence band around the prediction $\hat{\mu}(X_{n+1})$. The expert will recognize that this is identical to a well-known algorithm from predictive inference as explained later.

The proof of Theorem 1 is covered in Appendix A.1.

We return at last to the connection with predictive inference. Introduced in Papadopoulos et al. (2002) and Vovk et al. (2005) and studied in Lei et al. (2013), Barber et al. (2021b), Romano et al. (2020), and several other papers, the split conformal method was initially created to achieve distribution-free predictive coverage guarantees. In particular, Vovk et al. (2005) shows that Algorithm 1 satisfies $(1-\alpha/2)$-Predictive, implying that in order to capture $\textrm{Median}(Y_{n+1}|X_{n+1})$ with probability $1-\alpha$, our algorithm produces a wider confidence interval than an algorithm trying to capture $Y_{n+1}$ with the same probability.

Algorithm 1 is a good first step towards a usable method for conditional median inference; however, it may be too rudimentary to be used in practice. Algorithm 2 is a more general version of Algorithm 1 that results in conditional quantile coverage and better empirical performance. This provides a better understanding of how diverse parameter inference algorithms can be.

Algorithm 2 differs from Algorithm 1 in two ways. First, we use the $rq$ quantile of the lower scores to create the confidence interval’s lower bound, and the $1-s(1-q)$ quantile of the upper scores (corresponding to the top $s(1-q)$ of the score distribution) for the upper bound. Second, the functions we fit are no longer regression functions, but instead locally nondecreasing conformity scores. These scores are described in Definition 4; see Remark 2.3 for examples.

The proof of Theorem 2 is covered in Appendix A.2 and is similar to that of Theorem 1; the main modifications come from the changes described above. Regarding the first change, the asymmetrical quantiles on the lower and upper end of the $E_{i}$’s balance the fact that datapoints have asymmetrical probabilities of being on either side of the conditional quantile. Regarding the second change, because the conformity scores $E_{i}$ still preserve relative ordering, they do not affect the relationship between datapoints and the conditional quantile.

Now that we have seen that Algorithms 1 and 2 achieve coverage, an important question to ask is whether or not the terms for the error quantile can be improved. Do our methods consistently overcover the conditional median, and if so, is it possible to take a lower quantile of the error terms and still have Theorems 1 and 2 hold? In this section, we prove that this is impossible by going over a particular distribution $P^{\delta}$ for which the $1-\alpha/2$ term in Algorithm 1 is necessary. Additionally, we go over a choice $\hat{\mu}_{c}$ with the property that Algorithm 1 always results in $1-\alpha/2$ coverage when ran with input $\hat{\mu}_{c}$; this implies that there does not exist a distribution with the property that Algorithm 1 will always provide a sharp confidence interval for the conditional median regardless of the regression algorithm.

For each $\delta>0$, consider $(X,Y)\sim P^{\delta}$ over $\mathbb{R}\times\mathbb{R}$, where $P^{\delta}_{X}=\textrm{Unif}[-0.5,0.5]$ and $Y|X\stackrel{{\scriptstyle\textnormal{d}}}{{=}}X\,B$; $B\in\{0,1\}$ is here an independent Bernoulli variable with $\mathbb{P}(B=1)=0.5+\delta$. That is, $0.5+\delta$ of the distribution is on the line segment $Y=X$ from $(-0.5,-0.5)$ to $(0.5,0.5)$, and $0.5-\delta$ of the distribution is on the line segment $Y=0$ from $(-0.5,0)$ to $(0.5,0)$. Thus, $\textrm{Median}(Y|X=x)=x$. A visualization of $P^{\delta}$ is shown in Figure 1.

We know that Algorithm 1 is accurate for all distributions $P$ and all algorithms $\hat{\mu}$. Consider the regression algorithm $\hat{\mu}:\mathbb{R}\to\mathbb{R}$ such that $\hat{\mu}(x)=0$ for all $x\in\mathbb{R}$; in other words, $\hat{\mu}$ predicts $Y_{i}=0$ for all $X_{i}$. We show that Algorithm 1 returns a coverage almost exactly equal to $1-\alpha$.

The proof is in Appendix A.3. Theorem 3 does not directly prove that the $1-\alpha/2$ term in Algorithm 1 is sharp. However, we can see that if Algorithm 1 used the $1-\alpha^{\prime}/2$ quantile of the residuals with $\alpha^{\prime}>\alpha$, then by Theorem 3 there would exist a choice for $\delta$ and $n$ where the probability of conditional median coverage would be less than $1-\alpha$. Therefore, the $1-\alpha/2$ term is required for the probability of coverage to always be at least $1-\alpha$.

These results may seem somewhat pedantic because we are restricting $\hat{\mu}(x)$ to be the zero function and $f^{\textnormal{lo}}(X_{i},Y_{i})$ and $f^{\textnormal{hi}}(X_{i},Y_{i})$ to be $Y_{i}$; this simplification is done to better illustrate our point. Even when $f^{\textnormal{lo}}(X_{i},Y_{i})$ and $f^{\textnormal{hi}}(X_{i},Y_{i})$ are trained using more complicated approaches, there still exist distributions that result in only $1-\alpha$ coverage for Algorithm 2. For an example of a distribution where Algorithm 2 only achieves $1-\alpha$ coverage for standard conformity scores $f^{\textnormal{lo}}(X_{i},Y_{i})$ and $f^{\textnormal{hi}}(X_{i},Y_{i})$, refer to $P_{3}$ in Section 4. The existence of $P^{\delta}$ and similarly ‘confusing’ distributions helps to show why capturing the conditional median can be tricky in a distribution-free setting.

At the same time, there exist conformity scores for which Algorithms 1 and 2 have rates of coverage that are always near $1-\alpha/2$. For $c>0$, define the randomized regression function $\hat{\mu}_{c}$ as follows: set $M=\displaystyle\max_{i\in\mathcal{I}_{1}}|Y_{i}|$ and $\hat{\mu}_{c}(x)=A_{x}$ for all $x\in\mathbb{R}^{d}$, where $A_{x}\overset{i.i.d.}{\sim}\mathcal{N}(0,(cM)^{2})$. We prove the following:

The proof for this theorem is in Appendix A.4.

Due to the definition of $\hat{\mu}_{c}$, the resulting confidence intervals will be near-useless; the predictions will be so far off that the intervals will have width several times the range of the (slightly clipped) marginal distribution $P_{Y}$. However, they still will be finite, and will still achieve predictive inference at a rate roughly equal to $1-\alpha/2$. The existence of scores that always result in higher-than-needed rates of coverage means that any result like Theorem 3 that provides a nontrivial upper bound for either Algorithm 1 or Algorithm 2’s coverage of the conditional median on a specific distribution will have to restrict the class of regression functions and/or conformity scores.