Orthogonal Causal Calibration

Justin Whitehouse Stanford University Christopher Jung Meta Research Vasilis Syrgkanis Stanford University Bryan Wilder Carnegie Mellon University Zhiwei Steven Wu Carnegie Mellon University

Abstract

Estimates of heterogeneous treatment effects such as conditional average treatment effects (CATEs) and conditional quantile treatment effects (CQTEs) play an important role in real-world decision making. Given this importance, one should ensure these estimates are calibrated. While there is a rich literature on calibrating estimators of non-causal parameters, very few methods have been derived for calibrating estimators of causal parameters, or more generally estimators of quantities involving nuisance parameters.

In this work, we develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model. Throughout, we study a notion of calibration defined with respect to an arbitrary, nuisance-dependent loss $\ell$ , under which we say an estimator $\theta$ is calibrated if its predictions cannot be changed on any level set to decrease loss. For losses $\ell$ satisfying a condition called universal orthogonality, we present a simple algorithm that transforms partially-observed data into generalized pseudo-outcomes and applies any off-the-shelf calibration procedure. For losses $\ell$ satisfying a weaker assumption called conditional orthogonality, we provide a similar sample splitting algorithm the performs empirical risk minimization over an appropriately defined class of functions. Convergence of both algorithms follows from a generic, two term upper bound of the calibration error of any model: one term that measures the error in estimating unknown nuisance parameters and another that measures calibration error in a hypothetical world where the learned nuisances are true. We demonstrate the practical applicability of our results in experiments on both observational and synthetic data. Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings, with additional loss only arising from errors in nuisance estimation.

1 Introduction

Estimates of heterogeneous causal effects such as conditional average treatment effects (CATEs), conditional average causal derivatives (CACDs), and conditional quantile treatment effects (CQTEs) play a pervasive role in understanding various statistical, scientific, and economic problems. Due to the partially-observed nature of causal data, estimating causal quantities is naturally more difficult than estimating non-causal ones. Despite this, a vast literature has developed focused on estimating causal effects using both classical statistical methods [34, 36, 46, 1, 19] and modern machine learning (ML) approaches [18, 7, 6, 53, 14, 30]. In this work, we focus on an complementary and relatively understudied direction — developing algorithms for calibrating estimates of heterogeneous causal effects.

Calibration is a notion of model consistency that has been extensively studied both in theory and practice by the general ML community [23, 43, 39, 42, 56, 31, 50, 29, 13, 12, 17]. Classically, a model $\widehat{\theta}(X)$ predicting $Y$ is said to be calibrated if, when it makes a given prediction, the observed outcome on average is said prediction, i.e. if

\mathbb{E}[Y\mid\widehat{\theta}(X)=v]=v\qquad\text{for any }v\in\mathrm{range}(\widehat{\theta}).

There are deep connections between calibrated predictions and optimal downstream decision-making [48, 47]. Namely, for many utility functions, when a decision maker is equipped with calibrated predictions, the optimal mapping from prediction to decision simply treats those predictions as if they were the ground truth. Given that models predicting heterogeneous causal effects or counterfactual quantities are leveraged in decision-making in domains such as medicine [15, 37], advertising and e-commerce [32, 20], and public policy [61], it is essential that these estimates are calibrated. In non-causal settings, there exists a plethora of simple algorithms for performing calibration such of algorithms such as histogram binning [25], isotonic calibration [2, 58] , linear calibration, and Platt scaling [51, 45]. These algorithms are all univariate regression procedures, regressing an observed outcome $Y$ onto a model prediction $\widehat{\theta}(X)$ over an appropriately-defined function class. However, given the partially-observed nature of causal problems, it is non-obvious how to extend these algorithms for calibrating estimates of general causal effects.

To illustrate the importance and difficulty of calibrating causal models, we look at the example of a doctor using model predictions to aid in prescribing medicine to prevent heart attacks. The doctor may have access to observations of the form $Z=(X,A,Y)$ , where $X\in\mathcal{X}$ represent covariates, $A\in\{0,1\}$ is a binary treatment, and $Y=AY(1)+(1-A)Y(0)\in\mathbb{R}$ represents an observed outcome. For instance, a doctor may use a patient’s age, height, and blood pressure (covariates $X$ ) to decide whether or not to give medicine (treatment $A$ ) to a patient in order to prevent a heart attack (outcome $Y$ ). To aid in their decision making, the doctor may want to employ a model $\theta(X)$ predicting the conditional average treatment effect (CATE), which is defined as

\theta_{\mathrm{CATE}}(x):=\mathbb{E}[Y(1)-Y(0)\mid X=x]

In this example, the CATE measures the difference in probability of a heart attack under treatment and control. Naturally, a doctor may decide to prescribe medication to a patient if $\theta(X)<0$ , i.e. if the model predicts the probability of a heart attack will go down under treatment. If the model is miscalibrated—for instance if we have

\mathbb{E}[Y(1)-Y(0)\mid\theta(X)=-0.15]>0

the doctor may actually harm patients by prescribing medication! If $\theta(X)$ were calibrated, such a risk would not a occur. However, the doctor cannot run one of the aforementioned calibration algorithms because they only ever observe the outcome under treatment $Y(1)$ or control $Y(0)$ , never the target individual treatment effect $Y(1)-Y(0)$ .

To work around this, one typically needs to estimate nuisance parameters — functions of the underlying data-generating distribution that are used to “de-bias” partial observations (either $Y(1)$ or $Y(0)$ in the previous example) into pseudo-outcomes: quantities that, on average, look like the target treatment effect (here $Y(1)-Y(0)$ ). For the CATE, these nuisance functions are the propensity score $\pi_{0}(x):=\mathbb{P}(A=1\mid X=x)$ and the expected outcome mapping $\mu_{0}(a,x):=\mathbb{E}[Y\mid X=x,A=a]$ , and pseudo-outcomes are of the form [35]

\chi_{\mathrm{CATE}}((\mu,\pi);W):=\mu(1,X)-\mu(0,X)+\left(\frac{A}{\pi(X)}-\frac{1-A}{1-\pi(X)}\right)(Y-\mu(A,X)).

Naturally, the nuisances and pseudo-outcomes will naturally be different for other heterogeneous causal effects and can typically be derived by the statistician.

What is known about causal calibration? In the setting of the above CATE example, if we assume conditional ignorability¹¹1This is the condition that the potential outcomes $Y(0)$ and $Y(1)$ are conditionally independent of the treatment $A$ given covariates $X$ , i.e. $Y(0),Y(1)\perp A\mid X$ , van der Laan et al. [58] show that by estimating the propensity score and expected outcome mappings, one can perform isotonic regression of the doubly-robust pseudo-outcomes $\chi_{\mathrm{CATE}}$ onto model predictions $\theta(X)$ to ensure asymptotic calibration. However, this result is limited in application, and doesn’t extend to other heterogeneous effects of interest like conditional average causal derivatives, conditional local average treatment effects, or even CATEs in the presence of unobserved confounding (in which case one may need instrumental variables). Furthermore, the result of van der Laan et al. [58] don’t allow a learner to calibrate using other algorithms like Platt scaling, histogram binning, or even simple linear regression. While one could derive individual results for calibrating an estimate of “parameter A” under “algorithm B”, this would likely lead to a repeated reinventing of the wheel. The question considered in this paper is thus as follows: can one construct a framework that allows a statistician to calibrate estimates of general heterogeneous causal effects using an arbitrary, off-the-shelf calibration algorithms?

1.1 Our Contributions

In this paper, we reduce the problem of causal calibration, or calibrating estimates of heterogeneous causal effects, to the well-studied problem of calibrating non-causal predictive models. We assume the scientist is interested in calibrating an estimate $\theta(X)$ of some heterogeneous causal effect $\theta_{0}(X)$ that can be specified as the conditional minimizer of a loss function, i.e. $\theta_{0}(x)\in\arg\min_{\nu}\mathbb{E}[\ell(\nu,g_{0};W)\mid X=x]$ . Here, $\ell(\nu,g;w)$ involves a nuisance component, and we assume that there exists a true, unknown nuisance parameter $g_{0}$ . We say a $\theta$ is perfectly calibrated with respect to $\ell$ if $\mathbb{E}[\partial\ell(\theta(X),g_{0};Z)\mid\theta(X)]=0$ ,²²2Here, $\partial\ell(\theta(x),g;z)$ denotes the partial derivative of $\ell$ with respect to its first argument $\theta(x)$ , i.e. the quantity is defined by $\frac{\partial}{\partial\nu}\ell(\nu,g;z)|_{\nu=\theta(x)}$ . and that $\theta$ is approximately calibrated if the $L^{2}$ error

\mathrm{Cal}(\theta,g)=\mathbb{E}\left(\mathbb{E}[\partial\ell(\theta(X),g;Z)\mid\theta(X)]^{2}\right)^{1/2}

is small when $g$ is set as the true nuisance $g_{0}$ . In words, $\theta$ is calibrated if the predictions of $\theta$ are “unimprovable” with respect to current level sets, a condition similar to the one outlined by Noarov and Roth [47] and the concept of swap regret [4, 17].

As a concrete example, $\theta_{\mathrm{CATE}}(X)$ is the conditional minimizer of the doubly-robust loss $\ell_{\mathrm{CATE}}(\nu,(\mu,\pi);z)$ given by $\ell_{\mathrm{CATE}}(\nu,(\mu,\pi);z):=\frac{1}{2}\Big{(}\nu-\chi_{\mathrm{CATE}}((\mu,\pi);z)\Big{)}^{2}$ , where $\pi_{0}$ , $\mu_{0}$ , and $\chi_{\mathrm{CATE}}$ are as defined above. Perfect calibration for a CATE estimate $\theta$ becomes $\mathbb{E}[Y(1)-Y(0)\mid\theta(X)]=\theta$ , and approximate $L^{2}$ calibration becomes $\mathrm{Cal}(\theta,(\mu_{0},\pi_{0})):=\mathbb{E}\left(\left(\theta(X)-\mathbb{E}[Y(1)-Y(0)\mid\theta(X)]\right)^{2}\right)\approx 0$ .

Our reduction and algorithms are based around a robustness condition on the underlying loss $\ell$ called Neyman Orthogonality. We say a loss $\ell$ is Neyman orthogonal if

D_{g}\mathbb{E}[\partial\ell(\theta_{0}(X),g_{0};Z)\mid X](g-g_{0})=0,

(1)

i.e. if $\ell$ is insensitive to small estimation errors in the nuisance parameters $g_{0}$ and loss minimizing parameter $\theta_{0}$ .³³3Here, $D_{g}$ denotes a Gateaux derivative. In our work, we consider two mild variants of Neyman orthogonality. The first is universal orthogonality [18], a stronger notion of orthogonality in which $\theta_{0}(x)$ in Equation (1) can be replaced by any estimate $\theta(x)$ . Examples of causal effects that minimize such a loss are CATEs, conditional average causal derivatives (CACD), and conditional local average treatment effects (CLATEs). Another condition we consider is called conditional orthogonality, in which instead of conditioning of covariates $X$ in Equation (1), one conditions on a post-processing $\varphi(X)$ of covariates instead. Conditional orthogonality is a natural condition for calibration tasks, as we ultimately care about assessing the quality of an estimator $\theta(X)$ conditional on its own predictions. An example causal parameter that can be specified as the minimizer of a conditionally orthogonal loss is the conditional quantile under treatment (CQUT). Our specific contributions, broken down by assumption on the loss, are as follows:

1.

In Section 3, we study calibration with respect to universally orthogonal loss functions. We present a sample splitting algorithm (Algorithm 1) that uses black-box ML algorithms to estimate unknown nuisance functions on the first half of the data, transforms the second half into de-biased “pseudo-outcomes”, and then applies any off-the-shelf calibration algorithm on the second half of the data treating pseudo-outcomes as true labels. We provide high-probability, finite-sample guarantees for our algorithm that only depend on convergence rates for the black-box nuisance estimation and calibration algorithms. We additionally provide a cross-calibration algorithm (Algorithm 2) that makes more efficient use of the data and show that our calibration algorithms do not significantly increase the risk of a model (Appendix C).
2.

In Section 4, we study the calibration of models predicting effects that minimize a conditionally orthogonal loss function. In this more general setting, we describe a similar sample splitting algorithm that uses half of the data lo learn nuisances and then runs an off-the-shelf algorithm for “generalized’ calibration on the second half the data. We provide examples of such calibration algorithms in Appendix D, which are motivated by extending algorithms like isotonic calibration and linear calibration to general losses. We also prove finite-sample guarantees for this algorithm and construct a similar cross-calibration algorithm (Algorithm 4).
3.

Lastly Section 5, we empirically evaluate the performance of the aforementioned algorithms on a mix of observational and semi-synthetic data. In one experiment, we use observational data alongside Algorithm 2 to show how one can calibrate models predicting the CATE of 401(k) eligibility and CLATE of 401(k) participation on an individual’s net total financial assets. Likewise, in another experiment on synthetically generated data, we show how Algorithm 4 can be used to both decrease $L^{2}$ calibration error and average loss for models predicting CQUTs.

The design of our algorithms is inspired by a generic upper bound on the calibration error of any estimator $\theta(X)$ . We show that the $L^{2}$ calibration error of any estimator can be bounded above by two, decoupled terms: one involving nuisance estimation error and another representing calibration error under the orthogonalized loss evaluated at the learned nuisances.

Informal Theorem 1.

Suppose $\theta:\mathcal{X}\rightarrow\mathbb{R}$ is some estimator, $\ell$ is some base loss, and $\widetilde{\ell}$ is the corresponding orthogonalized loss. Let $g_{0}$ denote the true, unknown nuisance functions, and $g$ arbitrary nuisance estimates. We have

\underbrace{\mathrm{Cal}(\theta,g_{0})}_{\text{$L^{2}$ calibration error under $\ell(\theta(x),g_{0};w)$}}\lesssim\underbrace{\mathrm{err}(g,g_{0})}_{\text{nuisance estimation error}}\qquad\quad+\underbrace{\mathrm{Cal}(\theta,g)}_{\text{$L^{2}$ calibration error under $\ell(\theta(x),g;w)$}}

We view Informal Theorem 1 as a “change of measure” or “change of nuisance” result, allowing the learner to essentially pretend our learned nuisances represent reality while only paying a small error for misestimation. In many settings where there are two nuisances functions (e.g. $g=(\eta,\zeta)$ and $g_{0}=(\eta_{0},\zeta_{0})$ ), we will have $\mathrm{err}(g,g_{0})=O(\|(\eta-\eta_{0})\cdot(\zeta-\zeta_{0})\|)$ , and thus the error will be small if we estimate at least one nuisance function sufficiently well. This in particular is the case for the aforementioned CATE, where $g_{0}=(\mu_{0},\pi_{0})$ , as noted above. More broadly, we will have $\mathrm{err}(g,g_{0})=O(\|g-g_{0}\|^{2})$ . While simple in appearance, the above bound depends on deep connections between Neyman orthogonality and calibration error. We prove a bound of the above form in each of the main sections of the paper. This decoupled bound naturally suggest using some fraction of the data to learn nuisance parameters, thus minimizing the first term, and then using fresh data with the learned nuisances to perform calibration, thus minimizing the second term.

Given that our framework and results are quite general, we go through concrete examples to help with building intuition. In fact, we show that the work of van der Laan et al. [58] can be seen as a special instantiation of our framework.

1.2 Related Work

Calibration:

Important to our work is the vast literature on calibration. Calibration was considered first in the context producing calibrated probabilities, both in the online [12, 17] and i.i.d. [51, 62] settings, but has since been considered in other contexts such as distribution calibration [54], threshold calibration [52, 59], and parity calibration [11]. Calibration is typically orthogonal to model training, and usually occurs as a simple post-processing routine. Some well-known algorithms for post-hoc calibration include Platt scaling [51, 26], histogram binning [62, 25], and isotonic regression [63, 2]. Many of these algorithms simultaneously offer strong theoretical guarantees (see Gupta [24] for an overview) and strong empirical performance when applied to practically-relevant ML models [23]. We view our work as complementary to existing, non-causal results on calibration. Our two-step algorithm allows a practitioner to directly apply any of the above listed algorithms, inheriting existing error guarantees so long as nuisance estimation is efficiently performed.

Double/debiased Machine Learning:

In our work, we also draw heavily from the literature on double/debiased machine learning [6, 9, 8]. Methods relating to double machine learning aim to eschew classical non-parametric assumptions (e.g. Donsker properties) on nuisance functions, often through simple sample splitting schemes [28, 34, 3]. In particular, if target population parameters are estimated using a Neyman orthogonal loss function [44, 18], then these works show that empirical estimates of the population parameters converge rapidly to either a population or conditional loss minimizer.

Of the various works related to double/debiased machine learning, we draw most heavily on ideas from the framework of orthogonal statistical learning [18]. In their work, Foster and Syrgkanis [18] develop a simple two-step framework for statistical learning in the presence of nuisance estimation. In particular, they show that when the underlying loss is Neyman orthogonal, then the excess risk can be bounded by two decoupled error terms: error from estimating nuisances and the error incurring from applying a learning algorithm with a fixed nuisance. Following its introduction, the orthogonal statistical learning framework has found applications in tasks such as the design of causal random forests [49] and causal model ensembling via Q-aggregation [40]. In this work, we show that central ideas from orthogonal statistical learning are naturally applicable to the problem of calibrating estimators of causal parameters.

Lastly, our work can be seen as a significant generalization of existing results on the calibration of causal parameters. Primarily, we compare our results to the work of van der Laan et al. [58]. In their work, the authors construct a sample-splitting scheme for calibrating estimates of conditional average treatment effects (CATEs). The specific algorithm leveraged by the authors uses one half of the data to estimate nuisance parameters, namely propensities and expected outcomes under control/treatment. After nuisances are learned, the algorithm transforms the second half of the data into pseudo-observations and runs isotonic regression as a calibration procedure. Our results are applicable to estimates of any causal parameter that can be specified as the population minimizer of a loss function, not just CATEs. Additionally, our generic procedure allows the scientist to plug in any black-box method for calibration, not just a specific algorithm such as isotonic regression. Likewise, our work is also significantly more general than the work of Leng and Dimmery [41], who provide a maximum-likelihood based approach for performing linear calibration, a weaker notion of calibration, of CATE estimates.

2 Calibration of Causal Effects

We are interested in calibrating some estimator/ML model $\theta\in\Theta\subset\{f:\mathcal{X}\rightarrow\mathbb{R}\}$ that is predicting some heterogeneous causal effect $\theta_{0}(x)$ . We assume $\theta_{0}$ is specified as the conditional minimizer of some loss $\ell(\theta(x),g;z)$ , i.e.

\theta_{0}(x)\in\arg\min_{\nu\in\mathbb{R}}\mathbb{E}[\ell(\nu,g_{0};Z)\mid X=x].

We assume $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ is some generic loss function involving nuisance. We assume $\mathcal{Z}$ is some space containing observations, and write $Z$ as a prototypical random element from this space, and $P_{Z}$ as the distribution on $\mathcal{W}$ from which $W$ is drawn. In the above, $\mathcal{G}$ is some space of nuisance functions, which are of the form $g:\mathcal{W}\rightarrow\mathbb{R}^{d}$ . We assume that there is some true nuisance parameter $g_{0}\in\mathcal{G}$ , but that this parameter is unknown to the learner and must be estimated. We generally assume $\mathcal{G}$ is a subset of $L^{2}(P_{W}):=L^{2}(\mathcal{W},P_{W})$ , and so as a norm we can consider the $\|g\|_{L^{2}(P_{W})}:=\int_{\mathcal{W}}\|g(w)\|^{2}P_{W}(dw)$ , where $\|\cdot\|$ denotes the standard Euclidean norm on $\mathbb{R}^{d}$ . Given a function $T:\mathcal{G}\rightarrow\mathbb{R}$ and functions $f,g\in\mathcal{G}$ , we let $D_{g}T(f)(g)$ and $D_{g}^{2}T(f)(g,g)$ denote respectively the first and second Gateaux derivatives of $T$ at $f$ in the direction $g$ .

We typically have $Z=(X,A,Y)$ , where $X\in\mathcal{X}$ represents covariates, $A\in\mathcal{A}$ represents treatment, and $Y\in\mathbb{R}$ represents an outcome in an experiment. More generally, we assume the nested structure $\mathcal{X}\subset\mathcal{W}\subset\mathcal{Z}$ , where $\mathcal{X}$ intuitively represents the space of covariates or features, $\mathcal{W}$ represents an extended set of parameters on which the true nuisance parameter may also depend (e.g. instrumental variables, whether or not an individual actually accepted a treatment), and $\mathcal{Z}$ may contain additional observable information (e.g. outcome $Y$ under the given treatment). We write the marginal distributions of $X$ and $W$ respectively as $P_{X}$ and $P_{W}$ . We typically write $\ell(\theta,g;w)$ instead of $\ell(\theta(x),g;w)$ for succinctness, and we let $\partial\ell(\theta,g;w):=\frac{\partial}{\partial\theta(x)}\ell(\theta(x),g;w)$ be the partial derivative of $\ell$ with respect to it’s first argument, $\theta(x)$ .

In our work, we consider a general definition of calibration error that holds for any loss involving a nuisance component. This general notion of calibration, which is similar to notions that have been considered in Noarov and Roth [47], Gopalan et al. [22], Foster and Vohra [16], and Globus-Harris et al. [21], implies an estimator cannot be “improved” on any level set of its prediction.

Definition 2.1.

Let $\theta:\mathcal{X}\rightarrow\mathbb{R}$ be an estimator, $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ a nuisance-dependent loss function, and $g\in\mathcal{G}$ a fixed nuisance parameter. We define the $L^{2}$ calibration error of $\theta$ with respect to $\ell$ and $g$ to be

	$\displaystyle\mathrm{Cal}(\theta,g)$	$\displaystyle:=\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]^{2}\right)^{1/2}$
		$\displaystyle=\left(\int_{\mathcal{X}}\mathbb{E}[\partial\ell(\theta,g;Z)\mid\theta(X)=\theta(x)]^{2}P_{X}(dx)\right)^{1/2}.$

We say $\theta$ is perfectly calibrated if $\mathrm{Cal}(\theta,g_{0})=0$ , where $g_{0}$ is the true, unknown nuisance parameter.

In the special case where the loss $\ell(\nu,y)$ is the squared loss and doesn’t involve a nuisance component, we recover a more classical notion of calibration error.

Definition 2.2 (Classical calibration error).

Let $\theta:\mathcal{X}\rightarrow\mathbb{R}$ be a fixed estimator, and let $(X,Y)\sim P$ , where $P$ is some arbitrary distribution on $\mathcal{X}\times\mathbb{R}$ . The classical $L^{2}$ calibration error is defined by

\mathrm{Cal}(\theta;P):=\int_{\mathcal{X}}\left(\theta(x)-\mathbb{E}_{P_{Y}}[Y\mid\theta(X)=\theta(x)]\right)^{2}P_{X}(dx),

where we make dependence on the underlying distribution $P$ clear for convenience.

We will leverage the classical $L^{2}$ calibration error in the sequel when reasoning about the convergence of our sample splitting algorithm. The key for now is that Definition 2.1 generalizes the above definition to arbitrary losses involving a nuisance component.

We are always interested in controlling $\mathrm{Cal}(\theta,g_{0})$ . In words, $\theta$ is calibrated if, on the level set $\{x\in\mathcal{X}:\theta(x)=\nu\}$ , there is no constant value $\omega\in\mathbb{R}$ we can switch the prediction $\theta(x)$ to to obtain lower loss. We can gleam further semantic meaning from looking at several additional examples below.

Example 2.3.

Below, we almost always assume observations are of the form $Z=(X,A,Y)$ , where $X$ are covariates, $A\in\{0,1\}$ or $[0,1]$ indicates treatment, and $Y=\sum_{a}Y(a)\mathbbm{1}_{A=a}$ indicates an outcome. We assume conditional ignorability of the treatment given covariates. The one exception is for conditional local average treatment effects, when assigned treatment may be ignored by the subject.

Conditional Average Treatment Effect: Recalling from earlier that the CATE $\theta_{\mathrm{CATE}}(x):=\mathbb{E}[Y(1)-Y(0)\mid X=x]$ can be specified as the conditional minimizer of the doubly-robust loss $\ell_{\mathrm{CATE}}$ given by $\ell_{\mathrm{CATE}}(\nu,(\mu,\pi);z):=\frac{1}{2}\left(\nu-\chi_{\mathrm{CATE}}((\mu,\pi);z)\right)^{2}$ with $\chi_{\mathrm{CATE}}$ given by

\chi_{\mathrm{CATE}}((\mu,\pi);z):=\mu(1,x)-\mu(0,x)+\left(\frac{a}{\pi(x)}-\frac{1-a}{1-\pi(x)}\right)(y-\mu(a,x)).

(2)

Here, the true nuisances are $\pi_{0}(x):=\mathbb{P}(A=1\mid X=x)$ and $\mu_{0}(a,x):=\mathbb{E}[Y\mid X=x,A=a]$ . It is clear perfect calibration becomes $\mathbb{E}[Y(1)-Y(0)\mid\theta(X)]=\theta(X)$ .

Conditional Average Causal Derivative: In a setting where treatments are not binary, but rather continuous (e.g. $A\in[0,1]$ ), it no longer makes sense to consider treatment effects as a difference. Instead, we can consider the conditional average causal derivative, which is defined by

\theta_{\mathrm{ACD}}(x):=\mathbb{E}[\partial_{a}Y(A)\mid X]

$\theta_{\mathrm{ACD}}$ is in fact the conditional minimizer of the loss $\ell_{\mathrm{ACD}}$ given by

\ell_{\mathrm{ACD}}(\theta,g;z)=\frac{1}{2}(\theta(x)-\chi_{\mathrm{ACD}}(g;Z))^{2}\qquad\chi_{\mathrm{ACD}}(g;Z)=\partial_{a}\mu(a,x)+\frac{\partial_{a}\pi(a\mid x)}{\pi(a\mid x)}(y-\mu(a,x))

where $g_{0}=(\mu_{0},\pi_{0})$ , $\mu_{0}(a,x)=\mathbb{E}[Y\mid X=x,A=a]$ is again the expected outcome mapping, $\pi_{0}(a\mid x)$ is the density of the treatment given covariates (we could alternatively directly estimate the nuisance $\zeta_{0}(a,x)=\frac{\partial_{a}\pi_{0}(a\mid x)}{\pi_{0}(a\mid x)}$ — more on this later). Naturally, $\theta$ is perfectly calibrated with respect to $\ell_{\mathrm{ACD}}$ if

\theta(X)=\mathbb{E}\left(\partial_{a}\mathbb{E}[Y(A)\mid X]\mid\theta(X)\right).

Conditional Local Average Treatment Effect: In settings with non-compliance, the prescribed treatment $D\in\{0,1\}$ given to an individual may not be equivalent to the received treatment $A\in\{0,1\}$ . Formally, we have $Z=(X,D,A,Y)$ , where we assume $A=A(1)D+A(0)(1-D)$ , and $Y=AY(1)+(1-A)Y(0)$ , where $A(d),Y(a)$ represent potential outcomes for treatment assignment $a,d\in\{0,1\}$ . We also assume monotonicty, i.e. that $D(1)\geq D(0)$ , and that the propensity for the recommended treatment $\pi_{0}(x):=\mathbb{P}(D=1\mid X=x)$ is known. The parameter of interest here is

\theta_{\mathrm{LATE}}(x):=\mathbb{E}[Y(1)-Y(0)\mid A(1)>A(0),X=x],

which is identified (following standard computations, see Lan and Syrgkanis [40]) as

\theta_{\mathrm{LATE}}(x)=\frac{\mathbb{E}[Y\mid A=1,X=x]-\mathbb{E}[Y\mid A=0,X=x]}{\mathbb{E}[D\mid A=1,X=x]-\mathbb{E}[D\mid A=0,X=x]}=:\frac{p_{0}(x)}{q_{0}(x)}.

(3)

It follows that $\theta_{\mathrm{LATE}}$ conditionally minimizes the following somewhat complicated loss:

	$\displaystyle\ell_{\mathrm{LATE}}(\theta,g;z)=\frac{1}{2}\left(\theta(x)-\chi_{\mathrm{LATE}}(g;z)\right)^{2}\qquad\text{where}$
	$\displaystyle\chi_{\mathrm{LATE}}(g;z):=\frac{p(x)}{q(x)}+\frac{a(d-\pi_{0}(x))}{q(x)}\left(\frac{p(x)}{q(x)}-\frac{y(d-\pi_{0}(x))}{a(d-\pi_{0}(x))}\right).$		(4)

Calibration with respect to $\ell_{\mathrm{LATE}}$ becomes

\theta(X)=\mathbb{E}\left[Y(1)-Y(0)\mid D(1)>D(0),\theta(X)\right].

4.

Conditional Quantile Under Treatment: Lastly, we consider the conditional $Q$ th quantile under treatment, which, assume $Y(1)$ admits a conditional Lebesgue density, is specified as $\theta_{\mathrm{QUT}}(x)=F^{-1}_{Y(1)}(Q\mid X=x)$ ⁴⁴4 $F_{Y(1)}(\cdot\mid X=x)$ here denotes the conditional CDF of $Y(1)$ given covariates $X$ . More generally, the $Q$ th quantile under treatment is specified as

$\theta_{\mathrm{QUT}}(x)\in\arg\min_{\nu\in\mathbb{R}}\mathbb{E}\left[\ell_{\mathrm{QUT}}(\nu,p_{0};Z)\mid X=x\right].$

In the above, $\ell_{\mathrm{QUT}}:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ denotes the $Q$ -pinball loss, which is defined as

$\ell_{\mathrm{QUT}}(\theta,p;z):=ap(x)(y-\theta(x))\left(Q-\mathbbm{1}_{y\leq\theta(x)}\right),$

where the true, unknown nuisance is the inverse propensity score $p_{0}(x):=\frac{1}{\pi_{0}(x)}$ , where $\pi_{0}(x):=\mathbb{P}(A=1\mid X=x)$ . A direct computation yields that calibration under $\ell_{\mathrm{QUT}}$ becomes

$\mathbb{P}\left(Y(1)\leq\theta(X)\mid\theta(X)\right)=Q.$

The first three losses considered above are “easy” to calibrate with respect to, as $\ell_{\mathrm{CATE}},\ell_{\mathrm{ACD}},$ and $\ell_{\mathrm{LATE}}$ are universally orthogonal losses, a concept discussed in Section 3. The pinball loss, on the other hand, is the quintessential example of a “hard” loss to calibrate with respect to. Handling more complicated losses like this is the subject of Section 4.

3 Calibration for Universally Orthogonal Losses

In this section, we develop algorithms for calibrating estimates $\theta$ of causal effects $\theta_{0}$ that are specified as conditional minimizers of universally orthogonal losses $\ell$ . Universal orthogonality, first introduced in Foster and Syrgkanis [18], can viewed as a robustness property of losses that are “close” to squared losses. Heuristically, a loss is universally orthogonal if it is insensitive to small errors in estimating the nuisance functions regardless of the current estimate on the conditional loss minimizer, i.e. Equation (1) holds when $\theta_{0}$ is replaced by any function $\theta:\mathcal{X}\rightarrow\mathbb{R}$ . We formalize this in the following definition.

Definition 3.1 (Universal Orthogonality).

Let $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ be a loss involving nuisance, and let $g_{0}$ denote the true nuisance parameter associated with $\ell$ . We say $\ell$ is universally orthogonal, if for any $\theta:\mathcal{X}\rightarrow\mathbb{R}$ and $g\in\mathcal{G}$ , we have

D_{g}\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid X\right](g-g_{0})=0.

By direct computation, one can verify that $\ell_{\mathrm{CATE}},\ell_{\mathrm{ACD}},$ and $\ell_{\mathrm{LATE}}$ all satisfy Definition 3.1, whereas $\ell_{\mathrm{QUT}}$ does not — this aligns with the idea of universally orthogonal losses behaving like squared loss. The following example gives a general class of losses that are universally orthogonal.

Example 3.2.

Assume that we have a vector of nuisances $g_{0}=(\eta_{0},\zeta_{0})$ for some $\eta_{0},\zeta_{0}:\mathcal{W}\rightarrow\mathbb{R}^{k}$ . Further, assume there is some function $m(\eta;z)$ such that⁵⁵5Here, $\zeta_{0}$ can be thought of a the (conditional) Riesz representer of the linear functional $D_{\eta}\mathbb{E}[m(\eta;Z)\mid X]$ .

D_{\eta}\mathbb{E}[m(\eta;Z)\mid X](\eta-\eta_{0})=\mathbb{E}[\langle\zeta_{0}(W),(\eta-\eta_{0})(W)\rangle\mid X]\qquad\text{for any }g-g_{0}\in\mathcal{G}.

Then, any loss $\ell(\theta,g;z)$ that obeys the score equation

\partial\ell(\theta,g;z)=\theta(x)-\underbrace{m(\eta;z)+\mathrm{Corr}(g;z)}_{=:\chi(g;z)}

(5)

with $\mathbb{E}[\mathrm{Corr}(g;Z)\mid X]=\mathbb{E}[\langle\zeta(W),(\eta_{0}-\eta)(W)\rangle\mid X]$ will satisfy Definition 3.1. We think of $\chi(g;z):=m(\eta;z)-\mathrm{Corr}(g;z)$ as representing a generalized pseudo-outcome that, on average, looks like the desired heterogeneous causal effect. In fact, any loss of the above form is equivalent to a loss that looks like $\ell(\nu,g;z)=\frac{1}{2}\left(\nu-\chi(g;z)\right)^{2}$ .

All universally orthogonal losses we have seen up until this point satisfy can be written in terms of pseudo-outcomes in the canonical form above. We note the particular settings of $\eta_{0},\zeta_{0},m,$ and $\mathrm{Corr}$ for each of these losses in Table 1. Typically, we assume the statistician will estimate $\eta_{0}$ and $\zeta_{0}$ by plugging in estimates for each unknown constituent function, e.g. by plugging in estimates for $\pi_{0}$ and $\mu_{0}$ in the CATE example.

Loss	$\eta_{0}$	$\zeta_{0}$	$m(\eta;z)$	$\mathrm{Corr}(g;z)$
$\ell_{\mathrm{CATE}}$	$\mu_{0}(a,x)$	$\left(\frac{a}{\pi_{0}(x)}-\frac{1-a}{1-\pi_{0}(x)}\right)$	$\eta(1,x)-\eta(0,x)$	$\zeta(a,x)\cdot(y-\eta(a,x))$
$\ell_{\mathrm{ACD}}$	$\mu_{0}(a,x)$	$\frac{\partial_{a}\pi_{0}(a\mid x)}{\pi_{0}(a\mid x)}$	$\partial_{a}\mu(a,x)$	$\zeta(a,x)\cdot(y-\eta(a,x))$
$\ell_{\mathrm{LATE}}$	$\frac{p_{0}(x)}{q_{0}(x)}$	$\frac{a(d-\pi_{0}(x))}{q_{0}(x)}$	$\eta(x)$	$\zeta(a,d,x)\cdot\left(\frac{y(d-\pi_{0}(x))}{a(d-\pi_{0}(x))}-\eta(x)\right)$

Table 1: Canonical representations of universally orthogonal losses in terms of base nuisances

\eta_{0}

, Riesz representers

\zeta_{0}

, and pseudo-outcome components

m(\eta;z)

and

\mathrm{Corr}(g;z)

Universal orthogonality allows to bound $\mathrm{Cal}(\theta,g_{0})$ above by two decoupled terms. The first, denoted $\mathrm{err}(g,g_{0};\theta)$ , intuitively measures the distance between some fixed nuisance estimate $g\in\mathcal{G}$ and the unknown, true nuisance parameters $g_{0}\in\mathcal{G}$ . This error is second order in nature, generally depending on the squared norm of the nuisance estimation error. The second term is $\mathrm{Cal}(\theta,g)$ , and represents the calibration error of $\theta$ in a reality where the learned nuisances $g$ were actually the true nuisances. As mentioned in the introduction, we view our bound as a “change of nuisance” result (akin to change of measure), allowing the learner to pay an upfront price for nuisance misestimation and then subsequently reason about the calibration error under potentially incorrect, learned nuisances. We prove the following in Appendix A.

Theorem 3.3.

Let $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ be universally orthogonal, per Definition 3.1. Let $g_{0}\in\mathcal{G}$ denote the true nuisance parameters associated with $\ell$ . Suppose $D^{2}_{g}\mathbb{E}[\partial\ell(\theta,f;Z)\mid X](g-g_{0},g-g_{0})$ exists for any $g,f\in\mathcal{G}$ . Then, for any $g\in\mathcal{G}$ and $\theta:\mathcal{X}\rightarrow\mathbb{R}$ , we have

\mathrm{Cal}(\theta,g_{0})\leq\frac{1}{2}\mathrm{err}(g,g_{0};\theta)+\mathrm{Cal}(\theta,g),

where $\mathrm{err}(g,h;\theta):=\sup_{f\in[g,h]}\sqrt{\mathbb{E}\left(\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,f;Z)\mid X\right](h-g,h-g)\right\}^{2}\right)}$ .⁶⁶6For $f,h\in\mathcal{G}$ , we let the interval $[f,h]:=\{\lambda(w)f(w)+(1-\lambda(w))h(w):\lambda(w)\in[0,1]\}$ .

While the expression defining $\mathrm{err}(g,g_{0};\theta)$ looks unpalatable, when $g_{0}=(\eta_{0},\zeta_{0})$ (as in Example 3.2), this term can quite generally be bounded above by the cross-error in nuisance estimation, i.e. the quantity $\|\langle\eta-\eta_{0},\zeta-\zeta_{0}\rangle\|_{L^{2}(P_{W})}$ . The formal statement of this fact is presented in Proposition 3.4 below. More broadly, if the conditional Hessian satisfies $|D_{g}^{2}\mathbb{E}[\partial\ell(\theta,f;Z)\mid X](\Delta,\Delta)|\leq C\|\Delta\|_{L^{2}(P_{W})}$ for any $\Delta\in\mathcal{G}-\mathcal{G}$ and $f\in\mathcal{G}$ , then we simply have $\mathrm{err}(g,g_{0};\theta)\leq C\|g-g_{0}\|^{2}_{L^{2}(P_{W})}$ . These sorts of second order bound appears in other works on causal estimation [18, 58, 40, 6].

We view the above result below as a major generalization of the main result (Theorem 1) of van der Laan et al. [58], which shows a similar bound for measuring the calibration error of estimates of conditional average treatment effects when calibration is performed according to isotonic regression. Our result, which more explicitly leverages the concept of Neyman orthogonality, can be used to recover that of van der Laan et al. [58] as a special case, including the error rate in nuisance estimation.

Proposition 3.4.

Suppose the loss $\ell$ satisfies the score condition outlined in Equation (5) and suppose $m(\eta;z)$ is linear in $\eta$ . Then, we have

\mathrm{err}(g,g_{0};\theta)\leq 2\|\langle\eta-\eta_{0},\zeta-\zeta_{0}\rangle\|_{L^{2}(P_{W})},

where $g_{0}=(\eta_{0},\zeta_{0})$ represent the true, unknown nuisance parameters, and $g=(\eta,\zeta)$ represent arbitrary, fixed nuisance estimates. If instead one has $|D_{g}^{2}\mathbb{E}[\partial\ell(\theta,f;Z)(\Delta,\Delta)|\leq C\|\Delta\|_{L^{2}(P_{W})}^{2}$ for any $\Delta$ and $\theta$ , the one has

\mathrm{err}(g,g_{0};\theta)\leq C\|g-g_{0}\|_{L^{2}(P_{W})}^{2}.

3.1 Sample Splitting Algorithm

Throughout the remainder of this section, we make the following assumption on the the loss $\ell$ . Any loss of the form presented in Equation (5) naturally satisfies the following assumption, and thus our example is applicable to $\ell_{\mathrm{CATE}},\ell_{\mathrm{ACD}},$ and $\ell_{\mathrm{LATE}}$ described earlier.

Assumption 1 (Linear Score).

We assume there is some function $\chi:\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ such that the loss $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ satisfies

\mathbb{E}[\partial\ell(\theta,g;Z)\mid X]=\theta(X)-\mathbb{E}[\chi(g;Z)\mid X]

for any $g\in\mathcal{G}$ and $\theta:\mathcal{X}\rightarrow\mathbb{R}$ .

The largely theoretic bound presented in Theorem 3.3 suggests a natural algorithm for performing causal calibration. First, a learner should use some fraction (say half) of the data to produce nuisance estimate $\widehat{g}$ using any black-box algorithm. Then, the learner should transform second half of the data into generalized pseudo-outcomes $\chi(\widehat{g};Z)$ using the learned nuisances. Finally, they should apply some off-the-shelf calibration algorithm to the transformed data points. We formalize this in Algorithm 1.

Algorithm 1 Sample-Splitting for Universally Orthogonal Losses

1:Samples

Z_{1},\dots,Z_{2n}\sim P_{Z}

, nuisance alg.

\mathcal{A}_{1}

, calibration alg.

\mathcal{A}_{2}

, estimator

\theta

2:Estimate nuisances:

\widehat{g}\leftarrow\mathcal{A}_{1}(Z_{1:n})

3:Compute pseudo-outcomes:

\chi_{n+1},\dots,\chi_{2n}:=\chi(\widehat{g};Z_{n+1}),\dots,\chi(\widehat{g};Z_{2n})

4:Run calibration:

\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,(X_{m},\chi_{m})_{m=n+1}^{2n})

5:Calibrated estimator

\widehat{\theta}

Algorithm 1 generalizes the main algorithm in van der Laan et al. [58] (Algorithm 1) to allow for the calibration of an estimate of any heterogeneous causal effect using any off-the-shelf nuisance estimation and calibration algorithms. To make efficient use of the data, we recommend instead running the cross calibration procedure outlined in Algorithm 2. We do not provide a convergence analysis for this algorithm.

Algorithm 2 Cross Calibration for Universally Orthogonal Losses

1:Samples

\mathcal{D}=Z_{1},\dots,Z_{n}\sim P_{Z}

, nuisance alg.

\mathcal{A}_{1}

, calibration alg.

\mathcal{A}_{2}

, estimator

\theta

2:Split

\mathcal{I}:=[n]

into

K

equally-sized folds

\mathcal{I}_{1},\dots,\mathcal{I}_{K}

3:Define

\mathcal{D}_{k}:=\{Z_{i}:i\in\mathcal{I}_{k}\}

for all

k\in[K]

4:for

k\in 1,\dots,K

5: Estimate nuisances:

\widehat{g}^{(-k)}\leftarrow\mathcal{A}_{1}(\mathcal{D}\setminus\mathcal{D}_{k})

6: Compute pseudo-outcomes

\chi_{i}:=\chi(\widehat{g}^{(-k)};Z_{i})

for

i\in\mathcal{I}_{k}

7:Run calibration:

\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,(X_{m},\chi_{m})_{m=1}^{n})

8:Calibrated estimator

\widehat{\theta}

We now prove convergence guarantees for Algorithm 1. We start by enumerating several assumptions needed to guarantee convergence.

Assumption 2.

Let $\mathcal{A}_{1}:\mathcal{Z}^{\ast}\rightarrow\mathcal{G}$ be a nuisance estimation algorithm taking in an arbitrary number of points, and let $\mathcal{A}_{2}:\Theta\times(\mathcal{X}\times\mathbb{R})^{\ast}\rightarrow\Theta$ be a calibration algorithm taking some initial estimator and an arbitrary number of covariate/label pairs. We assume

1.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $Z_{1},\dots,Z_{n}\sim P$ i.i.d., and failure probability $\delta_{1}\in(0,1)$ , we have

$\mathrm{err}(\widehat{g},g_{0};\theta)\leq\mathrm{rate}_{1}(n,\delta_{1};P_{Z}),$

where $\widehat{g}\leftarrow\mathcal{A}_{1}(Z_{1:n})$ and $\mathrm{rate}_{1}$ is some rate function.
2.

For any distribution $Q$ on $\mathcal{X}\times\mathbb{R}$ , $(X_{1},Y_{1}),\dots,(X_{n},Y_{n})\sim Q$ i.i.d., initial estimator $\theta:\mathcal{X}\rightarrow\mathbb{R}$ , and failure probability $\delta_{2}\in(0,1)$ , we have

$\mathrm{Cal}(\widehat{\theta};Q)\leq\mathrm{rate}_{2}(n,\delta_{2};Q),$

where $\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,\{(X_{i},Y_{i})\}_{i=1}^{n})$ and $\mathrm{rate}_{2}$ is some rate function.

We briefly parse the above assumptions. For the first assumption, whenever $g_{0}=(\eta_{0},\zeta_{0})$ and $\mathrm{err}(\widehat{g},g_{0};\theta)\lesssim\|(\eta-\eta_{0})(\zeta-\zeta_{0})\|_{L^{2}(P_{W})}$ or $\mathrm{err}(\widehat{g},g_{0};\theta)\lesssim\|\widehat{g}-g_{0}\|_{L^{2}(P_{W})}^{2}$ , one can directly apply ML, non-parametric, or semi-parametric methods to estimate the unknown nuisances. For instance, if the nuisances are assumed to assumed to satisfy Holder continuity assumptions or are assumed to belong to a ball in a reproducing kernel Hilbert space, one can apply classical kernel smoothing methods or kernel ridge regression respectively to estimate the unknown parameters [58, 57, 60] to obtain optimal rates. Further, many well-known calibration algorithms satisfy the second assumption, often in a manner that doesn’t depend on the underlying distribution $Q$ . For instance, results in Gupta and Ramdas [25] on $L^{\infty}$ calibration error bounds directly imply that if $\mathcal{A}_{2}$ is taken to be uniform mass/histogram binning, then the rate function $\mathrm{rate}_{2}$ can be taken as $\mathrm{rate}_{2}(n,\delta;Q)=O\left(\|Y\|_{\infty}\sqrt{\frac{B\log(B/\delta)}{n}}\right)$ , where $B$ denotes the number of bins/buckets, $\|Y\|_{\infty}$ denotes the essential supremum of $Y$ (which will be finite so long as nuisances and observations are bounded) and the unknown constant is independent of $n$ , $Q$ , and $\delta$ . Likewise, the convergence in probability results for isotonic calibration proven by van der Laan et al. [58] can naturally be extended to high-probability guarantees using standard concentration of measure results.

Theorem 3.5.

Suppose $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ is an arbitrary, universally orthogonal loss satisfying Assumption 1. Let $\mathcal{A}_{1},\mathcal{A}_{2},$ and $\chi$ satisfy Assumption 2. Then, with probability at least $1-\delta_{1}-\delta_{2}$ , the output $\widehat{\theta}$ of Algorithm 1 run on a calibration dataset of $2n$ i.i.d. data points $Z_{1},\dots,Z_{2n}\sim P$ satisfies

\mathrm{Cal}(\widehat{\theta},g_{0})\leq\frac{1}{2}\mathrm{rate}_{1}(n,\delta_{1};P_{Z})+\mathrm{rate}_{2}(n,\delta_{2};P^{\chi}_{\widehat{g}}),

where $P^{\chi}_{\widehat{g}}$ denotes the (random) distribution of $(X,\chi(\widehat{g};Z))$ where $Z\sim P_{Z}$ and again $X\subset Z$ .

We prove the above theorem in Appendix B. The above result can be thought of as an analogue of Theorem 1 of Foster and Syrgkanis [18], which shows a similar bound on excess parameter risk, and also a generalization of Theorem 1 of van der Laan et al. [58], which shows a similar bound when isotonic regression is used to calibrate CATE estimates.

We note that while we state our bounds and assumptions in terms of high-probability guarantees, we could have equivalently assumed, $\mathrm{err}(\widehat{g},g_{0};\theta)=O_{\mathbb{P}}(\mathrm{rate}_{1}(n;P_{Z}))$ and $\mathrm{Cal}(\widehat{\theta};Q)=O_{\mathbb{P}}(\mathrm{rate}_{2}(n))$ for appropriately chosen rate functions $\mathrm{rate}_{1}$ and $\mathrm{rate}_{2}$ ⁷⁷7Here, $\mathrm{rate}_{2}(n)$ would have to be a function satisfying independent of the distribution of pseudo-outcomes to obtain convergence in probability guarantees. Such functions exists for isotonic calibration and histogram binning. This, for instance, would be useful if one wanted to apply the results on the convergence of isotonic regression due to van der Laan et al. [58], who show $\mathrm{Cal}(\theta;P)=O_{\mathbb{P}}\left(n^{-1/3}\right)$ .

Lastly, we note that it is desirable for calibration algorithms to possess a “do no harm” guarantee, which ensures that the risk of the calibrated parameter $\widehat{\theta}$ is not much larger than the risk of original parameter. We present such a guarantee in Theorem C.1 in Appendix C, which follows using standard risk bounding techniques due to Foster and Syrgkanis [18].

4 Calibration for Conditionally Orthogonal Losses

We now consider the more challenging problem where the causal effect $\theta_{0}(X)$ is not the minimizer of a universally orthogonal loss. To aid in our exposition, we introduce calibration functions. In short, the calibration function gives a canonical choice of a post-processing $\widehat{\theta}=\tau\circ\theta:\mathcal{X}\rightarrow\mathbb{R}$ such that $\mathrm{Cal}(\widehat{\theta};g_{0})=0$ . While computing the calibration function exactly requires knowledge of the data-generating distribution, it can be approximated in finitely-many samples.

Definition 4.1 (Calibration Function).

Given any $\theta:\mathcal{X}\rightarrow\mathbb{R}$ and $g\in\mathcal{G}$ , we define the calibration function for $\theta$ at $g$ as the mapping $\gamma_{\theta}(\cdot;g):\mathcal{X}\rightarrow\mathbb{R}$ given by

\gamma_{\theta}(x;g):=\arg\min_{\nu}\mathbb{E}[\ell(\nu,g;Z)\mid\theta(X)=\theta(x)].

In particular, when $g=g_{0}$ , we call $\gamma_{\theta}^{\ast}:=\gamma_{\theta}(\cdot;g_{0})$ the true calibration function.

As hinted, first-order optimality conditions alongside the tower rule for conditional expectations imply that $\mathbb{E}[\partial\ell(\gamma_{\theta}(\cdot;g),g;Z)\mid\gamma_{\theta}(X;g)]=0$ for any $g\in\mathcal{G}$ . This, in particular, implies that $\gamma_{\theta}^{\ast}$ is perfectly calibrated. As an example, when a loss satisfies Assumption 1, $\gamma_{\theta}^{\ast}(x,g)=\mathbb{E}[\chi(g;Z)\mid\theta(X)=\theta(x)]$ .

We now ask under what general assumptions on $\ell$ can we achieve calibration. In analogy with Foster and Syrgkanis [18], who consider the general task of empirical risk minimization in the presence of nuisance, the hope would be to weaken the assumption of universal orthogonality to that of Neyman orthogonality. Two commonly-used definitions for Neyman orthogonality (a marginal version and a version conditional on covariates) are provided below.

Definition 4.2 (Neyman Orthogonality).

We say $\ell$ is Neyman orthogonal conditional on covariates $X$ (or marginally) if, for $g\in\mathcal{G}$ , we have

D_{g}\mathbb{E}[\partial\ell(\theta_{0},g_{0};Z)\mid X](g-g_{0})=0\qquad(\text{respectively, }D_{g}\mathbb{E}[\partial\ell(\omega_{0},h_{0};Z)](g-h_{0})=0),

where $\theta_{0}(x):=\arg\min_{\nu}\mathbb{E}[\ell(\nu,g_{0};Z)\mid X=x]$ and $g_{0}$ denotes the true nuisances (respectively $\omega_{0}:=\arg\min_{\nu}\mathbb{E}[\ell(\nu,h_{0};Z)]$ and $h_{0}$ for the latter).

Neyman orthogonality is useful for a task such as risk minimization because it allows the statistician to relate the risk $\mathbb{E}[\ell(\widehat{\theta},g_{0};Z)]$ under the true nuisances to the risk under the computed nuisances $\mathbb{E}[\ell(\widehat{\theta},\widehat{g};Z)]$ up to second order errors. Why do we need two separate conditions on the loss $\ell$ ? In general, the conditions in Definition 4.2 are not equivalent. To illustrate this, we can look at the example of the conditional/marginal quantile under treatment (We will let $\theta_{\mathrm{QUT}}(X)$ denote the former and $\omega_{\mathrm{QUT}}$ the latter). Recalling that we defined the pinball loss $\widetilde{\ell}_{\mathrm{QUT}}(\nu,p;z):=ap(x)(y-\nu)(Q-\mathbbm{1}_{y\leq\nu})$ , it is not hard to see that we have

\theta_{\mathrm{QUT}}(x)=\arg\min_{\nu}\mathbb{E}[\widetilde{\ell}_{\mathrm{QUT}}(\nu,p_{0};Z)\mid X=x]\quad\text{and}\quad\omega_{\mathrm{QUT}}=\arg\min_{\nu}\mathbb{E}[\widetilde{\ell}_{\mathrm{QUT}}(\nu,p_{0};Z)],

where $p_{0}(x):=\mathbb{P}(A=1\mid X=x)^{-1}$ denotes the inverse propensity. Straightforward computation yields that $\widetilde{\ell}_{\mathrm{QUT}}$ is Neyman orthogonal conditional on covariates, but not marginally orthogonal. However, as noted in Kallus et al. [33], one can define the more complicated loss $\ell_{\mathrm{QUT}}(\nu,(p,f);z)$ by performing a first order correction:

\ell_{\mathrm{QUT}}(\nu,(p,f);z):=\widetilde{\ell}_{\mathrm{QUT}}(\nu,p;z)-\theta(x)\cdot\underbrace{(ap(x)(f(x)-Q)-f(x)+Q)}_{=:\mathrm{Corr}((p,f);z)}

(6)

where $f_{0}(x):=\mathbb{P}(Y(1)\leq\omega_{\mathrm{QUT}}\mid X=x)$ is an additional nuisance that must be estimated from the data. One can check that $\ell_{\mathrm{QUT}}$ satisfies the second condition of Definition 4.2, i.e. marginal Neyman orthogonality.

In calibration, we care about the quality of a model conditional on its own predictions. More specifically, given any initial model $\theta(X)$ , the goal of any calibration algorithm (e.g. histogram binning, isotonic regression) is to compute the calibration function $\gamma_{\theta}^{\ast}$ (Definition 4.1).⁸⁸8We remind the reader that $\gamma_{\theta}^{\ast}(x):=\arg\min_{\nu}\mathbb{E}[\ell(\nu,g_{0};w)\mid\theta(X)=\theta(x)]$ If $\theta(X)$ were a constant function, then we would have $\gamma_{\theta}^{\ast}(x)\equiv\omega_{0}$ , and thus we would want to leverage a loss satisfying marginal orthogonality. Likewise, if $\theta(X)$ were roughly of the same complexity as $\theta_{0}(X)$ , we may want to leverage the a loss $\ell$ satisfying the form of Neyman orthogonality conditional on covariates $X$ . In general, the complexity of the initial estimate $\theta(X)$ will interpolate between these two extremes.

For a variant of Neyman orthogonality to thus be useful, we would need to cross-derivative to vanish (a) when evaluated at the calibration function $\gamma_{\theta}^{\ast}$ instead of $\theta_{0}(X)$ or $\omega_{0}$ and (b) when the expectation is taken conditionally on the prediction $\theta(X)$ instead of either conditionally on $X$ or marginally. The extra structure provided by universal orthogonality allowed us to side-step this issue as the cross-derivative of the loss vanished when evaluated at any estimate of $\theta_{0}$ so long as nuisances were estimated correctly. The following, quite technical condition will give us the structure we need to perform calibration of estimates of more general heterogeneous causal effects.

Definition 4.3 (Conditional Orthogonality).

Suppose $\widetilde{\ell}(\theta,\eta;z)$ is some initial loss with true nuisance $\eta_{0}$ . Define the “corrected” loss $\ell$ by

\ell(\theta,(\eta,\zeta);z):=\widetilde{\ell}(\theta,\eta;z)-\theta(x)\cdot\mathrm{Corr}((\eta,\zeta);z),

where $\mathrm{Corr}((\eta,\zeta);z)$ is any correction term satisfying $\mathbb{E}[\mathrm{Corr}((\eta_{0},\zeta);z)\mid X]=0$ . Then, we say $\ell$ is conditionally orthogonal if, for any $\varphi:\mathcal{X}\rightarrow\mathbb{R}$ , there exists $\zeta_{\varphi}\in L^{2}(P_{X})$ such that

D_{g}\mathbb{E}[\partial\ell(\gamma_{\varphi}^{\ast},g_{\varphi};Z)\mid\varphi(X)](g-g_{0})=0,

for all $g\in\mathcal{G}$ , where $g_{\varphi}:=(\eta_{0},\zeta_{\varphi})$ .

Definition 4.3 may be difficult to parse, but we can work through several examples to gain some intuition. Returning to the example of conditional quantile under treatment and the (corrected) pinball loss $\ell_{\mathrm{QUT}}(\nu,(p,f);z)$ defined above, we simply took the correction term to be $\mathrm{Corr}((p,f);z):=ap(x)(f(x)-Q)-f(x)+Q$ . From a straightforward calculation, one can check $\ell_{\mathrm{QUT}}$ satisfies Definition 4.3 with additional nuisance $f_{\varphi}$ given by $f_{\varphi}(x):=\mathbb{P}(Y(1)\leq\gamma_{\varphi}^{\ast}(X)\mid X=x)$ .

More broadly, given some initial loss $\widetilde{\ell}$ , one can use the Riesz representation theorem to obtain $\zeta_{\varphi}:\mathcal{W}\rightarrow\mathbb{R}$ satisfying

D_{\eta}\mathbb{E}[\partial\widetilde{\ell}(\gamma_{\varphi}^{\ast},\eta_{0};Z)\mid\varphi(X)](\eta-\eta_{0})=\mathbb{E}[\langle\zeta_{\varphi}(W),(\eta-\eta_{0})(W)\rangle\mid\varphi(X)]

almost surely. Then, if we can find some variable $U\subset Z$ such that $\mathbb{E}[U\mid W]=\eta_{0}(W)$ , we can simply take $\mathrm{Corr}((\eta,\zeta);z):=\langle\zeta(w),\eta(w)-u\rangle$ , which gives us a “corrected” loss

\ell(\theta,(\eta,\zeta);z):=\widetilde{\ell}(\theta,\eta;z)-\theta(x)\cdot\langle\zeta(w),\eta(w)-u\rangle.

We conclude by pointing out the following observations about calibration with respect to such losses $\ell$ , which follows immediately from Definition 4.3.

Corollary 4.4.

Suppose a loss $\ell$ satisfies Definition 4.3. Then, the following hold:

1.

The calibration function

$\gamma_{\varphi}(x;(\eta,\zeta)):=\arg\min_{\nu}\mathbb{E}[\ell(\nu,(\eta,\zeta);Z)\mid\varphi(X)=\varphi(x)]$

doesn’t depend on $\zeta$ when $\eta=\eta_{0}$ . Thus, we can write $\gamma_{\varphi}^{\ast}(x)=\gamma_{\varphi}(x;(\eta_{0},\zeta))$ for any $\zeta$ .

The calibration error of any estimate $\theta:\mathcal{X}\rightarrow\mathbb{R}$ , given by

\mathrm{Cal}(\theta,(\eta,\zeta)):=\left(\int_{\mathcal{X}}\mathbb{E}[\partial\ell(\theta,(\eta,\zeta);Z)\mid\theta(X)]^{2}P_{X}(dx)\right)^{1/2}

also does not depend on $\zeta$ when $\eta=\eta_{0}$ . Thus, we can write $\mathrm{Cal}(\theta,\eta_{0})$ in place of $\mathrm{Cal}(\theta,(\eta_{0},\zeta))$ for any $\zeta$ .

3.

Lastly, not only do $\ell$ and $\widetilde{\ell}$ posses the same conditional minimizer when evaluated at $\eta_{0}$ (regardless of the choice of $\zeta$ ), but we also have

$\mathrm{Cal}(\theta,(\eta_{0},\zeta))=\widetilde{\mathrm{Cal}}(\theta,\eta_{0}),$

where $\theta:\mathcal{X}\rightarrow\mathbb{R}$ is arbitrary and $\widetilde{\mathrm{Cal}}$ denotes the calibration error under $\widetilde{\ell}$ .

4.1 A General Bound on Calibration Error

We now prove a decoupled bound on the calibration error $\mathrm{Cal}(\theta,g_{0})$ under the assumption that $\ell$ is conditionally orthogonal. This bound serves as a direct analogue of the one presented in Theorem 3.3, just in a more general setting. As we will see, bounding the calibration error of losses that are not universally orthogonalizable is a much more delicate task.

To prove our result, we will need place convexity assumptions on the underlying loss $\ell$ . We note that these convexity results are akin to those made in existing works, namely in the work of Foster and Syrgkanis [18].

Assumption 3.

We assume that the loss function conditioned on covariates $X$ is $\alpha$ -strongly convex, i.e. for any $v\in\mathbb{R}$ and any $g\in\mathcal{G}$ , we have

\mathbb{E}\left[\partial^{2}\ell(v,g;Z)\mid X\right]\geq\alpha.

Assumption 4.

We assume that the loss function conditioned on covariates $X$ is $\beta$ -smooth, i.e. for any $v\in\mathbb{R}$ and any $g\in\mathcal{G}$ , we have

\mathbb{E}\left[\partial^{2}\ell(v,g;Z)\mid X\right]\leq\beta.

We now state the main theorem of this section. The bound below appears largely identical to the one presented in Theorem 3.3 modulo two minor differences. First, we pay a multiplicative factor of $\beta/\alpha$ in both of the decoupled terms, which ultimately is just the condition number of the loss $\ell$ . Second, the error term $\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast})$ is evaluated at the calibration function $\gamma_{\theta}^{\ast}(x):=\arg\min_{\nu}\mathbb{E}[\ell(\nu,g_{0};Z)\mid\theta(X)=\theta(x)]$ instead of the parameter estimate $\theta$ . This difference is due to the fact that, in the proof of Theorem 4.5, we must perform a functional Taylor expansion around $\gamma_{\theta}^{\ast}$ in order to invoke the orthogonality condition. This subtlety was absent in the case of universal orthogonality, as $\ell$ was insensitive to nuisance misestimation for any parameter estimate $\theta$ . We ultimately view this difference as minor, as for many examples (e.g. the pinball loss $\ell_{\mathrm{QUT}}$ ) the dependence on $\gamma_{\theta}^{\ast}$ vanishes. We prove Theorem 4.5 in Appendix A.2.

Theorem 4.5.

Let $\ell$ be a conditionally orthogonal loss (Definition 4.3) that is $\alpha$ -strong convex (Assumption 3) and $\beta$ -smooth (Assumption 4). Suppose $D_{g}^{2}\mathbb{E}[\partial\ell(\gamma_{\theta}^{\ast},f;Z)\mid X](g-g_{0},g-g_{0})$ exists for $f,g\in\mathcal{G}$ . Then, for any estimate $\theta:\mathcal{X}\rightarrow\mathbb{R}$ and nuisance parameter $g=(\eta,\zeta)$ , we have

\mathrm{Cal}(\theta,\eta_{0})\leq\frac{\beta}{2\alpha}\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast})+\frac{\beta}{\alpha}\mathrm{Cal}(\theta,g),

where $g_{\theta}=(\eta_{0},\zeta_{\theta})$ are the true, unknown nuisance functions, $\gamma_{\theta}^{\ast}$ is the calibration function associated with $\theta$ , and $\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast})$ is as defined in Theorem 3.3.

Although the bound in Theorem 4.5 looks similar in spirit to the one presented in Theorem 3.3, there still remain questions to answer. For instance, what does the condition number $\beta/\alpha$ look like for practically-relevant losses? Likewise, will the error term $\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast})$ simplify into a cross-error term as in the case of universally orthogonalizable losses? We interpret Theorem 4.5 by spending some time looking at the example of the pinball loss $\ell_{\mathrm{QUT}}$ .

Example 4.6.

First, for any fixed quantile $Q$ , we assess the strong convexity/smoothness properties of $\ell_{\mathrm{QUT}}$ . Let $p:\mathcal{X}\rightarrow\mathbb{R}_{\geq 0}$ represent any inverse-propensity estimate, and let $\pi_{0}$ represent the true propensity score. Assume $Y(1)$ admits a conditional density $f_{Y(1)}(y\mid x)$ with respect to the Lebesgue measure on $\mathbb{R}$ . Straightforward calculation yields

\partial^{2}\mathbb{E}[\ell_{\mathrm{QUT}}(\theta,p;Z)\mid X]=f_{Y(1)}(\theta(x)\mid x).

Thus, if $l\leq f_{Y(1)}(y\mid x)\leq u$ for all $y\in\mathbb{R},x\in\mathcal{X}$ and $\epsilon<p_{0}(x)^{-1},p(x)^{-1}\leq 1-\epsilon$ for all $x\in\mathcal{X}$ for some $0<\epsilon<1/2$ , then we have:

\frac{\epsilon^{2}}{(1-\epsilon)^{2}}l\leq\partial^{2}\mathbb{E}[\ell_{\mathrm{QUT}}(\theta,p;Z)\mid X]\leq\frac{(1-\epsilon)^{2}}{\epsilon^{2}}u,

i.e. that $\ell_{\mathrm{QUT}}$ satisfies Assumption 4 with $\beta=\frac{1-\epsilon}{\epsilon}u$ and Assumption 3 with $\alpha=\frac{\epsilon}{1-\epsilon}l$ .

We can further interpret the error term $\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast})$ in the case of the pinball loss, where again $g=(\eta,f)$ . In particular, straightforward calculation yields

\mathbb{E}[\partial\ell_{\mathrm{QUT}}(\theta,g;Z)\mid X]=\frac{p(X)}{p_{0}(X)}(\mathbb{P}(Y(1)\leq\theta(X)\mid X)-Q)-\mathbb{E}\left[A(f(X)-Q)(p(W)-p_{0}(W))\mid X\right].

As the first term is linear in the nuisance estimate $p$ and doesn’t depend on $f$ , its second Gateaux derivative (with respect to $g$ ) is identically zero. Thus, the error term does not depend on $\gamma_{\theta}^{\ast}$ , and thus we can write $\mathrm{err}(g,g_{\theta})$ instead. Further, using the same analysis as in Proposition 3.4 and writing $\zeta(a,x)=a(f(x)-Q)$ . we have that

	$\displaystyle\frac{1}{2}\mathrm{err}(g,g_{\theta})$	$\displaystyle\leq\\|(p-p_{0})(\zeta-\zeta_{\theta})\\|_{L^{2}(P_{W})}$
		$\displaystyle=\left\\|A(f(X)-f_{\theta}(X))(p(X)-p_{0}(X))\right\\|_{L^{2}(P_{W})},$

where we recall that $f_{\theta}(x):=\mathbb{P}(Y(1)\leq\gamma_{\theta}^{\ast}\mid X=x)$ . Thus, even in the general case of conditional orthogonality, we can often obtain simple looking bounds on the error in nuisance estimation.

4.2 A Sample Splitting Algorithm

We conclude the section by presenting two algorithms for performing causal calibration with respect to conditionally orthogonal losses. As in Section 3, we first present a sample splitting algorithm (Algorithm 3) that enjoys finite sample convergence guarantees. We then present a corresponding cross-calibration algorithm that is likely more useful in practice.

Algorithm 3 Sample-Splitting for Conditionally Orthogonal Losses

1:Samples

Z_{1},\dots,Z_{2n}\sim P_{Z}

, nuisance alg.

\mathcal{A}_{1}

, general loss calibration alg.

\mathcal{A}_{2}

, estimator

\theta

, loss involving nuisance

\ell

2:Estimate nuisances:

\widehat{g}\leftarrow\mathcal{A}_{1}(Z_{1:n})

3:Compute loss partial-evaluations:

\ell_{n+1},\dots,\ell_{2n}:=\ell(\cdot,\widehat{g};Z_{n+1}),\dots,\ell(\cdot,\widehat{g};Z_{2n})

4:Run calibration:

\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,(X_{m},\ell_{m})_{m=n+1}^{2n})

5:Calibrated estimator

\widehat{\theta}

Algorithm 3 is essentially a generalization of Algorithm 1 to general losses. The key difference is that we can no longer compute pseudo-outcomes for general losses. Instead, we assume that the calibration algorithm $\mathcal{A}_{2}$ passed to Algorithm 1 can calibrate “with respect to general losses $\ell$ ”. What does this mean? Many calibration algorithms such as linear calibration, Platt scaling, and isotonic calibration, compute a mapping $\widehat{\tau}$ satisfying

\widehat{\tau}\in\arg\min_{\tau\in\mathcal{F}}\sum_{m=1}^{n}\ell\left((\tau\circ\theta)(X_{i}),Y_{i}\right)

(7)

where $(X_{1},Y_{1}),\dots,(X_{n},Y_{n})$ denotes a calibration dataset, $\mathcal{F}\subset\{f:\mathbb{R}\rightarrow\mathbb{R}\}$ is some appropriately-defined class of functions, and $\ell$ is an appropriately chosen class of functions. Table 2 below outlines the choices of $\mathcal{F}$ and $\ell$ for common calibration algorithms.

Algorithm	Loss $\ell$	Function class $\mathcal{F}$
Isotonic Calibration	$\ell(\nu,y)=\frac{1}{2}(\nu-y)^{2}$	$\mathcal{F}=\{\tau(x)\text{ is non-decreasing}\}$
Linear Calibration	$\ell(\nu,y)=\frac{1}{2}(\nu-y)^{2}$	$\mathcal{F}=\{\tau(x)=ax+b,\;a,b\in\mathbb{R}\}$
Histogram binning	$\ell(\nu,y)=\frac{1}{2}(\nu-y)^{2}$	$\mathcal{F}=\{\tau(x)=\sum c_{b}\mathbbm{1}_{\theta(x)\in[\theta(X)_{(b-1)n/B)},\theta(X)_{(bn/B)}}\}$
Platt Scaling	$\ell(\nu,y)=-y\log(\nu)-(1-y)\log(1-\nu)$	$\mathcal{F}=\{\tau(x)=\frac{1}{1+\exp(ax+b)},a,b\in\mathbb{R}\}$

Table 2: Representations of classical calibration algorithms in the form outlined in Equation (7).

Thus, for general losses involving nuisance $\ell$ , it makes sense that the calibration algorithm $\mathcal{A}_{2}$ should compute the following minimizer. We outline a general template for $\mathcal{A}_{2}$ in Algorithm 5 in Appendix D.

\widehat{\tau}\in\arg\min_{\tau\in\mathcal{F}}\sum_{m=1}^{n}\underbrace{\ell((\tau\circ\theta)(X_{i}),\widehat{g}_{i};Z_{i})}_{=:\ell_{i}(\theta(X_{i}))},

In the above, $Z_{1},\dots,Z_{n}\sim P_{Z}$ is now the calibration sample and $\widehat{g}_{i}$ denotes a nuisance estimate which generically may depend on the current sample $i$ . Also in Appendix D, we also prove the convergence of a simple, three-way sample splitting algorithm based on uniform mass binning. and actually prove a finite sample convergence $L^{2}$ calibration error convergence guarantee for uniform mass binning as well. We do not include this algorithm in the main paper as the focus of the work is on presenting a general framework for performing causal calibration.

We can similarly define a version of cross calibration for conditionally orthogonal losses, which we present in Algorithm 4.

Algorithm 4 Cross Calibration for Conditionally Orthogonal Losses

1:Samples

\mathcal{D}:=Z_{1},\dots,Z_{n}\sim P_{Z}

, nuisance alg.

\mathcal{A}_{1}

, general loss calibration alg.

\mathcal{A}_{2}

, estimator

\theta

, loss involving nuisance

\ell

2:Split

\mathcal{I}:=[n]

into

K

equally-sized folds

\mathcal{I}_{1},\dots,\mathcal{I}_{K}

3:Define

\mathcal{D}_{k}:=\{Z_{i}:i\in\mathcal{I}_{k}\}

for all

k\in[K]

4:for

k\in 1,\dots,K

5: Estimate nuisances:

\widehat{g}^{(-k)}\leftarrow\mathcal{A}_{1}(\theta,\mathcal{D}\setminus\mathcal{D}_{k})

6: Compute loss partial evaluations:

\ell_{i}:=\ell(\cdot,\widehat{g}^{(-k)};Z_{i})

for

i\in\mathcal{I}_{k}

7:Run calibration:

\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,(X_{m},\ell_{m})_{m=1}^{n})

8:Calibrated estimator

\widehat{\theta}

We now focus our efforts on proving a convergence guarantee for Algorithm 3. We present a set of assumptions we need to prove our convergence guarantee.

Assumption 5.

Let $\mathcal{A}_{1}:\mathcal{Z}^{\ast}\times\Theta\rightarrow\mathcal{G}$ be a nuisance estimation algorithm taking in an arbitrary number of points, and let $\mathcal{A}_{2}:\Theta\times(\mathcal{X}\rightarrow\mathbb{R})^{\ast}\rightarrow\Theta$ be a general loss calibration algorithm taking some initial estimator and a sequence of partially-evaluated loss functions $\ell_{i}:\mathcal{X}\rightarrow\mathbb{R}$ .

1.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $Z_{1},\dots,Z_{n}\sim_{i.i.d.}P_{Z}$ , $\theta\in\Theta$ , and failure probability $\delta_{1}\in(0,1)$ , we have

$\mathrm{err}(\widehat{g},g_{\theta};\gamma_{\theta}^{\ast})\leq\mathrm{rate}_{1}(n,\delta_{1},\theta;P_{Z}),$

where $\widehat{g}=(\widehat{\eta},\widehat{\zeta})\leftarrow\mathcal{A}_{1}(Z_{1:n},\theta)$ , $g_{\theta}=(\eta_{0},\zeta_{\theta})$ , and $\mathrm{rate}_{1}$ is some rate function.
2.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $Z_{1},\dots,Z_{n}\sim_{i.i.d.}P_{Z}$ , $\theta\in\Theta$ , $g\in\mathrm{range}(\mathcal{A}_{1})\subset\mathcal{G}$ , and failure probability $\delta_{2}\in(0,1)$ , we have

$\mathrm{Cal}(\widehat{\theta};g)\leq\mathrm{rate}_{2}(n,\delta_{2},\ell;P_{Z}),$

where $\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,\{(X_{m},\ell_{m})\}_{m=1}^{n})$ , $\ell_{m}:=\ell(\cdot,\widehat{g};Z_{m})$ , and $\mathrm{rate}_{2}$ is some rate function.
3.

With probability one, $\widehat{\theta}=\tau\circ\theta$ for some injective mapping $\tau:\mathbb{R}\rightarrow\mathbb{R}$ .

The first two assumptions are direct analogues of those made in Section 3, giving the learner control over nuisance estimation and calibration rates. In general, $\zeta_{\theta}\neq\zeta_{\widehat{\theta}}$ for arbitrary univariate mappings $\tau:\mathbb{R}\rightarrow\mathbb{R}$ . This is a problem, as the learner will estimate the additional nuisance associated with the initial parameter, $\zeta_{\theta}$ , but Theorem 4.5 will be instantiated with respect to the nuisance $\zeta_{\widehat{\theta}}$ . The following lemma shows that injectivity of $\tau$ ensures $\zeta_{\theta}=\zeta_{\widehat{\theta}}$ .

Lemma 4.7.

Suppose $\ell$ is conditionally orthogonal, and suppose $\varphi_{1},\varphi_{2}:\mathcal{X}\rightarrow\mathbb{R}$ have the same level sets, i.e. they satisfy $\{\varphi_{1}^{-1}(c):c\in\mathrm{range}(\varphi_{1})\}=\{\varphi_{2}^{-1}(c):c\in\mathrm{range}(\varphi_{2})\}$ . ⁹⁹9For a function $f:\mathcal{X}\rightarrow\mathcal{Y}$ that is not necessarily injective, we let $f^{-1}(c):=\{x\in\mathcal{X}:f(x)=c\}$ . Then, the calibration functions satisfy $\gamma_{\varphi_{1}}^{\ast}=\gamma_{\varphi_{2}}^{\ast}$ . Additionally, without loss of generality, we can assume $\zeta_{\varphi_{1}}\equiv\zeta_{\varphi_{2}}$ .

Calibration algorithms that learn an injective post-processing mapping, such as Platt scaling and linear calibration, will preserve level sets. For an algorithm like isotonic calibration, one can either (a) estimate $\zeta_{\theta}$ and hope $\zeta_{\theta}\approx\zeta_{\widehat{\theta}}$ or (b) if $\widehat{\tau}\in\{f:\mathbb{R}\rightarrow\mathbb{R}\mid f\text{ is monotonic}\}$ learned via isotonic regression is not strictly increasing, one can release $\widehat{\theta}=(\widehat{\tau}+\xi)\circ\theta$ where $\xi$ is any strictly increasing map. For algorithms such as histogram binning and uniform mass binning, one can first learn then the level sets of $\widehat{\theta}$ , which are just based on the quantiles of the predictions of the initial estimator $\theta(X)$ . Then, these level sets entirely specify the target nuisance $\zeta_{\widehat{\theta}}$ to estimate. We provide a version of Algorithm 3 in Appendix D that does this. We now provide a convergence guarantee for Algorithm 3.

Theorem 4.8.

Suppose $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ is an arbitrary, conditionally orthogonal loss Let $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ satisfy Assumption 5, and suppose $\widehat{\theta}=\widehat{\tau}\circ\theta$ where $\widehat{\tau}$ is almost surely injective. Then, with probability at least $1-\delta_{1}-\delta_{2}$ , the output $\widehat{\theta}$ of Algorithm 1 run on a calibration dataset of $2n$ i.i.d. data points $Z_{1},\dots,Z_{2n}\sim P$ satisfies

\mathrm{Cal}(\widehat{\theta},\eta_{0})\leq\frac{1}{2}\mathrm{rate}_{1}(n,\delta_{1},\theta;P_{Z})+\mathrm{rate}_{2}(n,\delta_{2},\ell;P_{Z}).

5 Experiments

We now evaluate the performance of our cross-calibration algorithms, namely Algorithms 2 and 4. We consider two settings. First, we examine the viability of of our calibration algorithm for universally orthogonal losses using observational data. In particular, we show how Algorithm 2 can be used to calibrate estimates of the CATE of 401(k) eligibility and the conditional LATE of 401(k) participation on an individual’s net financial assets. Second, we measure the ability of Algorithm 4 to calibrate estimates of conditional quantiles under treatment on synthetic data. We examine the performance of our algorithm for several quantile both in terms of calibration error and average loss.

5.1 Effects of 401(k) Participation/Eligibility on Financial Assets

First, we consider the task of constructing and calibrating estimates of the heterogeneous effect of 401(k) eligibility/participation on an individual’s net financial assets. To do this, we use the 401(k) dataset leveraged in Chernozhukov and Hansen [5], Kallus et al. [33], and Chernozhukov et al. [10]. Since Chernozhukov and Hansen [5] argue that eligibility for 401(k) satisfies conditional ignorability given a sufficiently rich set of features¹⁰¹⁰10The entire set of features are as follows: [‘age’, ‘inc’, ‘fsize’, ‘educ’, ‘db’, ‘marr’, ‘male’, ‘twoearn’, ‘pira’, ‘nohs’, ‘hs’, ‘smcol’, ‘col’, ‘hown’], we aim to measure the CATE of eligibility on net financial assets. This ignorability is not known to be satisfied for 401(k) participation, and thus we instead aim to measure to conditional LATE of 401(k) participation on net financial assets with eligibility serving as an instrument. For each parameter (either CATE or conditional LATE), we randomly split the dataset into three folds of uneven size: we use 60% of the data to construct the initial parameter estimate, 25% of the data to perform calibration, and reserve 15% of the data as a test set.

Model Training and Calibration:

To fit the initial model CATE/conditional LATE model, we split the training data randomly into $K=5$ evenly-sized folds. We use cross-fitting to construct appropriate pseudo-outcomes, i.e. for each $k\in[5]$ we use data in all but the $k$ th fold to estimate nuisances and then appropriately use these estimates to transform observations in the $k$ th fold. In the case of the CATE, we produce estimates $\widehat{\pi}^{(-k)}$ , $\widehat{\mu}^{(-k)}$ of the propensity score $\pi_{0}(x):=\mathbb{P}(A=1\mid X=x)$ and expected outcome mapping $\mu_{0}(x):=\mathbb{E}[Y\mid X=x,A=a]$ using gradient boosted decision and regression tress, respectively. We then produce pseudo-outcomes on the $k$ th fold, per Equation (2), and use gradient-boosted regression trees to regress these pseudo-outcomes onto covariates.

In the case of the conditional LATE, as the instrument policy $\zeta_{0}(x):=\mathbb{P}(Z=1\mid X=x)$ is not assumed to be known, instead of using the universally orthogonal loss discussed in Equation (4), we instead use the following loss detailed in Syrgkanis et al. [55] and Lan and Syrgkanis [40]:

\ell(\theta,(\mu,\pi,\zeta);W):=\frac{1}{2}\Big{(}\theta(x)-\quad\underbrace{\tau(x)-\frac{(\widetilde{y}-\tau(x)\widetilde{a})\widetilde{Z}}{\zeta(x)}}_{=:\chi_{\mathrm{LATE}}^{\prime}(g;W)}\quad\Big{)}^{2}

where $\tau(x):=\frac{\mu(1,x)-\mu(0,x)}{\zeta(x)}$ , $\widetilde{y}:=y-\mu(a,x)$ , $\widetilde{Z}:=Z-\zeta(x)$ , and $\widetilde{a}:=a-\pi(x)$ , where $\mu_{0}$ and $\pi_{0}$ are as defined above. We estimate all nuisances and construct pseudo-outcomes $\chi^{\prime}_{\mathrm{LATE}}$ as in the case of the CATE, once again using either gradient-boosted regression or decision trees based on appropriateness. Again, we regress pseudo-outcomes onto covariates using gradient-boosted regression trees to obtain initial parameter estimates.

After initial models are trained, we run Algorithm 2 (cross-calibration) on the 25% of the data reserved for calibration, once again using $K=5$ folds. We estimate all nuisances again by either using gradient-boosted decision or regression trees. We perform calibration using three different algorithms: isotonic calibration, histogram binning with $B=20$ buckets, and linear calibration, which performs simple linear regression with intercept of the constructed pseudo-outcomes onto the initial model predictions.

Comparing Calibration Error in Quartiles

We assess calibration of both the pre- and post-calibrated models by approximating the actual target treatment effect with the quartiles of the models’ predictions. We let $\widehat{\theta}$ denote either the pre or post calibrated model (either CATE or conditional LATE). Re-using the calibration dataset (25% of the data), we compute the order statistics of model predictions $\widehat{\theta}(X)_{(1)},\dots,\widehat{\theta}(X)_{(N)}$ . We then define four buckets based on the quartiles of the above order statistics, i.e.

B_{i}=\left\{x\in\mathcal{X}:\widehat{\theta}(x)\in\Big{(}\widehat{\theta}(X)_{(\lfloor iN/4\rfloor)},\widehat{\theta}(X)_{(\lfloor(i+1)N/4\rfloor)}\Big{]}\right\}\qquad\text{for }i=0,1,2,3

where we assume $\widehat{\theta}(X)_{(0)}=-\infty$ and $\widehat{\theta}(X)_{(N)}=\infty$ . Next, we use cross-fitting with $K=5$ folds to transform the 15% of the data reserved for testing into pseudo-outcomes $\chi_{m}$ .¹¹¹¹11This is done in the same manner as discussed in the previous subsection We assign each transformed sample $(X_{m},\chi_{m})$ in the test set to an appropriate bin based on the predicted value of $\widehat{\theta}(X_{m})$ , and average the pseudo outcomes falling into each bin. We then approximate the $L^{2}$ calibration error, which is computed as: $\widehat{\mathrm{Cal}}(\widehat{\theta},g_{0})=\frac{1}{4}\sum_{i=1}^{4}(\overline{\chi}_{i}-\overline{\theta}_{i})^{2}$ , where $\overline{\theta}_{i}$ denotes the average of the $\widehat{\theta}(X_{m})$ falling into bin $i$ and $\widehat{\chi}_{i}$ denotes the average pseudo-outcome $\chi_{m}$ in bin $i$ .

Refer to caption — (a) Calibration Error for CATE

We repeat the above experiment over $M=100$ random splits of the initial 401(k) dataset. Figure 1 shows in a box and whisker plot the empirical distribution of the $L^{2}$ calibration errors over these $M$ runs for the base (uncalibrated) model, and the model calibrated using each of thee aforementioned calibration algorithms. Each box is centered at the median calibration error $m$ , the bottom of the box is given by the first quartile of calibration error $Q1$ , and the top of the box is given by the third quartile $Q3$ of calibration error. The top (bottom) of the whiskers are given by the maximum (resp. minimum) of the observations falling within $m\pm 1.5\times(Q3-Q1)$ .

From Figure 1, we see that all calibration algorithms result in noticeably smaller calibration error. In particular in the setting of the CATE of 401(k) eligibility on net financial assets, the third quartile for the calibration error using isotonic calibration and linear calibration falls below the first quartile of calibration error for the uncalibrated model. Likewise, in the LATE model, which aims to measure the effect of 401(k) participation on net financial assets, the third quartile of calibration error under histogram binning and linear calibration falls below the first quartile of calibration error for the uncalibrated model. This indicates that our calibration algorithms have a significant impact on the calibration error of the produced models.

5.2 Calibrating Estimates of Conditional Quantiles Under Treatment

We now show how Algorithm 4 (cross-calibration for conditionally orthogonal loss functions) can be used to calibrate estimates of the conditional $Q$ th quantile under treatment. In particular, we study the the impact of sample size and chosen quantile have on both $L^{2}$ calibration error and average loss.

For this experiment, we leverage synthetically generated data. At the start of the experiment, we generate two slope vector $\beta^{Y},\beta^{\pi}\in\mathbb{R}^{100}$ where $\beta^{Y}_{1:20},\beta^{\pi}_{1:20},\sim\mathcal{N}(0,I_{20})$ are i.i.d. and $\beta^{Y}_{i}=\beta^{\pi}_{i}=0$ for all $i>20$ . Then, for $N\in[500,1000,1500,2000,2500,3000]$ , we generate i.i.d. covariates $X_{1},\dots,X_{N}\sim\mathcal{N}(0,I_{100})$ and treatments $A_{1},\dots,A_{N}$ where $A_{i}\sim\mathrm{Bern}(p_{i})$ and

p_{i}:=\max\left\{C,\min\left\{\frac{1}{1+\exp(\langle\beta^{\pi},X_{i}\rangle)},1-C\right\}\right\}.

In the above, $C=0.05$ is a fixed clipping parameter. Finally, we generate potential outcomes under treatment and control as $Y_{i}(1)=Y_{i}(0)=\langle\beta^{Y},X_{i}\rangle+\epsilon_{i}$ , where $\epsilon_{1},\dots,\epsilon_{N}$ are i.i.d. standard normal noise variables. We note that, because we are only interested in the conditional quantile under treatment, we make potential equivalent under both treatment and control for simplicity.

Given data as generated above, we then train an initial model $\widehat{\theta}$ using gradient-boosted regression trees. In more detail, in training these regression trees, we leverage the loss $\widetilde{\ell}_{\mathrm{QUT}}(\theta,p;z):=a\cdot p(x)(y-\theta(x))(Q-\mathbbm{1}_{y\leq\theta(x)})$ , where we determine the nuisances using $K=5$ fold cross-fitting, i.e. where we estimate the propensity $\pi_{0}(x):=\mathbb{P}(A=1\mid X=x)\equiv p_{0}(x)^{-1}$ using gradient-boosted decision trees. We note that this loss is Neyman orthogonal conditional on covariates $X$ , and thus using this loss allows for the efficient estimate of the quantile under treatment. However, this loss is not conditionally orthogonal, and thus is a poor fit for performing calibration.

We then use an additional $N$ samples generated in the same manner as above to perform calibration using Algorithm 4 and loss function $\ell_{\mathrm{QUT}}(\theta,(p,f);z)$ (outlined in Equation (6)). We estimate the inverse propensity $p_{0}(x)$ by using gradient-boosted decision trees to estimate the propensity $\pi_{0}(x)=p_{0}(x)^{-1}$ . We estimate the additional CDF-like nuisance $f_{\widehat{\theta}}(x):=\mathbb{P}(Y(1)\leq\gamma_{\widehat{\theta}}^{\ast}(X)\mid X=x)$ by instead estimating $\mathbb{P}(Y(1)\leq\widehat{\theta}(X)\mid X=x)$ , again using gradient-boosted decision trees. Heuristically, we are hoping that $\widehat{\theta}$ is a “reasonable rough estimate” for the calibration function $\gamma_{\widehat{\theta}}^{\ast}$ . In both settings, we perform calibration using linear calibration. After we estimate nuisance, we run the cross-calibration algorithm by using linear calibration. We repeat the entire above process $M=50$ times for three quantile values $Q\in\{0.5,0.75,0.9\}$ , and plot both the empirical $L^{2}$ calibration errors and average losses below (measuring loss according to $\widetilde{\ell}_{\mathrm{QUT}}(\theta,p_{0};z)$ , i.e. the loss evaluated at the true nuisances).

Results:

Figure 2 displays the results of the experimental procedure outlined above. Displayed in the left-hand column are plots demonstrating the empirical $L^{2}$ calibration at various sample sizes averaged over the $M=50$ runs. Likewise, in the right-hand column, the sample loss is displayed averaged again over $M$ runs. We include point-wise valid 95% confidence intervals for all plots.

Regardless of the sample size $N$ and chosen quantile $Q$ , we see a significant decrease in the $L^{2}$ calibration error. This shows that not only does Algorithm 4 exhibit favorable theoretical convergence guarantees, but that it also offer strong performance in practice. Typically, one hopes calibration algorithms enjoy a “do no harm” property, i.e. that calibrating a parameter estimate will not significantly increase loss. While we do not formally prove this, the right-hand column of Figure 2 demonstrates that calibrating typically decreases loss, as desired. Moreover, the loss obtained by using $N$ samples to construct an initial estimate and $N$ to perform calibration is comparable to the loss had $2N$ samples directly been used to estimate the unknown conditional parameter. This suggests that reserving some samples for calibration yields significantly lower calibration error without adversely affecting performance.

6 Conclusion

In this work, we constructed a framework for calibrating general estimates of heterogeneous causal effects with respect to nuisance-dependent losses. By leveraging variants of Neyman orthogonality, we were able to bound the $L^{2}$ calibration error of $\theta$ by two decoupled terms. One term, roughly, represented the error in nuisance estimation, while the other term represented the $L^{2}$ calibration error of $\theta$ in a world where the learned nuisances were true. These bounds suggested natural sample-splitting and cross-calibration algorithms, for which we proved high-probability convergence guarantees. Our algorithms also admitted “do no harm” style guarantees. We empirically evaluated our algorithms in Section 5, in which we considered both observational and synthetic data.

While our provided contributions are quite general, there are still interesting directions for future work. First, in our work, we only measure the convergence of our algorithms via the $L^{2}$ -calibration error. Depending on the situation, other notions of calibration error may be more appropriate. For instance, Gupta and Ramdas [25] analyze the convergence of histogram/uniform mass binning in terms of $L^{\infty}$ calibration error. Likewise, Globus-Harris et al. [21] study $L^{1}$ muli-calibration error. We leave it as interesting future work to extend our results to the general setting of measuring $L^{p}$ calibration error. Further, it would be interesting to study how the calibration of estimates of heterogeneous causal effects impacts utility in decision making tasks.

7 Acknowledgments

VS is supported by NSF Award IIS 2337916.

References

Abrevaya et al. [2015] Jason Abrevaya, Yu-Chin Hsu, and Robert P Lieli. Estimating conditional average treatment effects. Journal of Business & Economic Statistics, 33(4):485–505, 2015.
Barlow and Brunk [1972] Richard E Barlow and Hugh D Brunk. The isotonic regression problem and its dual. Journal of the American Statistical Association, 67(337):140–147, 1972.
Bickel et al. [1993] Peter J Bickel, Chris AJ Klaassen, Peter J Bickel, Ya’acov Ritov, J Klaassen, Jon A Wellner, and YA’Acov Ritov. Efficient and adaptive estimation for semiparametric models, volume 4. Springer, 1993.
Blum and Mansour [2007] Avrim Blum and Yishay Mansour. From external to internal regret. Journal of Machine Learning Research, 8(6), 2007.
Chernozhukov and Hansen [2004] Victor Chernozhukov and Christian Hansen. The effects of 401 (k) participation on the wealth distribution: an instrumental quantile regression analysis. Review of Economics and statistics, 86(3):735–751, 2004.
Chernozhukov et al. [2018] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters, 2018.
Chernozhukov et al. [2022a] Victor Chernozhukov, Whitney Newey, Vıctor M Quintas-Martınez, and Vasilis Syrgkanis. Riesznet and forestriesz: Automatic debiased machine learning with neural nets and random forests. In International Conference on Machine Learning, pages 3901–3914. PMLR, 2022a.
Chernozhukov et al. [2022b] Victor Chernozhukov, Whitney K Newey, and Rahul Singh. Automatic debiased machine learning of causal and structural effects. Econometrica, 90(3):967–1027, 2022b.
Chernozhukov et al. [2022c] Victor Chernozhukov, Whitney K Newey, and Rahul Singh. Debiased machine learning of global and local parameters using regularized riesz representers. The Econometrics Journal, 25(3):576–601, 2022c.
Chernozhukov et al. [2024] Victor Chernozhukov, Christian Hansen, Nathan Kallus, Martin Spindler, and Vasilis Syrgkanis. Applied causal inference powered by ml and ai. arXiv preprint arXiv:2403.02467, 2024.
Chung et al. [2023] Youngseog Chung, Aaron Rumack, and Chirag Gupta. Parity calibration. arXiv preprint arXiv:2305.18655, 2023.
Dawid [1982] A Philip Dawid. The well-calibrated bayesian. Journal of the American Statistical Association, 77(379):605–610, 1982.
Dawid [1985] A Philip Dawid. Calibration-based empirical probability. The Annals of Statistics, 13(4):1251–1274, 1985.
Fan et al. [2022] Qingliang Fan, Yu-Chin Hsu, Robert P Lieli, and Yichong Zhang. Estimation of conditional average treatment effects with high-dimensional data. Journal of Business & Economic Statistics, 40(1):313–327, 2022.
Feng et al. [2012] Ping Feng, Xiao-Hua Zhou, Qing-Ming Zou, Ming-Yu Fan, and Xiao-Song Li. Generalized propensity score for estimating the average treatment effect of multiple treatments. Statistics in medicine, 31(7):681–697, 2012.
Foster and Vohra [1999] Dean P Foster and Rakesh Vohra. Regret in the on-line decision problem. Games and Economic Behavior, 29(1-2):7–35, 1999.
Foster and Vohra [1998] Dean P Foster and Rakesh V Vohra. Asymptotic calibration. Biometrika, 85(2):379–390, 1998.
Foster and Syrgkanis [2023] Dylan J Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics, 51(3):879–908, 2023.
Fröolich and Melly [2010] Markus Fröolich and Blaise Melly. Estimation of quantile treatment effects with stata. The Stata Journal, 10(3):423–457, 2010.
Gao et al. [2024] Chen Gao, Yu Zheng, Wenjie Wang, Fuli Feng, Xiangnan He, and Yong Li. Causal inference in recommender systems: A survey and future directions. ACM Transactions on Information Systems, 42(4):1–32, 2024.
Globus-Harris et al. [2023] Ira Globus-Harris, Declan Harrison, Michael Kearns, Aaron Roth, and Jessica Sorrell. Multicalibration as boosting for regression. arXiv preprint arXiv:2301.13767, 2023.
Gopalan et al. [2024] Parikshit Gopalan, Michael Kim, and Omer Reingold. Swap agnostic learning, or characterizing omniprediction via multicalibration. Advances in Neural Information Processing Systems, 36, 2024.
Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.
Gupta [2022] Chirag Gupta. Post-hoc calibration without distributional assumptions. PhD thesis, PhD thesis, Carnegie Mellon University Pittsburgh, PA 15213, USA, 2022.
Gupta and Ramdas [2021] Chirag Gupta and Aaditya Ramdas. Distribution-free calibration guarantees for histogram binning without sample splitting. In International Conference on Machine Learning, pages 3942–3952. PMLR, 2021.
Gupta and Ramdas [2023] Chirag Gupta and Aaditya Ramdas. Online platt scaling with calibeating. arXiv preprint arXiv:2305.00070, 2023.
Gupta et al. [2020] Chirag Gupta, Aleksandr Podkopaev, and Aaditya Ramdas. Distribution-free binary classification: prediction sets, confidence intervals and calibration. Advances in Neural Information Processing Systems, 33:3711–3723, 2020.
Hasminskii and Ibragimov [1979] Rafail Z Hasminskii and Ildar A Ibragimov. On the nonparametric estimation of functionals. In Proceedings of the Second Prague Symposium on Asymptotic Statistics, volume 473, pages 474–482. North-Holland Amsterdam, 1979.
Hébert-Johnson et al. [2018] Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning, pages 1939–1948. PMLR, 2018.
Heckman and Vytlacil [2005] James J Heckman and Edward Vytlacil. Structural equations, treatment effects, and econometric policy evaluation 1. Econometrica, 73(3):669–738, 2005.
Hendrycks et al. [2019] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781, 2019.
Hitsch et al. [2024] Günter J Hitsch, Sanjog Misra, and Walter W Zhang. Heterogeneous treatment effects and optimal targeting policy evaluation. Quantitative Marketing and Economics, 22(2):115–168, 2024.
Kallus et al. [2019] Nathan Kallus, Xiaojie Mao, and Masatoshi Uehara. Localized debiased machine learning: Efficient inference on quantile treatment effects and beyond. arXiv preprint arXiv:1912.12945, 2019.
Kennedy [2022] Edward H Kennedy. Semiparametric doubly robust targeted double machine learning: a review. arXiv preprint arXiv:2203.06469, 2022.
Kennedy [2023] Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008–3049, 2023.
Kennedy et al. [2023] Edward H Kennedy, Sivaraman Balakrishnan, and LA Wasserman. Semiparametric counterfactual density estimation. Biometrika, 110(4):875–896, 2023.
Kent et al. [2018] David M Kent, Ewout Steyerberg, and David Van Klaveren. Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects. Bmj, 363, 2018.
Kumar et al. [2019] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32, 2019.
Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Lan and Syrgkanis [2023] Hui Lan and Vasilis Syrgkanis. Causal q-aggregation for cate model selection. arXiv preprint arXiv:2310.16945, 2023.
Leng and Dimmery [2024] Yan Leng and Drew Dimmery. Calibration of heterogeneous treatment effects in randomized experiments. Information Systems Research, 2024.
Malinin and Gales [2018] Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Minderer et al. [2021] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682–15694, 2021.
Neyman [1979] Jerzy Neyman. C( $\alpha$ ) tests and their use. Sankhyā: The Indian Journal of Statistics, Series A (1961-2002), 41(1/2):1–21, 1979. ISSN 0581572X. URL http://www.jstor.org/stable/25050174.
Niculescu-Mizil and Caruana [2005] Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, pages 625–632, 2005.
Nie and Wager [2021] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. Biometrika, 108(2):299–319, 2021.
Noarov and Roth [2023] Georgy Noarov and Aaron Roth. The scope of multicalibration: Characterizing multicalibration via property elicitation. arXiv preprint arXiv:2302.08507, 2023.
Noarov et al. [2023] Georgy Noarov, Ramya Ramalingam, Aaron Roth, and Stephan Xie. High-dimensional prediction for sequential decision making. arXiv preprint arXiv:2310.17651, 2023.
Oprescu et al. [2019] Miruna Oprescu, Vasilis Syrgkanis, and Zhiwei Steven Wu. Orthogonal random forest for causal inference. In International Conference on Machine Learning, pages 4932–4941. PMLR, 2019.
Ovadia et al. [2019] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Platt et al. [1999] John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74, 1999.
Sahoo et al. [2021] Roshni Sahoo, Shengjia Zhao, Alyssa Chen, and Stefano Ermon. Reliable decisions with threshold calibration. Advances in Neural Information Processing Systems, 34:1831–1844, 2021.
Semenova and Chernozhukov [2021] Vira Semenova and Victor Chernozhukov. Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal, 24(2):264–289, 2021.
Song et al. [2019] Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. In International Conference on Machine Learning, pages 5897–5906. PMLR, 2019.
Syrgkanis et al. [2019] Vasilis Syrgkanis, Victor Lei, Miruna Oprescu, Maggie Hei, Keith Battocchi, and Greg Lewis. Machine learning estimation of heterogeneous treatment effects with instruments. Advances in Neural Information Processing Systems, 32, 2019.
Thulasidasan et al. [2019] Sunil Thulasidasan, Gopinath Chennupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. Advances in neural information processing systems, 32, 2019.
Tsybakov [2009] Alexandre B. Tsybakov. Introduction to Nonparametric Estimation. Springer series in statistics. Springer, 2009. ISBN 978-0-387-79051-0. doi: 10.1007/B13794. URL https://doi.org/10.1007/b13794.
van der Laan et al. [2023] Lars van der Laan, Ernesto Ulloa-Pérez, Marco Carone, and Alex Luedtke. Causal isotonic calibration for heterogeneous treatment effects. arXiv preprint arXiv:2302.14011, 2023.
Vovk [2002] Vladimir Vovk. On-line confidence machines are well-calibrated. In The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings., pages 187–196. IEEE, 2002.
Wainwright [2019] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
Wilder and Welle [2024] Bryan Wilder and Pim Welle. Learning treatment effects while treating those in need. arXiv preprint arXiv:2407.07596, 2024.
Zadrozny and Elkan [2001] Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, pages 609–616, 2001.
Zadrozny and Elkan [2002] Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 694–699, 2002.

Appendix A Calibration Error Decomposition Proofs

In this appendix, we prove the main error decompositions from Sections 3 an d4. These results provide two-term, decoupled bounds on the $L^{2}$ calibration error of an arbitrary, fixed parameter $\theta:\mathcal{X}\rightarrow\mathbb{R}$ in terms of $L^{2}$ calibration error assuming the learned nuisances were correct, and a term measuring the distance between the learned nuisances and the true, unknown nuisance parameters.

A.1 Universally Orthogonality

We start by proving Theorem 3.3, which provides the claimed decoupled bound under the assumption that $\ell$ is universally orthogonal (Definition 3.1). See 3.3

Proof of Theorem 3.3.

We start by adding a useful form of zero to the integrand, which yields

	$\displaystyle\mathrm{Cal}(\theta,g_{0})^{2}$	$\displaystyle=\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]^{2}\right)$
		$\displaystyle=\underbrace{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]\cdot\left\{\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]-\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]\right\}\right)}_{T_{1}}$
		$\displaystyle\qquad+\underbrace{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]\cdot\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]\right)}_{T_{2}}.$

We bound $T_{1}$ and $T_{2}$ separately. As a first step in bounding $T_{1}$ , note that by a second order Taylor expansion with respect to the nuisance estimate $g\in\mathcal{G}$ with Lagrange remainder, we have

	$\displaystyle-\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]+\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]$
	$\displaystyle\qquad=D_{g}\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right](g-g_{0})+\frac{1}{2}D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,\overline{g};Z)\mid\theta(X)\right](g-g_{0},g-g_{0})$
	$\displaystyle\qquad=\frac{1}{2}D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,\overline{g};Z)\mid\theta(X)\right](g-g_{0},g-g_{0}).$

In the above, $\overline{g}\in[g_{0},g]$ , and the first-order derivative (with respect to $g$ ) vanishes due to the assumption of Definition 3.1. This is because we have Taylor expanded around the true, unknown nuisance $g_{0}=g_{0}$ .

With this, we can apply the Cauchy-Schwarz inequality, which furnishes

	$\displaystyle T_{1}$	$\displaystyle\leq\frac{1}{2}\sqrt{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]^{2}\right)}\sqrt{\mathbb{E}\left(\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,\overline{g};Z)\mid\theta(X)\right](g-g_{0},g-g_{0})\right\}^{2}\right)}$
		$\displaystyle\leq\frac{1}{2}\sqrt{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]^{2}\right)}\sqrt{\mathbb{E}\left(\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,\overline{g};Z)\mid X\right](g-g_{0},g-g_{0})\right\}^{2}\right)}$
		$\displaystyle\leq\frac{1}{2}\mathrm{Cal}(\theta,g_{0})\cdot\mathrm{err}(g,g_{0};\theta)$

In the second line, we apply Jensen’s inequality inside the conditional expectation.

Bounding $T_{2}$ is more straightforward. Applying the Cauchy-Schwarz inequality, we have:

	$\displaystyle T_{2}$	$\displaystyle\leq\sqrt{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{0};Z)\mid\theta(X)\right]^{2}\right)}\sqrt{\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]^{2}\right)}$
		$\displaystyle=\mathrm{Cal}(\theta,g_{0})\cdot\mathrm{Cal}(\theta,g)$

This line of reasoning, in total, yields that

\mathrm{Cal}(\theta,g_{0})^{2}\leq\frac{1}{2}\mathrm{Cal}(\theta,g_{0})\cdot\mathrm{err}(g,g_{0};\theta)+\mathrm{Cal}(\theta,g_{0})\cdot\mathrm{Cal}(\theta,g).

Dividing through by $\mathrm{Cal}(\theta,g_{0})$ yields the claimed bound. ∎

A.2 Conditional Orthogonality

We now turn to proving the second error bound, which holds in the case that $\ell$ satisfies the weaker assumption of conditional orthogonality (Definition 4.3). We generically write $g=(\eta,\zeta)$ for a fixed pair of nuisance parameters. Further, for any fixed post-processing function $\varphi:\mathcal{X}\rightarrow\mathcal{X}^{\prime}$ , we write $g_{\varphi}=(\eta_{0},\zeta_{\varphi})$ where $\zeta_{\varphi}$ is a nuisance parameter ensuring the vanishing cross-derivative condition. To prove Theorem 4.5, we will need two technical lemmas. In what follows, we remind the reader of the definition of the calibration function $\gamma_{\varphi}:\mathcal{X}\times\mathcal{G}\rightarrow\mathbb{R}$ denote the calibration function under the orthogonalized loss $\ell$ , i.e. $\gamma_{\varphi}$ is specified by

\gamma_{\theta}(x;g):=\arg\min_{\nu}\mathbb{E}[\ell(\theta,g;Z)\mid\theta(X)=\theta(x)],

We recall the identity $\gamma_{\varphi}^{\ast}\equiv\gamma_{\varphi}(\cdot;(\eta_{0},\zeta))$ for any estimate $\zeta$ of the second nuisance parameter, which will be useful in the sequel.

The first lemma we prove measures the distance (in terms of the $L^{2}(P_{X})$ norm) between the true calibration $\gamma_{\theta}^{\ast}$ and the calibration function under any other nuisance pair $g=(\eta,\zeta)$ , $\gamma_{\theta}(\cdot;g)$ . Here, $\theta:\mathcal{X}\rightarrow\mathbb{R}$ should be viewed as some arbitrary estimator. We can bound this distance in terms of the complicated looking error term, which was first introduced in Theorem 3.3. This term actually simplifies rather nicely, as was seen in the prequel when we computed the quantity for the task of calibrating estimates of conditional $Q$ th quantiles under treatment.

Lemma A.1.

Let $\theta:\mathcal{X}\rightarrow\mathbb{R}$ be an arbitrary function, assume $\ell$ is conditionally orthogonal, and let $g_{\theta}=(\eta_{0},\zeta_{\theta})$ be a pair of nuisance functions guaranteeing the vanishing cross-derivative condition in Definition 4.3. Let $g=(\eta,\zeta)$ be some fixed pair of nuisance functions. Then, assuming the base loss $\ell$ satisfies $\alpha$ -strong convexity (Assumption 3), we have

\|\gamma_{\theta}^{\ast}-\gamma_{\theta}(\cdot;g)\|_{L^{2}(P_{X})}\leq\frac{1}{2\alpha}\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast}),

where we define $\mathrm{err}(g,h;\varphi):=\sup_{f\in[g,h]}\sqrt{\mathbb{E}\left(\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\varphi,f;Z)\mid X\right](g-h,g-h)\right\}^{2}\right)}$ .

Proof.

First, strong convexity (Assumption 3) alongside equivalent conditions for strong convexity (namely that $\alpha(x-y)^{2}\leq(\partial f(x)-\partial f(y))(x-y)$ ) yields:

	$\displaystyle\alpha\left(\gamma_{\theta}(X;g_{\theta})-\gamma_{\theta}(X;g)\right)^{2}$
	$\displaystyle\qquad\leq\left(\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g;Z)\mid\theta(X)\right]-\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g),g;Z)\mid\theta(X)\right]\right)\left(\gamma_{\theta}(X;g_{\theta})-\gamma_{\theta}(X;g)\right)$
	$\displaystyle\qquad=\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g;Z)\mid\theta(X)\right]\left(\gamma_{\theta}(X;g_{\theta})-\gamma_{\theta}(X;g)\right)$

In the above, the equality on the third line follows from the definition of $\gamma_{\theta}(\cdot;g)$ , as first order optimality conditions on $\gamma_{\theta}(x;g)=\arg\min_{\nu\in\mathbb{R}}\mathbb{E}\left[\ell(\nu,g;Z)\mid\theta(X)\right]$ imply $\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g),g;Z)\mid\theta(X)\right]=0$ .

Rearranging the above inequality and taking absolute values yields

\alpha\left|\gamma_{\theta}(X;g_{\theta})-\gamma_{\theta}(X;g)\right|\leq\left|\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g;Z)\mid\theta(X)\right]\right|.

Next, observe that from the condition $\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g_{\theta};Z)\mid\theta(X)\right]=0$ alongside a second order Taylor expansion (with respect to nuisance pairs $g$ ) with Lagrange form remainder plus conditional orthogonality (Definition 4.3), we have

	$\displaystyle\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g;Z)\mid\theta(X)\right]$
	$\displaystyle\qquad=\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g;Z)\mid\theta(X)\right]-\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g_{\theta};Z)\mid\theta(X)\right]$
	$\displaystyle\qquad=D_{g}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),g_{\theta};Z)\mid\theta(X)\right](g-g_{\theta})+\frac{1}{2}D_{g}^{2}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),\overline{g};Z)\mid\theta(X)\right](g-g_{\theta},g-g_{\theta})$
	$\displaystyle\qquad=\frac{1}{2}D_{g}^{2}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),\overline{g};Z)\mid\theta(X)\right](g-g_{\theta},g-g_{\theta})$
	$\displaystyle\qquad=\frac{1}{2}\mathbb{E}\left(D_{g}^{2}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),\overline{g};Z)\mid X\right](g-g_{\theta},g-g_{\theta})\mid\theta(X)\right),$

where $\overline{g}\in[g,g_{\theta}]$ (here, $\overline{g}\in[g,g_{\theta}]$ indicates $\overline{g}=\lambda g+(1-\lambda)g_{\theta}$ for some $\lambda\in[0,1]$ ). Consequently, we have

	$\displaystyle\\|\gamma_{\theta}(X;g_{\theta})-\gamma_{\theta}(X;g)\\|_{L^{2}(P_{X})}$
	$\displaystyle\qquad=\left(\int_{\mathcal{X}}\left\|\gamma_{\theta}(x;g_{\theta})-\gamma_{\theta}(x;g)\right\|^{2}P_{X}(dx)\right)^{1/2}$
	$\displaystyle\qquad\leq\frac{1}{\alpha}\left(\int_{\mathcal{X}}\mathbb{E}\left[\partial\ell(\gamma_{\theta}^{\ast},g;Z)\mid\theta(X)=\theta(x)\right]^{2}P_{X}(dx)\right)^{1/2}$
	$\displaystyle\qquad\leq\frac{1}{2\alpha}\left(\int_{\mathcal{X}}\mathbb{E}\left(D_{g}^{2}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),\overline{g};Z)\mid X\right](g-g_{\theta},g-g_{\theta})\mid\theta(X)=\theta(x)\right)^{2}P_{X}(dx)\right)^{1/2}$
	$\displaystyle\qquad\leq\frac{1}{2\alpha}\left(\int_{\mathcal{X}}\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot;g_{\theta}),\overline{g};Z)\mid X=x\right](g-g_{\theta},g-g_{\theta})\right\}^{2}P_{X}(dx)\right)^{1/2}$
	$\displaystyle\qquad\leq\frac{1}{2\alpha}\mathrm{err}(g,g_{\theta};\gamma_{\theta}(\cdot;g_{\theta})).$

Noting the identity $\gamma_{\theta}(\cdot;g_{\theta})\equiv\gamma_{\theta}^{\ast}$ proves the claimed result. ∎

The second lemma we prove bounds the $L^{2}(P_{X})$ distance between the parameter estimate $\theta$ and the calibration function $g_{\theta}(\cdot;g)$ under a fixed pair of nuisances $g=(\eta,\zeta)$ in terms of the calibration error.

Lemma A.2.

Let $\theta:\mathcal{X}\rightarrow\mathbb{R}$ be a fixed estimator, $g=(\eta,\zeta)\in\mathcal{G}$ an arbitrary, fixed nuisance pair, and $\gamma_{\theta}:\mathcal{X}\times\mathcal{G}\rightarrow\mathbb{R}$ the calibration function associated with $\theta$ . Assume $\ell$ is $\alpha$ -strongly convex (Assumption 3). We have

\left\|\theta-\gamma_{\theta}(\cdot;g)\right\|_{L^{2}(P_{X})}\leq\frac{1}{\alpha}\mathrm{Cal}(\theta,g).

Proof.

First, observe that from strong convexity (as in the proof of the above lemma), we have

	$\displaystyle\alpha\left(\theta(X)-\gamma_{\theta}(X;g)\right)^{2}$
	$\displaystyle\qquad\leq\left(\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]-\mathbb{E}\left[\partial\ell(\gamma_{\theta}(\cdot,g),g;Z)\mid\theta(X)\right]\right)\left(\theta(X)-\gamma_{\theta}(X;g)\right)$
	$\displaystyle\qquad=\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]\left(\theta(X)-\gamma_{\theta}(X;g)\right).$

Thus, dividing through and taking the absolute value yields:

\left|\theta(X)-\gamma_{\theta}(X;g)\right|\leq\frac{1}{\alpha}\left|\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)\right]\right|.

We now integrate to get the desired result. In particular, we have that

	$\displaystyle\\|\theta-\gamma_{\theta}(\cdot;g)\\|_{L^{2}(P_{X})}$	$\displaystyle=\left(\int_{\mathcal{X}}\left\|\theta(x)-\gamma_{\theta}(x;g)\right\|^{2}P_{X}(dx)\right)^{1/2}$
		$\displaystyle\leq\frac{1}{\alpha}\left(\int_{\mathcal{X}}\mathbb{E}\left[\partial\ell(\theta,g;Z)\mid\theta(X)=\theta(x)\right]^{2}P_{X}(dx)\right)^{1/2}$
		$\displaystyle=\frac{1}{\alpha}\mathrm{Cal}(\theta,g).$

∎

With the above two lemmas in hand, we can now prove Theorem 4.5, which we recall shows a decoupled bound on the calibration of a parameter $\theta$ with respect to a conditionally orthogonal loss function $\ell$ .

See 4.5

Proof of Theorem 4.5.

First, we note that by Corollary 4.4, we have that

\mathrm{Cal}(\theta,\eta_{0})\equiv\mathrm{Cal}(\theta,(\eta_{0},\zeta))

for any second nuisance parameter $\zeta$ , so it suffices to bound $\mathrm{Cal}(\theta,g_{\theta})$ where $g_{\theta}=(\eta_{0},\zeta_{\theta})$ . We have:

	$\displaystyle\mathbb{E}\left[\partial\ell(\theta,g_{\theta};Z)\mid\theta(X)\right]$	$\displaystyle=\mathbb{E}\left[\partial\ell(\theta,g_{\theta};Z)\mid\theta(X)\right]-\mathbb{E}\left[\partial\ell(\gamma_{\theta}^{\ast},g_{\theta};Z)\mid\theta(X)\right]$
		$\displaystyle=\mathbb{E}\left[\partial^{2}\ell(\overline{\theta},g_{\theta};Z)\mid\theta(X)\right](\theta(X)-\gamma_{\theta}^{\ast}(X))$
		$\displaystyle=\mathbb{E}\left[\partial^{2}\ell(\overline{\theta},g_{\theta};Z)\mid\theta(X)\right](\theta(X)-\gamma_{\theta}(X;g))$
		$\displaystyle\quad+\mathbb{E}\left[\partial^{2}\ell(\overline{\theta},g_{\theta};Z)\mid\theta(X)\right](\gamma_{\theta}(X;g)-\gamma_{\theta}^{\ast}(X)),$

where the first equality follows from the above calculation, the second from the fact $\gamma_{\theta}(x,(g_{0},b))=\gamma_{\theta}^{\ast}(x)$ regardless of choice of additional nuisance $b$ , and the third from a first order Taylor expansion with Lagrange form remainder on $\theta(X)$ (here $\overline{\theta}\in[\theta,\gamma_{\theta}^{\ast}]$ ). The final equality follows from adding a subtracting $\gamma_{\theta}(X;g)$ . Thus, we have

	$\displaystyle\mathrm{Cal}(\theta,g_{\theta})^{2}=\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{\theta};Z)\mid\theta(X)\right]^{2}\right)$
	$\displaystyle\qquad=\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{\theta};Z)\mid\theta(X)\right]\cdot\mathbb{E}\left[\partial^{2}\ell(\overline{\theta},g_{\theta};Z)\mid\theta(X)\right]\cdot(\theta(X)-\gamma_{\theta}(X;g))\right)$
	$\displaystyle\qquad\quad+\mathbb{E}\left(\mathbb{E}\left[\partial\ell(\theta,g_{\theta};Z)\mid\theta(X)\right]\cdot\mathbb{E}\left[\partial^{2}\ell(\overline{\theta},g_{\theta};Z)\mid\theta(X)\right]\cdot(\gamma_{\theta}(X;g)-\gamma_{\theta}^{\ast}(X))\right)$
	$\displaystyle\qquad\leq\beta\mathrm{Cal}(\theta,g_{\theta})\sqrt{\mathbb{E}\left[\left(\theta(X)-\gamma_{\theta}(X;g)\right)^{2}\right]}+\beta\mathrm{Cal}(\theta,g_{\theta})\sqrt{\mathbb{E}\left[\left(\gamma_{\theta}(X;g)-\gamma_{\theta}^{\ast}(X)\right)^{2}\right]}$
	$\displaystyle\qquad=\beta\mathrm{Cal}(\theta,g_{\theta})\left\{\left\\|\theta-\gamma_{\theta}(\cdot;g)\right\\|_{L^{2}(P_{X})}+\left\\|\gamma_{\theta}(\cdot;g)-\gamma_{\theta}^{\ast}\right\\|_{L^{2}(P_{X})}\right\}.$

Now, dividing through by $\mathrm{Cal}(\theta,g_{0})$ and plugging in the bounds provided by Lemma A.1 and Lemma A.2 and again leveraging the equivalence $\mathrm{Cal}(\theta,\eta_{0})=\mathrm{Cal}(\theta,g_{\theta})$ , we have

\mathrm{Cal}(\theta,\eta_{0})\leq\frac{\beta}{\alpha}\mathrm{Cal}(\theta,g)+\frac{\beta}{2\alpha}\mathrm{err}(g,g_{\theta};\gamma_{\theta}^{\ast}),

which is precisely the claimed result. ∎

Appendix B Algorithm Convergence Proofs

B.1 Universal Orthogonality

We now restate and prove the convergence guarantee of the sample splitting algorithm for calibration with respect to universally orthogonalizable loss functions. The result below is largely just an application of Theorem 3.3, with the only caveat being that some care must be taken to handle the fact that the output parameter $\widehat{\theta}$ and the nuisance estimate $\widetilde{g}$ are now random variables, not fixed parameters. See 3.5

Proof of Theorem 3.5.

First, observe that, under Assumption 1, for any fixed $\theta$ and $g$ , we have

\displaystyle\underbrace{\mathrm{Cal}(\theta,g)}_{\text{causal cal. error}}=\underbrace{\mathrm{Cal}(\theta,P^{\chi}_{g})}_{\text{non-causal cal. error}},

where $P^{\chi}_{g}$ denotes the joint distribution of $(X,\chi(g;Z))$ over draws $Z\sim P_{Z}$ . and the latter quantity is controlled with high-probability by Assumption 2. Since the above equality holds for any $\theta$ and $g$ , it still holds when $\theta$ is replaced by $\widehat{\theta}$ , the random output of Algorithm 1, and when $g$ is replaced by $\widehat{g}$ , the corresponding nuisance estimate obtained from $\mathcal{A}_{1}$ .

Under Assumption 1, it is also clear that $\mathrm{err}(g,g_{0};\theta)$ does not depend on $\theta$ , so it suffices to write $\mathrm{err}(g,g_{0})$ going forward. Define the “bad” events as $B_{1}:=\{\mathrm{err}(\widehat{g},g_{0})>\mathrm{rate}_{1}(n,\delta_{1};P_{Z})\}$ and $B_{2}:=\{\mathrm{Cal}(\widehat{\theta},\widehat{P}^{\chi}_{\widehat{g}})>\mathrm{rate}_{2}(n,\delta_{2};P^{\chi}_{\widehat{g}})\}$ . Clearly, the first part of Assumption 2 yields that $\mathbb{P}(B_{1})\leq\delta_{1}$ . Likewise, the second part of Assumption 2 also yields that $\mathbb{P}(B_{2}\mid Z_{n+1:2n})\leq\delta_{1},$ as fixing $Z_{n+1},\dots,Z_{2n}$ fixes the learned nuisances $(\widehat{g},\widehat{b})$ , per Algorithm 1. Thus, applying the law of total probability, we have that the marginal probability of $B_{2}$ (over both draws of $Z_{1},\dots,Z_{n}$ and $Z_{n+1},\dots,Z_{2n}$ ) is bounded by

\mathbb{P}(B_{2})=\mathbb{E}[\mathbb{P}(B_{2}\mid Z_{1:n})]\leq\delta_{2}.

This is because the $Z_{i}$ are independent and conditioning on the first $n$ observations fixes the nuisance estimate $\widehat{g}$ . Thus, on the “good” event $B_{1}^{c}\cap B_{2}^{c}$ , which occurs with probability at least $1-\delta_{1}-\delta_{2}$ , we have

	$\displaystyle\mathrm{Cal}(\widehat{\theta},g_{0})$	$\displaystyle\leq\frac{1}{2}\mathrm{err}(\widehat{g},g_{0})+\mathrm{Cal}(\widehat{\theta},\widehat{g})$
		$\displaystyle=\frac{1}{2}\mathrm{err}(\widehat{g},g_{0})+\mathrm{Cal}(\widehat{\theta};P^{\chi}_{\widehat{g}})$
		$\displaystyle\leq\frac{1}{2}\mathrm{rate}_{1}(n,\delta_{1};P_{Z})+\mathrm{rate}_{2}(n,\delta_{2};P^{\chi}_{\widehat{g}}),$

where the first inequality follows from Theorem 3.3 and the second equality follows from the preamble at the beginning of this proof.

∎

B.2 Conditionally Orthogonality

We now prove the convergence guarantees for Algorithm 3. The proof is largely the same as the convergence proof for Algorithm 1, but we nonetheless include it for completeness. The only key difference is that (a) we not longer have access to “pseudo-outcomes” that, in expectation, look like the target treatment effect and (b) we have to be careful, since the second nuisance parameter that is used in the definition of conditional orthogonality does not satisfy $\zeta_{\theta}=\zeta_{\widehat{\theta}}$ in general. Once again, $\theta:\mathcal{X}\rightarrow\mathbb{R}$ is the fixed, initial estimate treated as an input to Algorithm 3 and $\widehat{\theta}$ is the random, calibrated estimate that is the output of the algorithm. See 4.8

Proof of Theorem 4.8.

First, we observe that, by the assumption that $\widehat{\theta}=\widehat{\tau}\circ\theta$ where the random map $\widehat{\tau}:\mathbb{R}\rightarrow\mathbb{R}$ is almost surely injective, we know that initial estimate $\theta$ and the calibrated estimate $\widehat{\theta}$ have the same level sets. Consequently, by Lemma 4.7, we have $\zeta_{\theta}=\zeta_{\widehat{\theta}}$ without loss of generality. Further, we have equivalence of the calibration functions corresponding to $\theta$ and $\widehat{\theta}$ , i.e. we have $\gamma_{\theta}^{\ast}\equiv\gamma_{\widehat{\theta}}^{\ast}$ . Thus, letting $g_{\varphi}=(\eta_{0},\zeta_{\varphi})$ for any $\varphi:\mathcal{X}\rightarrow\mathbb{R}$ , we have

\mathrm{err}(\widehat{g},g_{\widehat{\theta}};\gamma_{\widehat{\theta}}^{\ast})=\mathrm{err}(\widehat{g},g_{\theta};\gamma_{\theta}^{\ast}).

The rest of the proof is now more or less identical to that of Theorem 3.5, but we nonetheless include the proof for completeness. Define the “bad” events as $B_{1}:=\{\mathrm{err}(\widehat{g},g_{\theta};\gamma_{\theta}^{\ast})>\mathrm{rate}_{1}(n,\delta_{1},\theta;P_{Z})\}$ and $B_{2}:=\{\mathrm{Cal}(\widehat{\theta};\widehat{g})>\mathrm{rate}_{2}(n,\delta_{2},\ell;P_{Z})\}$ . Clearly, the first part of Assumption 5 yields that $\mathbb{P}(B_{1})\leq\delta_{1}$ . Likewise, the second part of Assumption 5 also yields that $\mathbb{P}(B_{2}\mid Z_{n+1:2n})\leq\delta_{1},$ as fixing $Z_{n+1},\dots,Z_{2n}$ fixes the learned nuisances $(\widehat{g},\widehat{b})$ , per Algorithm 1. Thus, applying the law of total probability, we have that the marginal probability of $B_{2}$ (over both draws of $Z_{1},\dots,Z_{n}$ and $Z_{n+1},\dots,Z_{2n}$ ) is bounded by

\mathbb{P}(B_{2})=\mathbb{E}[\mathbb{P}(B_{2}\mid Z_{1:n})]\leq\delta_{2}.

	$\displaystyle\mathrm{Cal}(\widehat{\theta},\eta_{0})$	$\displaystyle\leq\frac{1}{2}\mathrm{err}(\widehat{g},g_{\widehat{\theta}};\gamma_{\widehat{\theta}}^{\ast})+\mathrm{Cal}(\widehat{\theta},\widehat{g})$
		$\displaystyle=\frac{1}{2}\mathrm{err}(\widehat{g},g_{\theta};\gamma_{\theta}^{\ast})+\mathrm{Cal}(\widehat{\theta};\widehat{g})$
		$\displaystyle\leq\frac{1}{2}\mathrm{rate}_{1}(n,\delta_{1},\theta;P_{Z})+\mathrm{rate}_{2}(n,\delta_{2},\ell;P_{Z}),$

where the first inequality follows from Theorem 4.5 and the second equality follows from the preamble at the beginning of this proof.

∎

Appendix C A Do No Harm Property for Universally Orthogonal Losses

Throughout the main body of the paper, we focused on deriving bounds on excess calibration error for our causal calibration algorithm. One desideratum for any calibration algorithm is that the calibrated estimator does not have significantly higher loss than the original estimator. This property, called a “do no harm” property, is satisfied by many off-the-shelf calibration algorithms. We prove such a bound for universally orthogonal losses in the following section. In what follows, we denote the risk of a certain predictor $\theta:\mathcal{X}\rightarrow\mathbb{R}$ with respect to a nuisance $g:\mathcal{W}\rightarrow\mathbb{R}^{k}$ as $R(\theta,g):=\mathbb{E}\ell(\theta,g;Z)$ .

We start by proving a generic upper bound on the difference in risk between two arbitrary estimators. The argument presented below is similar in spirit to the proof of Theorem 2 in Foster and Syrgkanis [18].

Theorem C.1.

Let $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ be a universally orthogonal loss function. Let $\theta,\theta^{\prime}:\mathcal{X}\rightarrow\mathbb{R}$ be two estimators, and $g\in\mathcal{G}$ a nuisance function, and $g_{0}\in\mathcal{G}$ the true nuisance function. Then, we have

R(\theta^{\prime},g_{0})-R(\theta,g_{0})\leq R(\theta^{\prime},g)-R(\theta,g)+\mathrm{err}^{\prime}(g,g_{0}),

where $\mathrm{err}^{\prime}(g,g_{0}):=C\|g-g_{0}\|_{L^{2}(P_{W})}^{2}$ if we assume $\ell$ is $C$ -smooth in $g$ , i.e. if we have

|D_{g}^{2}R(\varphi,h)[g-g_{0},g-g_{0}]|\leq C\|g-g_{0}\|_{L^{2}(P_{W})}^{2}\forall h\in[g,g_{0}],\varphi\in\Theta.

(8)

If we instead assume $g=(\eta,\zeta)$ and $\ell(\theta,g;z)=\frac{1}{2}(\theta(x)-\chi(g;z))^{2}$ where

\chi(g;z):=m(\eta;z)+\mathrm{Corr}(g;z),

$m(\eta;z)$ is linear in $\eta$ , and $\mathbb{E}[\mathrm{Corr}(g_{0};Z)\mid X]=\mathbb{E}[\langle\zeta(W),(\eta_{0}-\eta)(W)\rangle\mid X]$ , then we take

\mathrm{err}^{\prime}(g,g_{0}):=\|\langle\eta-\eta_{0},\zeta-\zeta_{0}\rangle\|_{L^{2}(P_{W})}^{2}.

Proof.

First, observe that we have the bound

R(\theta^{\prime},g_{0})-R(\theta,g_{0})=\underbrace{R(\theta^{\prime},g_{0})-R(\theta^{\prime},g)}_{T_{1}}+\underbrace{R(\theta^{\prime},g)-R(\theta,g)}_{T_{2}}+\underbrace{R(\theta,g)-R(\theta,g_{0})}_{T_{3}}.

Now, observe that by performing a second order Taylor expansion with respect to the nuisance component around $g_{0}$ , we have;

	$\displaystyle T_{1}$	$\displaystyle=-D_{g}R(\theta^{\prime},g_{0})(g-g_{0})+\frac{1}{2}D_{g}^{2}R(\theta^{\prime},\overline{g})(g-g_{0},g-g_{0})$
	$\displaystyle T_{3}$	$\displaystyle=D_{g}R(\theta,g_{0})(g-g_{0})-\frac{1}{2}D_{g}^{2}R(\theta,\overline{h})(g-g_{0},g-g_{0}),$

where $\overline{g},\overline{h}\in[g,g_{0}]$ , understood in a point-wise sense. Thus, we can bound the sum:

	$\displaystyle T_{1}+T_{3}$	$\displaystyle=\underbrace{D_{g}R(\theta,g_{0})(g-g_{0})-D_{g}R(\theta^{\prime},g_{0})(g-g_{0})}_{\text{first order difference}}$
		$\displaystyle\qquad+\underbrace{\frac{1}{2}D_{g}^{2}R(\theta^{\prime},\overline{g})(g-g_{0},g-g_{0})-\frac{1}{2}D_{g}^{2}R(\theta,\overline{h})(g-g_{0},g-g_{0})}_{\text{second order difference}}$

We can use universal orthogonality to show the first order difference vanishes. In particular, we have

	$\displaystyle D_{g}R(\theta,g_{0})(g-g_{0})-D_{g}R(\theta^{\prime},g_{0})(g-g_{0})$	$\displaystyle=\mathbb{E}\left[D_{g}\mathbb{E}(\ell(\theta,g_{0};Z)\mid X)(g-g_{0})-D_{g}\mathbb{E}(\ell(\theta^{\prime},g_{0};Z)\mid X)(g-g_{0})\right]$
		$\displaystyle=\mathbb{E}\left[D_{g}\mathbb{E}(\partial\ell(\overline{\theta},g_{0};Z)\mid X)(g-g_{0})(\theta-\theta^{\prime})(X)\right]$
		$\displaystyle=0$

where the second equality holds for some $\overline{\theta}\in[\theta,\theta^{\prime}]$ ¹²¹²12Understood in the sense that $\overline{\theta}(x)\in[\theta(x),\theta^{\prime}(x)]$ for each $x\in\mathcal{X}$ by performing a first-order Taylor expansion, and the final equality follows from universal orthogonality of $\ell$ .

If we are in the first setting, and assume that the Hessian of the loss $\ell$ has a maximum eigenvalue uniformly bounded (over $\theta$ and $g$ ) by $\beta$ , then we can bound the second order difference above as

\frac{1}{2}D_{g}^{2}R(\theta,\overline{h})(g-g_{0},g-g_{0})-\frac{1}{2}D_{g}^{2}R(\theta^{\prime},\overline{g})(g-g_{0},g-g_{0})\leq\beta^{2}\|g-g_{0}\|_{L^{2}(P_{W})}^{2}.

Else, if we instead assume the linear score condition outlined in Assumption 1 and further assume $m(\eta,z)$ is linear in $\eta$ , and thus have $\ell(\theta,g;z)=\frac{1}{2}(\theta(x)-\chi(g;z))^{2}$ for some pseudo-outcome mapping $\chi:\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ , then it is clear we have for any $h\in\mathcal{G}$

	$\displaystyle D_{\eta}^{2}\mathbb{E}[\ell(\theta,h;Z)\mid X](g-g_{0},g-g_{0})$	$\displaystyle=0$
	$\displaystyle D_{\zeta}^{2}\mathbb{E}[\ell(\theta,h;Z)\mid X](g-g_{0},g-g_{0})$	$\displaystyle=0$
	$\displaystyle D_{\zeta,\eta}\mathbb{E}[\ell(\theta,h;Z)\mid X](g-g_{0},g-g_{0})$	$\displaystyle=-\left(\mathbb{E}[\langle(\zeta-\zeta_{0})(W),(\eta-\eta_{0})(W)\rangle\mid X]\right)^{2}.$

Thus, by leveraging Jensen’s inequality, we can bound the second order difference by

\leq 2\mathbb{E}[\langle(\eta-\eta_{0})(W),(\zeta-\zeta_{0})(W)\rangle^{2}]=\mathrm{err}^{\prime}(g,g_{0}),

which completes the proof.

∎

As before, with the above error decomposition bound, we can now prove a general convergence result given access to nuisance estimation and calibration algorithms that satisfy certain, high-probability convergence guarantees. Given the similarity of this result to Theorems 3.5 and 4.8 and the fact Theorem C.1 does the majority of the heavy lifting , we omit the proof.

Theorem C.2.

Assume the same setup as Theorem C.1. Assume $\mathcal{A}_{1};\mathcal{Z}^{\ast}\rightarrow\mathcal{G}$ is a nuisance estimation algorithm and $\mathcal{A}_{2}:\Theta\times(\mathcal{X}\rightarrow\mathbb{R})^{\ast}\rightarrow\Theta$ is a general loss calibration algorithm. Assume

1.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $Z_{1},\dots,Z_{n}\sim_{i.i.d}P_{Z}$ , $\theta\in\Theta$ , and failure probability $\delta_{1}\in(0,1)$ , we have

$\mathrm{err}(\widehat{g},g_{0})\leq\mathrm{rate}_{3}(n,\delta_{1},\theta;P_{Z}),$

where $\widehat{g}\leftarrow\mathcal{A}_{1}(Z_{1:n})$ and $\mathrm{rate}_{3}$ is some rate function.
2.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $Z_{1},\dots,Z_{n}\sim_{i.i.d.}P_{Z}$ , $\theta\in\Theta$ , $g\in\mathrm{range}(\mathcal{A}_{1})$ , and failure probability $\delta_{2}\in(0,1)$ , we have

$R(\widehat{\theta},\widehat{g})-R(\theta,\widehat{g})\leq\mathrm{rate}_{4}(n,\delta_{2},\ell;P_{z}),$

where $\widehat{\theta}\leftarrow\mathcal{A}_{2}(\theta,\{(X_{m},\ell_{m})\}_{m=1}^{n})$ , $\ell_{m}:=\ell(\cdot,\widehat{g};Z_{m})$ , and $\mathrm{rate}_{4}$ is some rate function.

Then, with probability at least $1-\delta_{1}-\delta_{2}$ , we have

R(\widehat{\theta},g_{0})-R(\theta,g_{0})\leq\mathrm{rate}_{3}(n,\delta_{1},\theta;P_{Z})+\mathrm{rate}_{4}(n,\delta_{2},\ell;P_{Z}).

Appendix D Calibration Algorithms for General Losses

Here, we present analogues of classical calibration algorithms defined with respect to general losses involving nuisance components. In particular, we give generalizations of histogram binning/uniform mass binning, isotonic calibration, and linear calibration. We only prove convergence guarantees for one algorithm (uniform mass binning), but we experimentally showed in Section 5 that other algorithm (e.g. linear calibration) work well in practice.

We first present a general algorithm that computes a univariate post-processing over a suitable class of functions $\mathcal{F}\subset\{f:\mathbb{R}\rightarrow\mathbb{R}\}$ and then returns a calibrated parameter estimate.

Algorithm 5 General ERM-based Calibration

1:Samples

Z_{1},\dots,Z_{n}\sim P_{Z}

, losses

\ell_{1},\dots,\ell_{n}:=\ell(\cdot,g_{1};Z_{1}),\dots,\ell(\cdot,g_{n};Z_{n})

, estimator

\theta

, function class

\mathcal{F}

2:Compute

\widehat{\tau}\in\arg\min_{\tau\in\mathcal{F}}\sum_{m=1}^{n}\ell_{m}(\tau\circ\theta(X_{m}))

3:Define

\widehat{\theta}:=\widehat{\tau}\circ\theta

4:Calibrated estimator

\widehat{\theta}

Depending on how $\mathcal{F}$ is defined, when we obtain generalizations of many classical algorithms. We refer the reader to Table 2 in paper for appropriate choices of $\mathcal{F}$ . As noted in Section 4, our convergence guarantee required that we learn an injective post-processing map from model predictions to calibrated predictions. We discussed in detail how many common algorithms either naturally fit this desideratum or can be easily modified to fit this desideratum. The one exception to this is uniform mass/histogram binning, which maps model predictions to a finite number of values. To address this, we prove the convergence of a separate, three-way sample splitting algorithm that performs uniform mass binning for general losses.

D.1 An analogue of Algorithm 3 for UMB

We now develop a three-way sample splitting algorithm for causal calibration based on uniform mass/histogram binning [27, 25, 38]. As in the case of Algorithm 3, our algorithm (Algorithm 6 below) is implicitly based on the calibration error decomposition presented in Theorem 4.5. To prove the convergence of Algorithm 3, we needed to assume $\widehat{\theta}=\widehat{\tau}\circ\theta$ for some injective, randomized post-processing map $\widehat{\tau}:\mathbb{R}\rightarrow\mathbb{R}$ . While this desideratum is satisfied for some calibration algorithms (e.g. linear calibration and Platt scaling) and can be made to be satisfied for others (e.g. by slightly perturbing the post-processing map learned by isotonic calibration), it is clearly not a reasonable assumption for uniform mass binning/histogram binning. This is because, as the name suggest, binning algorithms “compress” the initial estimator into a pre-specified number of data-dependent bins.

Why did we assume $\widehat{\theta}$ was an injective post-processing of $\theta$ ? In any causal calibration algorithm, we need to estimate the unknown nuisance functions. For conditionally orthogonal losses, there are two parameters we need etimate. The first parameter, $\eta_{0}$ , does not depend on either the initial model $\theta(X)$ or the calibrated model $\widehat{\theta}(X)$ , and thus can be readily estimated from data. However, the second parameter, $\zeta_{\widehat{\theta}}$ , depends on the calibrated model $\widehat{\theta}$ , which in general cannot be reliably estimated from data. Assuming the injectivity of the post-processing map $\widehat{\tau}$ allowed us to conclude (via Lemma 4.7) that $\zeta_{\theta}=\zeta_{\widehat{\theta}}$ . Since we know $\theta$ at the outset of the problem, it is reasonable to assume we can produce a convergent estimate $\widehat{\zeta}$ of $\zeta_{\widehat{\theta}}$ .

The question is now what can we do for binning-based algorithms, which generally cannot be written as an injective post-processing of the initial model. The key idea is that, for histogram binning/uniform mass binning, the level sets of $\widehat{\theta}(X)$ only depend on the order statistics of $\theta(X_{1}),\dots,\theta(X_{n})$ , not on the nuisance parameters. Thus, we propose a natural three-way sample-splitting analogue of uniform mass binning for conditionally orthogonal losses $\ell$ .

Our algorithm works as follows. First, we use the covariates $X_{1},\dots,X_{n}$ associated with one third of the data to compute the buckets/bins of $\widehat{\theta}$ (these will be determined by evenly-spaced order statistics of $\theta(X_{1}),\dots,\theta(X_{n})$ ). Fixing the level sets up front like this in turn fixes the additional nuisance we need to estimate. This is because, under Lemma 4.7, any function $\varphi:\mathcal{X}\rightarrow\mathbb{R}$ with the same level sets as $\widehat{\theta}$ will have $\zeta_{\varphi}=\zeta_{\widehat{\theta}}$ . Then, we use the second third $Z_{n+1},\dots,Z_{2n}$ to estimate the unknown nuisance functions $\eta_{0}$ and $\zeta_{\widehat{\theta}}$ . Finally, we use the learned nuisance parameters and the final third $Z_{2n+1},\dots,Z_{3n}$ to compute the empirical loss minimizer in each bucket. We formally state this heuristically-described procedure below in Algorithm 6.

Algorithm 6 Three-Way Uniform Mass Binning (UMB)

1:Sample

Z_{1},\dots,Z_{3n}\sim_{i.i.d.}P_{Z}

, loss

\ell(\nu,g;z)

, estimator

\theta

, nuisance est. alg.

\mathcal{A}_{1}

, calibration alg.

\mathcal{A}_{2}

, number of buckets

B

2:Compute order statistics

\theta(X)_{(1)},\dots,\theta(X)_{(n)}

\theta(X_{1}),\dots,\theta(X_{n})

3:Set

\theta(X)_{(0)}=0,\theta(X)_{(n)}=\infty

4:for

b\in[B]

5: Set

V_{b}:=[\theta(X)_{\lfloor(b-1)N/B\rfloor},\theta(X)_{\lfloor bN/B\rfloor})

6:Define

\varphi(x):=\sum_{b=1}^{B}\nu_{b}\mathbbm{1}[\theta(x)\in V_{b}]

for any distinct

\nu_{1},\dots,\nu_{B}

7:Compute nuisances

\widehat{g}=(\widehat{\eta},\widehat{\zeta})\leftarrow\mathcal{A}_{1}(\{Z_{m}\}_{m=n+1}^{2n},\varphi)

8:for

b\in[B]

\widehat{\nu}_{b}:=\arg\min_{\nu}\sum_{m=2n+1}^{3n}\ell(\nu,\widehat{g};Z_{m})\mathbbm{1}[\theta(X_{m})\in V_{b}]

10:Set

\widehat{\theta}(x):=\sum_{b=1}^{B}\widehat{\nu}_{b}\mathbbm{1}[\theta(x)\in V_{b}]

11:Calibrated estimator

\widehat{\theta}

We now present the set of assumptions that we will use to prove convergence of the above algorithm.

Assumption 6.

Let $\mathcal{A}_{1}:\Theta\times\mathcal{Z}^{\ast}\rightarrow\mathcal{G}$ be a nuisance estimation algorithm and let $\varphi:\mathcal{X}\rightarrow\mathbb{R}$ be any initial estimator. We make the following assumptions.

1.

$\mathrm{range}(\theta)\subset[0,1]$ .
2.
The conditionally orthogonal loss function $\ell:\mathbb{R}\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ satisfies
1. (a)
  
  For any $(\eta,\zeta)\in\mathcal{G},z\in\mathcal{Z}$ , the minimizer of the loss $\nu^{\ast}=\arg\min_{\nu}\ell(\nu,g;z)$ satisfies $\nu^{\ast}\in[0,1]$ .
2. (b)
  
  For any $(\eta,\zeta)\in\mathcal{G},z\in\mathcal{Z}$ , and $\nu\in[0,1]$ , we have $\partial\ell(\nu,g;z)\in[-C,C]$ .
3.

For any distribution $P_{Z}$ on $\mathcal{Z}$ , $n\geq 1$ , and failure probability $\delta_{1}\in(0,1)$ , with probability at least $1-\delta_{1}$ over the draws of $Z_{1},\dots,Z_{n}\sim P_{Z}$ i.i.d., we have

$\mathrm{err}(\widehat{g},g_{\varphi};\gamma_{\varphi}^{\ast})\leq\mathrm{rate}_{1}(n,\delta_{1},\varphi;P_{Z}),$

where $\widehat{g}:=(\widehat{\eta},\widehat{\zeta})\leftarrow\mathcal{A}_{1}(\varphi,Z_{1:n})$ and $\mathrm{rate}_{1}$ is some rate function.

The above assumptions are mostly standard and essentially say the (a) the initial estimate $\theta(X)$ is bounded, (b) the partial derivative of the loss is bounded and the minimizer takes values in a bounded interval, and (c) we have access to some algorithm that can accurately estimate nuisance parameters.

For simplicity, we assume that the values $\widehat{\nu}_{1},\dots,\widehat{\nu}_{B}$ Algorithm 6 assigns to each of the buckets $V_{1},\dots,V_{B}$ are unique. This is to ensure two distinct buckets $V_{i}\neq V_{j}$ do not merge, which would invalidate our application of Lemma 4.7. If, in practice, we have $\widehat{\nu}_{i}=\widehat{\nu}_{j}$ for $i\neq j$ , the learner can simply add $\mathcal{U}([-\epsilon,\epsilon])$ noise to $\widehat{\nu}_{i}$ for $\epsilon>0$ arbitrarily small to guarantee uniqueness.

Assumption 7.

With probability 1 over the draws $Z_{1},\dots,Z_{3n}\sim P_{Z}$ i.i.d., we have that $\widehat{\nu}_{1},\dots,\widehat{\nu}_{B}\in[0,1]$ are distinct.

We now state the main result of this subsection, a technical convergence guarantee for Algorithm 6. We prove Theorem D.1 (along with requisite lemmas and propositions) the next subsection of this appendix.

Theorem D.1.

Fix any initial estimator $\theta:\mathcal{X}\rightarrow\mathbb{R}$ , conditionally orthogonal loss function $\ell:[0,1]\times\mathcal{G}\times\mathcal{Z}\rightarrow\mathbb{R}$ , and failure probabilities $\delta_{1},\delta_{2}\in(0,1)$ . Suppose Assumptions 3, 4, and 6 hold, and assume $n\gtrsim B\log(B/\min(\delta_{1},\delta_{2}))$ . Then, with probability at least $1-\delta_{1}-\delta_{2}$ over the randomness of $Z_{1},\dots,Z_{3n}\sim P_{Z}$ , the output $\widehat{\theta}$ of Algorithm 6 satisfies

\displaystyle\mathrm{Cal}(\widehat{\theta},\eta_{0})

\displaystyle\leq\frac{\beta}{2\alpha}\mathrm{rate}_{1}(n,\delta_{1},\widehat{\theta};P_{Z})+\frac{2\beta}{\alpha}\left(\frac{\beta}{n}+C\sqrt{\frac{2B\log(2nB/\delta_{2})}{n}}.\right),

where $C>0$ is some constant that bounds the partial derivative as discussed in Assumption 6: i.e. $|\partial\widetilde{\ell}(\nu,g;z)|<C$ .

D.2 Proving Theorem D.1

We now pivot to the task of proving Theorem D.1. We first cite a result that guarantees, with high probability, that close to a $1/B$ fraction of points will fall into each bucket.

Lemma D.2 ([38], Lemma 4.3; [27], Lemma 13).

For a universal constant $c>0$ , if $n\geq cB\log(B/\delta)$ , the bucket $\{V_{b}:b\in[B]\}$ produced in Algorithm 6 will satisfy

\frac{1}{2B}\leq\mathbb{P}_{X\sim P_{X}}(\theta(X)\in V_{b})\leq\frac{2}{B},

(9)

simultaneously for all $b\in[B]$ with probability at least $1-\delta$ over the randomness of $X_{1},\dots,X_{n}$

Next, we argue that, conditional on the random bins satisfying Lemma D.2, the population average derivative conditional on the observation falling into any given bucket will be close to the corresponding sample average.

Lemma D.3.

\frac{1}{2B}\leq\mathbb{P}(\theta(X)\in V_{i})\leq\frac{2}{B}

for every $b\in[B]$ , $n\geq 8B\log(B/\delta)$ , and Assumption 6 holds. Then, with probability at least $1-\delta$ over the randomness of $Z_{2n+1},\dots,Z_{3n}\sim P^{n}_{Z}$ , we have for all $b\in[B]$ and all $\nu\in[0,1]$

\left|\mathbb{E}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]\right|\leq\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(\frac{4nB}{\delta})}{n}}

where

\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]:=\frac{\sum_{m=2n+1}^{3n}\partial\ell(\nu,g;Z_{m})\cdot\mathbbm{1}[\theta(X_{m})\in V_{b}]}{\sum_{m=2n+1}^{3n}\mathbbm{1}[\theta(X_{m})\in V_{b}]}

denotes the empirical conditional mean over the calibration dataset $Z_{2n+1},\dots,Z_{3n}$ .

Proof.

For convenience, let

S_{b}:=\{2n+1\leq m\leq 3m:\theta(X_{m})\in V_{b}\}

to denote set of indices that fall in $V_{b}$ . Given that $n\geq 8B\log(B/\delta)$ , the multiplicative Chernoff bound (Lemma D.4) tells us that with probability $1-\delta$ ,

\displaystyle|S_{b}|\geq\frac{n}{4B}.

(10)

for all $i\in[B]$ .

Note that $\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]$ is the empirical mean over $|S_{b}|$ many points. Therefore, with inequality (10) and Hoeffding’s inequality (Lemma D.5), we have for any fixed $\nu\in[0,1]$ , with probability at least $1-\delta$ , simultaneously for all $b\in[B]$ ,

\displaystyle\left|\mathbb{E}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]\right|\leq 2C\sqrt{\frac{\log(4B/\delta)}{2|S_{b}|}}\leq 2C\sqrt{\frac{2B\log(4B/\delta)}{n}}.

Now, for some $\epsilon>0$ to be chosen later (which we will implicitly assume satisfies $1/\epsilon\in\mathbb{N}$ ), we now take a union bound over an $\epsilon$ -covering of $[0,1]$ : with probability $1-\delta$ , we have for all $\nu\in\{0,\epsilon,2\epsilon,\dots,1-\epsilon\}$ and $b\in[B]$ :

\displaystyle\left|\mathbb{E}[\partial\ell(\nu,g;Z_{m})\cdot\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu,g;Z_{m})\mid\theta(X)\in V_{b}]\right|\leq 2C\sqrt{\frac{2B\log(\frac{4B}{\epsilon\delta})}{n}}.

For any $\nu\not\in\{0,\epsilon,2\epsilon,\dots,1-\epsilon\}$ , taking its closest point $\nu_{\epsilon}$ in the $\epsilon$ -grid yields

	$\displaystyle\left\|\mathbb{E}[\partial\ell(\nu,g;Z)\cdot\mid\theta(X)\in V_{b}]-\mathbb{E}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|\leq\beta\epsilon$
	$\displaystyle\left\|\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|\leq\beta\epsilon$

as $\partial\ell$ is $\beta$ -Lipschitz by Assumption 4. Hence, with probability $1-\delta$ , we have for any $b\in[B]$ and $\nu\in[0,1]$ ,

	$\displaystyle\left\|\mathbb{E}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{b},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle\leq\left\|\mathbb{E}[\partial\ell(\nu,g;Z)\|\theta(X)\in V_{b}]-\mathbb{E}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle+\left\|\mathbb{E}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle+\left\|\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle\leq 2\beta\epsilon+2C\sqrt{\frac{\log(2B/\epsilon\delta)}{2n}}.$

Therefore, we have with probability $1-\delta$ ,

\left|\mathbb{E}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]\right|\leq 2\beta\epsilon+2C\sqrt{\frac{2B\log(\frac{4B}{\epsilon\delta})}{n}}.

for all $b\in[B]$ and $\nu\in[0,1]$ . Setting $\epsilon=\frac{1}{n}$ yields the result ∎

Lemma D.4 (Multiplicative Chernoff Boud).

Let $X_{1},\dots,X_{n}$ be independent random variables such that $X_{i}\in\{0,1\}$ and $\mathbb{E}[X_{i}]=p$ for all $i\in[n]$ . For all $t\in(0,1)$ ,

\mathbb{P}\left(\sum_{i=1}^{n}X_{i}\leq(1-t)np\right)\leq\exp\left(-\frac{npt^{2}}{3}\right).

Lemma D.5 (Hoeffding’s Inequality).

Let $X_{1},\dots,X_{n}$ be independent random variables such that $X_{i}\in[a,b]$ . Consider the sum of these random variables $S_{n}=X_{1}+\dots+X_{n}$ . For all $t>0$ ,

\mathbb{P}\left(|S_{n}-\mathbb{E}[S_{n}]|>t\right)\leq 2\exp\left(\frac{2t^{2}}{n(b-a)^{2}}\right).

We use Lemma D.3 to now prove the following Proposition.

Proposition D.6.

Assume the same setup as Lemma D.3 and suppose Assumption 6 holds. Let $Z_{1},\dots,Z_{n}\sim P_{Z}$ be i.i.d., let $g=(\eta,\zeta)\in\mathcal{G}$ be a fixed nuisance pair, and set $\widehat{\theta}:=\sum_{b=1}^{B}\widehat{\nu}_{b}\mathbbm{1}[\theta(x)\in V_{b}]$ , where

\widehat{\nu}_{b}=\arg\min_{\nu}\sum_{m=1}^{n}\ell(\nu,g;Z_{m})\mathbbm{1}[\theta(X_{m})\in V_{b}].

Assume the $\widehat{\nu}_{b}$ are distinct almost surely. Then, for any $\delta\in(0,1)$ , we have with probability at least $1-\delta$ ,

\mathrm{Cal}(\widehat{\theta},g)\leq\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(4nB/\delta)}{n}}

where $C>0$ is some constant that bounds the partial derivative as discussed in Assumption 6: i.e. $|\partial\ell(\nu,g;z)|<C$ .

Proof.

As we have assumed that all $\widehat{\nu}_{b}$ ’s are all distinct, we have

\displaystyle\mathrm{Cal}(\widehat{\theta},g)

\displaystyle=\sqrt{\sum_{b\in[B]}\mathbb{P}(\widehat{\theta}(X)=\widehat{\nu}_{b})\cdot\mathbb{E}\left[\partial\ell(\widehat{\nu}_{b},g;Z)\mid\widehat{\theta}(X)=\nu_{b}\right]^{2}}

Since we have assumed $n\geq 8B\log(B/\delta)$ , we have from Lemma D.3 that, with probability at least $1-\delta$ , simultaneously for each $b\in[B]$

\displaystyle\mathbb{E}[\partial\ell(\widehat{\nu}_{b},g;Z)\mid\widehat{\theta}(X)=\widehat{\nu}_{b}]=\mathbb{E}[\partial\ell(\widehat{\nu}_{b},g;Z)\mid\theta(X)\in V_{b}]\leq\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(\frac{4nB}{\delta})}{n}},

which follows since $\mathbb{E}_{n}[\partial\ell(\widehat{\nu}_{b},g;Z)\mid\theta(X)\in V_{b}]=0$ by definition of $\widehat{\nu}_{b}$ . Thus, with probability at least $1-\delta$ , we get

	$\displaystyle\mathrm{Cal}(\widehat{\theta},g)$	$\displaystyle=\sqrt{\sum_{b\in[B]}\mathbb{P}(\theta(X)\in V_{b})\cdot\mathbb{E}\left[\partial\ell(\widehat{\nu}_{b},g;Z)\mid\widehat{\theta}(X)=\widehat{\nu}_{b}\right]^{2}}$
		$\displaystyle\leq\sqrt{\sum_{b\in[B]}\mathbb{P}(\theta(X)\in V_{b})\cdot\left(\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(\frac{4nB}{\delta})}{n}}\right)^{2}}$
		$\displaystyle=\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(\frac{4nB}{\delta})}{n}}.$

∎

We now have the requisite tools to prove Theorem D.1. Our argument proceeds in largely the same way that the proof of Theorems 3.5 and 4.8 — we start by defining appropriate “good” events, and then subsequently bound the overall probability of their failure.

Proof of Theorem D.1.

As before we start by defining some “bad” events. In particular, consider the events $B_{1}$ and $B_{2}$ defined respectively by

B_{1}:=\left\{\mathrm{err}(\widehat{g},g_{\widehat{\theta}};\gamma_{\widehat{\theta}}^{\ast})>\mathrm{rate}_{1}(n,\delta_{1},\widehat{\theta};P_{Z})\right\}\quad\text{and}\quad B_{2}:=\left\{\mathrm{Cal}(\widehat{\theta},\widehat{g})>\frac{2\beta}{n}+2C\sqrt{\frac{2B\log(8nB/\delta_{2})}{n}}\right\}

where $\widehat{g}=(\widehat{\eta},\widehat{\zeta})\leftarrow\mathcal{A}_{1}(\varphi,Z_{n+1:2n})$ , where $\varphi$ is as defined in Algorithm 6. It is clear that $\mathbb{P}(B_{1}\mid Z_{1:n})\leq\delta_{1}$ by Assumption 6, since fixing $Z_{1:n}$ fixes the estimate $\varphi$ . Thus, the law of total expectation yields that $\mathbb{P}(B_{1})\leq\delta_{1}$ .

Bounding $\mathbb{P}(B_{2})$ takes mildly more care. Define the event

E:=\left\{\forall b\in[B],\frac{1}{2B}\leq\mathbb{P}(\theta(X)\in V_{b})\leq\frac{2}{B}\right\},

which by Lemma D.2 occurs with probability at least $1-\delta_{2}/2$ by the assumption that $n\geq cB\log(2B/\delta_{2})$ . We have the following bound:

	$\displaystyle\mathbb{P}(B_{2})$	$\displaystyle=\mathbb{P}(B_{2}\mid E)\mathbb{P}(E)+\mathbb{P}(B_{2}\mid E^{c})\mathbb{P}(E^{c})$
		$\displaystyle\leq\mathbb{P}(B_{2}\mid E)+\mathbb{P}(E^{c})$
		$\displaystyle=\mathbb{E}\left[\mathbb{P}(B_{2}\mid E,Z_{1:2n})\mid E\right]+\mathbb{P}(E^{c})$
		$\displaystyle\leq\delta_{2}/2+\delta_{2}/2=\delta_{2},$

where the second to last inequality follows form the fact that $\mathbb{P}(B_{2}\mid E,Z_{1:2n})\leq\delta_{2}$ by Proposition D.6. We now apply Theorem 4.5 to see that, on the “good” event $G=B_{1}^{c}\cap B_{2}^{c}$ (which occurs with probability at least $1-\delta_{1}-\delta_{2}$ ) we have

	$\displaystyle\mathrm{Cal}(\widehat{\theta},\eta_{0})$	$\displaystyle\leq\frac{\beta}{2\alpha}\mathrm{err}(\widehat{g},g_{\widehat{\theta}};\gamma_{\widehat{\theta}}^{\ast})+\frac{\beta}{\alpha}\mathrm{Cal}(\widehat{\theta},\widehat{g})$
		$\displaystyle\leq\frac{\beta}{2\alpha}\mathrm{rate}_{1}(n,\delta_{1},\widehat{\theta};P_{Z})+\frac{2\beta}{\alpha}\left(\frac{\beta}{n}+C\sqrt{\frac{2B\log(8nB/\delta_{2})}{n}}\right),$

which proves the desired result. ∎

Appendix E Additional Proofs

In this appendix, we proofs of additional claims that do not constitute our primary results. We proceed by section of the paper. See 3.4

Proof of Proposition 3.4.

Let $\overline{g}=(\overline{\eta},\overline{\zeta})\in[g,g_{0}]$ . Observe that we have,

	$\displaystyle D_{g}^{2}\mathbb{E}[\partial\ell(\theta,\overline{g};Z)\mid X](g-g_{0},g-g_{0})$
	$\displaystyle=D_{g}^{2}\mathbb{E}[\theta(X)-m(\overline{\eta};Z)\mid X](g-g_{0},g-g_{0})$
	$\displaystyle\qquad-D_{g}^{2}\mathbb{E}[\langle\overline{\zeta}(W),\overline{\eta}(W)-\eta_{0}(W))\mid X](g-g_{0},g-g_{0})$	(Assumption 1)
	$\displaystyle=D_{g}^{2}\mathbb{E}[\langle\overline{\zeta}(W),(\overline{\eta}-\eta_{0})(W)\rangle\mid X](g-g_{0},g-g_{0})$	$\displaystyle\text{(Linearity of $m(\eta;z)$)}.$

Now, we compute the Hessian that appears on the second line. In particular, further calculation yields that

	$\displaystyle D_{g}^{2}\mathbb{E}[\langle\overline{\zeta}(W),\overline{\eta}(W)-\eta_{0}(W)\rangle\mid X](g-g_{0},g-g_{0})$
	$\displaystyle\qquad=\mathbb{E}\left[\left(\eta(W)-\eta_{0}(W),\zeta(W)-\zeta_{0}(W)\right)^{\top}\begin{pmatrix}0&-1\\ -1&0\end{pmatrix}\left(\eta(W)-\eta_{0}(W),\zeta(W)-\zeta_{0}(W)\right)\mid X\right]$

Thus, we can write the error term down as

	$\displaystyle\mathrm{err}(g,g_{0};\theta)^{2}:=\sup_{f\in[g,g_{0}]}\mathbb{E}\left(\left\{D_{g}^{2}\mathbb{E}\left[\partial\ell(\theta,f;Z)\mid X\right](g-g_{0},g-g_{0})\right\}^{2}\right)$
	$\displaystyle\qquad=\mathbb{E}\left(\mathbb{E}\left[((\eta-\eta_{0})(W),(\zeta-\zeta_{0})(W))^{\top}\begin{pmatrix}0&-1\\ -1&0\end{pmatrix}((\eta-\eta_{0})(W),(\zeta-\zeta_{0})(W))\mid X\right]^{2}\right)$
	$\displaystyle\qquad=4\mathbb{E}\left(\mathbb{E}\left[\langle\eta-\eta_{0})(W),\zeta-\zeta_{0})(W)\rangle\mid X\right]^{2}\right)$
	$\displaystyle\qquad\leq 4\\|(g-g_{0})(b-b_{\theta})\\|_{L^{2}(P_{W})}^{2}\qquad\qquad\qquad\text{(Jensen's Inequality and Tower Rule)}.$

Thus, taking square roots, we see that we have

\mathrm{err}(g,g_{0})\leq 2\|\langle\eta-\eta_{0},\zeta-\zeta_{0}\rangle\|_{L^{2}(P_{W})}.

Dividing both sides by two yields the claimed result. The second claim is trivial and follows from an application of Jensen’s inequality.

∎

See 4.7

Proof of Lemma 4.7.

The first claim is straightforward, as we have

\displaystyle\mathbb{E}[\ell(\nu,g;Z)\mid\varphi_{1}(X)]=\mathbb{E}[\ell(\nu,g;Z)\mid\varphi_{2}(X)]

for all $\nu\in\mathbb{R}$ , $g=(\eta,\zeta)\in\mathcal{G}$ , so we just plug in $g_{0}=(\eta_{0},\zeta)$ for any $\zeta$ . The equivalence of calibration functions, in particular, implies we have

0=D_{g}\mathbb{E}[\partial\ell(\gamma_{\varphi_{c_{2}}}^{\ast},(\eta,\zeta_{\varphi_{c_{1}}});Z)\mid\varphi_{c_{1}}(X)](g-g_{0})=D_{g}\mathbb{E}[\partial\ell(\gamma_{\varphi_{c_{2}}}^{\ast},(\eta_{0},\zeta_{\varphi_{c_{1}}});Z)\mid\varphi_{c_{2}}(X)](g-g_{0})

for any direction $g\in\mathcal{G}$ and choice of $c_{1},c_{2}\in\{1,2\}$ . That is, $\ell$ satisfies the second point of Definition 4.3 when either $\zeta_{\varphi_{1}}$ or $\zeta_{\varphi_{2}}$ are plugged in. Thus, without loss of generality, we can assume they are the same. ∎

	$\displaystyle\left\|\mathbb{E}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{b},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle\leq\left\|\mathbb{E}[\partial\ell(\nu,g;Z)\|\theta(X)\in V_{b}]-\mathbb{E}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle+\left\|\mathbb{E}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle+\left\|\mathbb{E}_{n}[\partial\ell(\nu,g;Z)\mid\theta(X)\in V_{b}]-\mathbb{E}_{n}[\partial\ell(\nu_{\epsilon},g;Z)\mid\theta(X)\in V_{b}]\right\|$
	$\displaystyle\leq 2\beta\epsilon+2C\sqrt{\frac{\log(2B/\epsilon\delta)}{2n}}.$