Flexible risk design using bi-directional dispersion

To start, let us clarify the nature of the random losses we consider. With the context of the underlying probability space $(\Omega,\mathcal{F},\mu)$, we write $\operatorname{\mathsf{L}}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{\mathsf{L}}(h;\cdot):\Omega\to\mathbb{R}$ to refer to a random variable (i.e., a $\mathcal{F}$-measurable function) on $\Omega$, though we only use the form $\operatorname{\mathsf{L}}(h)$ in the body of this paper. When we talk about “sampling” losses or a “random draw” of the losses, this amounts to computing a realization $\operatorname{\mathsf{L}}(h;\omega)\in\mathbb{R}$. We use standard notation for taking expectation, e.g., $\operatorname{\mathbf{E}}_{\mu}\operatorname{\mathsf{L}}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\int_{\Omega}\operatorname{\mathsf{L}}(h;\omega)\,\mu(\mathop{}\!\mathrm{d}\omega)$. These conventions extend to random quantities based on the losses (e.g., the gradient ${\operatorname{\mathsf{L}}}^{\prime}(h)$ considered in §4.1). Similarly, we will use $\operatorname{\mathbf{P}}$ as a general-purpose probability function, representing both $\mu$ and product measures; when the source of randomness is not immediate from the context, it will be stated explicitly. We will use $\operatorname{R}(\cdot)$ as a generic symbol for risk functions (often modified with subscripts), with the understanding that $\operatorname{R}(\cdot)$ maps random losses $\operatorname{\mathsf{L}}(h)$ to real values. We will overload this notation, writing $\operatorname{R}(\operatorname{\mathsf{L}})$ when the role of $h$ is unimportant, and writing $\operatorname{R}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{R}(\operatorname{\mathsf{L}}(h))$ when we want to emphasize the dependence on $h$. This convention will be applied to other quantities as well, such as writing $\operatorname{D}_{\rho}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{D}_{\rho}(\operatorname{\mathsf{L}}(h))\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{\mathbf{E}}_{\mu}\rho(\operatorname{\mathsf{L}}(h)-\theta)$ for the expected dispersion induced by $\rho$, first defined in (22).

We will use $\lVert{\cdot}\rVert$ as a general-purpose notation for all norms that appear in this paper. That is, we do not use different notation to distinguish different norm spaces. The reason for this is that we will never consider two distinct norms on the same set; each norm is associated with a distinct set, and thus as long as it is clear which set a particular element belongs to, there should be no confusion. The only exception to this rule is the special case of $\mathbb{R}$, in which we write $\lvert{\cdot}\rvert$ for the absolute value, as is traditional. For a function $f:\mathbb{R}\to\mathbb{R}$ in one variable, we use ${f}^{\prime}$ to denote the usual derivative. More general notions (e.g., Gateaux or Fréchet differentials) only make an appearance in §4, and the generality they afford us is not crucial to the main narrative, so the details can be easily skipped over if the reader is unfamiliar with such concepts. All the other undefined notation we use is essentially standard, and can be found in most introductory analysis textbooks.

For a function $f:\mathbb{R}\to\mathbb{R}$ in one variable, we use ${f}^{\prime}$ to denote the usual derivative. More general notions (e.g., Gateaux or Fréchet differentials) only make an appearance in §4, and the generality they afford us is not crucial to the main narrative, so the details can be easily skipped over if the reader is unfamiliar with such concepts. All the other undefined notation we use is essentially standard, and can be found in most introductory analysis textbooks. Particularly in formal proofs, we will frequently make use of the notation $\rho_{\sigma}(x)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\rho(x/\sigma)$, where $\sigma>0$ is a scaling parameter.

As a computationally convenient way to interpolate between the mean and the extreme values of $\operatorname{\mathsf{L}}(h)$, the tilted risk [37, 38] is a natural choice, defined for $\gamma\neq 0$ as

$\displaystyle\operatorname{R}_{\textup{tilt}}(h;\gamma)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\frac{1}{\gamma}\log\left(\operatorname{\mathbf{E}}_{\mu}\mathrm{e}^{\gamma\operatorname{\mathsf{L}}(h)}\right).$

(3)

This is simply a re-scaling of the cumulant distribution function of $\operatorname{\mathsf{L}}(h)$, viewed as a function of $h$, where taking $\gamma\to\infty$ and $\gamma\to-\infty$ lets us approach the supremum and infimum of $\operatorname{\mathsf{L}}(h)$, respectively. Another important class of risk functions is based upon the conditional value-at-risk (CVaR) [46], defined for $0\leq\beta<1$ as

$\displaystyle\operatorname{R}_{\textup{CVaR}}(h;\beta)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{\mathbf{E}}_{\mu}\left[\operatorname{\mathsf{L}}(h)\,|\,\operatorname{\mathsf{L}}(h)\geq\operatorname{Q}_{\beta}(h)\right].$

(4)

This is the expected loss at $h$, conditioned on the event that the loss exceeds the $\beta$-quantile of $\operatorname{\mathsf{L}}(h)$, denoted here by $\operatorname{Q}_{\beta}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\inf\left\{x\in\mathbb{R}:\operatorname{\mathbf{P}}\{\operatorname{\mathsf{L}}(h)\leq x\}\geq\beta\right\}$. Both of these risk functions can be re-written in a form similar to that of (2), namely

$\displaystyle h\mapsto\inf_{\theta\in\mathbb{R}}\left[\theta+\operatorname{\mathbf{E}}_{\mu}\phi(\operatorname{\mathsf{L}}(h)-\theta)\right]$

(5)

where $\phi(x)=(\mathrm{e}^{\gamma x}-1)/\gamma$ yields $\operatorname{R}_{\textup{tilt}}$ (basic calculus), and $\phi(x)=\max\{0,x\}/(1-\beta)$ yields $\operatorname{R}_{\textup{CVaR}}$ (see Rockafellar and Uryasev, [46, 47]). When $\phi:\mathbb{R}\to\mathbb{R}$ is restricted to be a non-decreasing, closed, convex function which satisfies both $\phi(0)=0$ and $1\in\partial\phi(0)$, the mapping given in (5) is called an optimized certainty equivalent (OCE) risk [7, 8, 36]. The class of OCE risks strictly generalizes the expected value (noting $\phi(x)=x$ is valid), and includes $\operatorname{R}_{\textup{tilt}}(\cdot;\gamma)$ when $\gamma>0$, as well as $\operatorname{R}_{\textup{CVaR}}$. The mean-variance is sometimes stated to be an OCE risk [36, Table 1], but this fails to hold when losses are unbounded above and below.

Another important class of risk functions is that of robustly regularized risks which are designed to ensure that risk minimizers are robust to a certain degree of divergence in the underlying data model. Making this concrete, it is typical to assume the random losses are the outputs of a loss function $\ell$ depending on the candidate $h$ and some random data $\mathsf{Z}$, i.e., $\operatorname{\mathsf{L}}(h)=\ell(h;\mathsf{Z})$, with $\mathsf{Z}\sim\mu$ as our reference model. To measure divergence from this reference model, it is convenient to use the Cressie-Read family of divergence functions [17, 60], namely for any $c>1$ and assuming $\nu\ll\mu$ (absolutely continuity) holds, functions of the form

$\displaystyle\operatorname{Div}_{c}(\nu;\mu)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\operatorname{\mathbf{E}}_{\mu}f_{c}\left(\frac{\mathop{}\!\mathrm{d}\nu}{\mathop{}\!\mathrm{d}\mu}\right),\text{ where }f_{c}(x)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\frac{x^{c}-cx+c-1}{c(c-1)}$

(6)

and $\mathop{}\!\mathrm{d}\nu/\mathop{}\!\mathrm{d}\mu$ is the Radon-Nikodym density of $\nu$ with respect to $\mu$.²²2For background on absolute continuity and density functions, see Ash and Doléans-Dade, [3, §2.2]. The resulting robustly regularized risk, called the DRO risk, is defined as

$\displaystyle\operatorname{R}_{\textup{DRO}}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\sup\left\{\operatorname{\mathbf{E}}\operatorname{\mathsf{L}}(h):\operatorname{\mathsf{L}}\in\mathcal{L}\right\}$

(7)

where the constrained set of random losses $\mathcal{L}$, determined by $c>1$ and $a>0$, is defined as

$\displaystyle\mathcal{L}\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\left\{\operatorname{\mathsf{L}}(\cdot)=\ell(\cdot;\mathsf{Z}):\mathsf{Z}\sim\nu\text{ and }\operatorname{Div}_{c}(\nu;\mu)\leq a\right\}.$

(8)

For this particular family of divergences, the risk can be characterized as the optimal value of a simple optimization problem [17], namely we have that

$\displaystyle\operatorname{R}_{\textup{DRO}}(h)=\inf_{\theta\in\mathbb{R}}\left[\theta+\left(1+c(c-1)a\right)^{1/c}\left(\operatorname{\mathbf{E}}_{\mu}\left(\operatorname{\mathsf{L}}(h)-\theta\right)_{+}^{c_{\ast}}\right)^{1/c_{\ast}}\right]$

(9)

where $c_{\ast}\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=c/(c-1)$. While strictly speaking this is not an OCE risk, note that if we set $\phi(x)=(1+c(c-1)a)^{c_{\ast}/c}(x)_{+}^{c_{\ast}}$, then the DRO risk can be written as

$\displaystyle\operatorname{R}_{\textup{DRO}}(h)=\inf_{\theta\in\mathbb{R}}\left[\theta+[\operatorname{\mathbf{E}}_{\mu}\phi(\operatorname{\mathsf{L}}(h)-\theta)]^{1/c_{\ast}}\right],$

(10)

giving us an expression of this risk as the sum of a threshold and an asymmetric dispersion. When we set $c=2$, this yields the well-known special case of $\chi^{2}$-DRO risk [26, 60]. In addition to the one-directional nature of the dispersion term in these risks, all of these risks are at least as sensitive to loss tails (on the upside) as the classical expected loss $\operatorname{\mathbf{E}}_{\mu}\operatorname{\mathsf{L}}(h)$ is; this holds for CVaR (with $\beta>0$), tilted risk (with $\gamma>0$), and even robust variants of DRO risk [60].

While it is clear that the form of the preceding risk classes given in (5) and (10) based on various choices of $\phi(\cdot)$ is the same as our $\rho(\cdot)$-based risk of interest in (2), they are fundamentally different in that none of the choices of $\phi(\cdot)$ induce a meaningful M-location; since all these $\phi(\cdot)$ are monotonic on $\mathbb{R}$, both minimization and maximization of $\theta\mapsto\operatorname{\mathbf{E}}_{\mu}\phi(\operatorname{\mathsf{L}}(h)-\theta)$ is trivially accomplished by taking $\lvert{\theta}\rvert\to\infty$. In stark contrast, $\rho(\cdot)$ is assumed to be such that the solution set in (1) is a subset of the real line. We will introduce a concrete and flexible class from which $\rho$ will be taken in §3, and in Figure 1 give a side-by-side comparison with the $\phi$ functions discussed in the preceding paragraphs.

The form given in (11) is very general, but it can be understood as a straightforward generalization of the convex objective used to characterize quantiles. More precisely, taking $\beta\in(0,1)$ and denoting the $\beta$-quantile of the base loss using $\operatorname{Q}_{\beta}(h)\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=\inf\left\{x\in\mathbb{R}:\operatorname{\mathbf{P}}\{\operatorname{\mathsf{L}}(h)\leq x\}\geq\beta\right\}$, it is well-known that in the special case of $\rho(\cdot)=\lvert{\cdot}\rvert$, we have

$\displaystyle\operatorname{Q}_{\beta}(h)\in\theta_{\rho}(h;\theta,1-2\beta)$

(14)

for any choice of $0<\beta<1$, as long as $\operatorname{\mathbf{E}}_{\mu}\lvert{\operatorname{\mathsf{L}}(h)}\rvert$ is finite [35]. The T-risk in (11) simply allows for a more flexible choice of $\rho$, and thus generalizes the dispersion term in this objective function.

We generate random values to simulate loss distributions, and evaluate how the values returned by each risk function change as we modify their respective parameters. Letting $\operatorname{\mathsf{L}}$ denote the random base loss we are simulating, we specify a parametric distribution for $\operatorname{\mathsf{L}}$, from which we take an independent sample $\{\operatorname{\mathsf{L}}_{1},\ldots,\operatorname{\mathsf{L}}_{m}\}$. In all cases, we center the true distribution such that $\operatorname{\mathbf{E}}_{\mu}\operatorname{\mathsf{L}}=0$. We use this common sample to compare the values returned by each of the aforementioned risks, as well as the optimal choice of threshold parameter $\theta$. To ensure that key trends are consistent across samples, we take a large sample size of $m=10^{4}$. For T-risk, we adjust $\alpha$ and $\eta$, and for M-location, just $\alpha$. In both cases, we leave $\sigma=0.5$ fixed. For CVaR, we modify the quantile level $0<\beta<1$. For tilted risk, we modify the parameter $\gamma\in\mathbb{R}$. For $\chi^{2}$-DRO risk, we modify $0<\widetilde{a}<1$, having re-parameterized $a$ in (8) by $a\mathrel{\vbox{\hbox{\scriptsize.}\hbox{\scriptsize.}}}=((1-\widetilde{a})^{-1}-1)^{2}/2$, as is common practice [60].

An illustrative example is given in Figure 2, where we look at how each risk class behaves under a centered asymmetric distribution, before and after flipping it (i.e., under $\operatorname{\mathsf{L}}$ and −$\operatorname{\mathsf{L}}$). Starting from the two left-most plots, we show $\underline{\operatorname{R}}_{\rho}(\operatorname{\mathsf{L}};\eta)$ (dashed curves) and $\theta_{\rho}(\operatorname{\mathsf{L}};\eta)$ (solid curves) from (12) as a function of $\alpha$, coloring the area between these graphs in gray. The first plot corresponds to $\eta=1$, the second to $\eta=-1$. Similarly for the M-location we plot $\operatorname{M}_{\rho}(\operatorname{\mathsf{L}})$ (solid) and $\operatorname{M}_{\rho}(\operatorname{\mathsf{L}})+\operatorname{\mathbf{E}}_{\mu}\rho(\operatorname{\mathsf{L}}-\operatorname{M}_{\rho}(\operatorname{\mathsf{L}}))$ (dashed). Analogous values are plotted for each of the other classes; note that for the tilted risk (3) with $\gamma>0$, the optimal threshold and the risk value are in fact the same value (see §B.1). The right-most plot is a histogram of the random sample $\{\operatorname{\mathsf{L}}_{1},\ldots,\operatorname{\mathsf{L}}_{m}\}$, here from a centered log-Normal distribution. All plots share a common vertical axis, and horizontal rules are drawn at the median (red, solid) and at the mean (gray, dotted; always zero due to centering). The critical point to emphasize here is how all the OCE and DRO risks here are highly asymmetric in terms of their tail sensitivity, in stark contrast with both the M-location and the T-risk. Turning tail sensitivity high enough in each of these classes (e.g., $\alpha>1.5$, $\beta>0.5$, $\gamma>1.0$, $\widetilde{a}>0.5$), note how flipping the distribution tails from the upside (top row) to the downside (bottom row) leads to a dramatic decrease in all risks but the T-risk and M-location. Finally, note how the T-risk thresholds $\theta_{\rho}(\operatorname{\mathsf{L}};\eta)$ close in on the M-location $\operatorname{M}_{\rho}(\operatorname{\mathsf{L}})$ from above/below depending on whether $\eta$ is negative/positive. Results for numerous distributions are available in our online repository (cf. §1).

An obvious limitation of our analysis is that we have assumed that each $h_{t}\in\mathcal{H}$; this matches the setup of [14], but the condition that $\operatorname{\mathbf{E}}_{\mu}\lVert{{\operatorname{\mathsf{L}}}^{\prime}}\rVert_{\mathcal{H}}^{2}<\infty$ will sometimes require $\mathcal{H}$ to have a finite diameter. Modifying the procedure to allow for projection of the iterates to $\mathcal{H}$ is a point of technical interest, but is out of this paper’s scope. Another limitation is that our current approach using smoothness properties (Lemma 12) necessitates an assumption of gradients with finite second-order moments. In the special case where $\alpha=1$, arguments based on smoothness can be replaced with arguments based on weak convexity [15, 30]; but this fails for more general $\alpha$ since $\rho(\cdot;\alpha)$ is not convex for $\alpha<1$, and not Lipschitz for $\alpha>1$. Another potential option is to split the sample, leverage stronger guarantees available in expectation for gradient descent run on the subsets, and robustly choose the best candidate based on a validation set of data [29].

As an initial example using simulated data, we design a binary classification problem by generating 500 labeled data points as shown in Figure 3 (left-most plot). The majority class comprises 95% of the sample, and we flip 5% of the labels uniformly at random. Aside from the flipped labels, the data is linearly separable. We consider two candidate classifiers to initialize an iterative learning algorithm: one candidate that does well on the majority class (dotted purple), and one candidate that does well on the minority class (dashed pink). Due to class imbalance, the loss distributions incurred by these two candidates are highly asymmetric and differ in their long tail direction (Figure 3, remaining plots). We run empirical risk minimization for each risk class described in §2.2 (joint in $(h,\theta)$), implemented by 15,000 iterations of full-batch gradient descent, using a step size of $0.01$, and three different choices of base loss function (logistic, hinge, unhinged). We run this procedure on the same data for a wide range of risk parameters $\alpha$ (T-risk), $\beta$ (CVaR), $\gamma$ (tilted), and $a$ ($\chi^{2}$-DRO), and choose a representative setting for each risk class as the one achieving the best final training classification error.

The full-batch (training) classification error and norm trajectories for these representatives initialized at each of the two candidates just mentioned (“mostly correct” and “mostly incorrect”) and trained using the unhinged loss as a base loss are shown in Figure 4. Regardless of the direction of the initial loss distribution tails, we see that using the Barron-type T-risk has the flexibility to achieve both a stable and superior long-term error rate, while at the same time penalizing exceedingly overconfident correct examples (via bi-directionality), and thereby keeping the norm of the linear classifier small. An almost identical trend is also observed under the (binary) logistic loss, whereas classification error rates under the hinge loss tend to be less stable (see Figures 10–11 in §B.6).

For a crystal-clear example of real data that induces losses with heavy tails, we consider the famous Belgian phone call dataset [51, 59] with input features normalized to the unit interval, and raw outputs used as-is. We conduct two regression tasks: the first uses the original data just stated, and the second has us modify such that it is an outlier on both the horizontal and vertical axes (we just multiply the original data point by 5); such points are said to have “high leverage” [32, Ch. 7]. For these two tasks, once again we run empirical risk minimization under each of the risk classes of interest, implemented using 15,000 iterations of full-batch gradient descent, with fixed step size 0.005. To illustrate the flexibility of the T-risk, here instead of minimizing (11) jointly in $(h,\theta)$, we fix $\theta$ at the start of the learning process along with $\sigma$ and $\eta$, iteratively optimizing $h$ only (i.e., $\theta$ is not updated at any point). For simplicity, we set $\theta$ and $\sigma$ both to be the median of the losses incurred at initialization. All other risk classes are precisely as in the previous experiment.

The final regression lines obtained under each risk setting by running the learning procedure just described are plotted along with the data in Figure 5. Colors correspond to the individual risk function choices within each risk family, as denoted by color bars under each plot. The gray regression line denotes the common initial value used by all methods. Algorithm outputs using CVaR and $\chi^{2}$-DRO are always at least as sensitive to outliers as the ordinary least-squares (OLS) solution. While the tilted risk does let us interpolate between lower and upper quantiles, this transition is not smooth; even trying 20 values within the small window of $\gamma\in[-0.025,0.025]$, the algorithm outputs essentially jump between two extremes. This difference is particularly lucid in the bottom plots of Figure 5, where we have a high-leverage point. In contrast, with all other parameters of T-risk fixed, note that just tweaking $\alpha$ gives us a remarkable degree of flexibility to control the final output; while the base loss is fixed to the squared error, the regression lines range from those that ignore outliers to those that are very sensitive to outliers. In the high-leverage case, it is well-known that this cannot be achieved by simply changing to a different convex base loss (e.g., MAE instead of OLS), giving a concise illustration of the flexibility inherent in the T-risk class. Algorithm behavior can be sensitive to initial value; see §B.5 and Figure 13 in the appendix for an example where the naive median-based procedure described here causes gradient-based algorithms to stall out.

Finally we consider tests using datasets and models which are orders of magnitude larger than the previous two experimental setups. We use several well-known benchmark datasets for multi-class classification; details are given in §B.5. All features are normalized to the unit interval (categorical variables are one-hot), and scores for each class are computed by a linear combination of features. As a base loss, we use the multi-class logistic regression loss. Here we investigate how a stochastic gradient descent implementation (with averaging) of the empirical T-risk minimization (joint in $(h,\theta)$) behaves as we control the shape parameter $\alpha$ of the Barron class, with $\sigma=0.99$ and $\eta=1$ fixed throughout. We record the base loss (logistic loss) and zero-one loss (misclassification error) at the start and end of each epoch. We also record the full base loss distribution after the last epoch concludes. We run 5 independent trials, in which the data is shuffled and initial parameters are determined randomly. In each trial, for all datasets and methods, we use a mini-batch size of 32, and we run 30 epochs. As reference algorithms, we consider “vanilla” ERM, using the traditional risk $\operatorname{\mathbf{E}}_{\mu}\operatorname{\mathsf{L}}(h)$, and mean-variance implemented as a special case of T-risk (with $\rho(\cdot)=(\cdot)^{2}/2$ and $\eta=1$). 80% of the data is used for training, 10% for validation, and 10% for testing. For each risk class, we try five different step sizes. Validation data is used to evaluate different step sizes and choose the best one for each risk setting. All the results we present here are based on loss values computed on the test set: solid lines represent averages taken over trials, and shaded areas denote $\pm$ standard deviation over trials.

In Figures 6–7, we give results based on the following two datasets: “extended MNIST” (47 classes, balanced) [11], and “cover type” (7 classes, imbalanced) [9]. We plot the average and standard deviation of the base and zero-one losses as a function of epoch number, plus give a histogram of test (base) losses for a single trial, compared with the loss distribution incurred by a random initialization of the same model (left-most plot; gray is test, black is training). Colors are analogous to previous experiments, here evenly spaced over the allowable range ($1\leq\alpha\leq 2$). It is clear how modifying the Barron dispersion function shape across this range lets us flexibly interpolate between the test (base) loss distribution achieved by a traditional ERM solution and a mean-variance solution, in terms of both the mean and standard deviation. This monotonicity (as a function of $\alpha$) is salient in the base loss, but this does not always appear for the zero-one loss. Since these datasets have normalized features with negligible label noise, egregious outliers are rare, and thus the trends observed for the mean and standard deviation here also hold for outlier-resistant location-dispersion pairs such as the median and the median-absolute-deviations about the median. Similar results for several other datasets are provided in §B.6, and we remark that the key trends hold across all datasets tested. We have seen in our previous experiments how under heavy-tailed losses/gradients the T-risk solution can differ greatly from that of the vanilla ERM solution, so it is interesting to observe how under large, normalized, clean classification datasets, the T-risk allows us to very smoothly control a tradeoff between average test loss and variance on the test set.