Optimal Decision Rules for Simple Hypothesis Testing under General Criterion Involving Error Probabilities

Consider a detection problem with $M$ simple hypotheses:

where the random observation $\boldsymbol{Y}$ takes values from an observation set $\Gamma$ with $\Gamma\subset{\mathbb{R}}^{N}$. Depending on whether the observed random vector $\boldsymbol{Y}\in\Gamma$ is continuous-valued or discrete-valued, $f_{j}(\boldsymbol{y})$ denotes either the probability density function (pdf) or the probability mass function (pmf) under hypothesis $\mathcal{H}_{j}$. For compactness of notation, the term density is used for both pdf and pmf. In order to decide among the hypotheses, we consider the set of pointwise randomized decision functions, denoted by $\mathsf{D}$, i.e., $\boldsymbol{\delta}\vcentcolon=(\delta_{0},\delta_{1},\ldots,\delta_{M-1})\in\mathsf{D}$ such that $\sum_{i=0}^{M-1}\delta_{i}(\boldsymbol{y})=1$ and $\delta_{i}(\boldsymbol{y})\in[0,1]$ for $0\leq i\leq M-1$ and $\boldsymbol{y}\in\Gamma$. More explicitly, given the observation $\boldsymbol{y}$, the detector decides in favor of hypothesis $\mathcal{H}_{i}$ with probability $\delta_{i}(\boldsymbol{y})$. Then, the probability of choosing hypothesis $\mathcal{H}_{i}$ when hypothesis $\mathcal{H}_{j}$ is true, denoted by $p_{ij}$ with $0\leq i,j\leq M-1$, is given by

where $\mathbb{E}_{j}[\cdot]$ denotes expected value under hypothesis $\mathcal{H}_{j}$ and $\mu(d\boldsymbol{y})$ is used in (2) to denote the $N-$fold integral and sum for continuous and discrete cases, respectively. Let $\boldsymbol{p}(\boldsymbol{\delta})$ denote the (column) vector containing all pairwise error probabilities $p_{ij}$ for $0\leq i,j\leq M-1$ and $i\neq j$ corresponding to the decision rule $\boldsymbol{\delta}$. It is sufficient to include only the pairwise error probabilities in $\boldsymbol{p}(\boldsymbol{\delta})$, i.e., $p_{ij}$ with $i\neq j$. To see this, note that (2) in conjunction with $\sum_{i=0}^{M-1}\delta_{i}(\boldsymbol{y})=1$ imply $\sum_{i=0}^{M-1}p_{ij}=1$, from which we get the probability of correctly identifying hypothesis $\mathcal{H}_{i}$ as $p_{ii}=1-\sum_{i=0,i\neq j}^{M-1}p_{ij}$.

For $M$-ary hypothesis testing, we consider a generic decision criterion that can be expressed in terms of the error probabilities as follows:

where $g_{i}$ and $h_{j}$ denote arbitrary functions of the pairwise error probability vector. Classical hypothesis testing criteria such as Bayesian, minimax, Neyman-Pearson (NP) [1], generalized Neyman-Pearson [2], restricted Bayesian [3], and prospect theory based hypothesis testing [4] are all special cases of the formulation in (3). For example, in the restricted Bayesian framework, the Bayes risk with respect to (w.r.t.) a certain prior is minimized subject to a constraint on the maximum conditional risk [3]:

for some $\alpha\geq\alpha_{m}$, where $\alpha_{m}$ is the maximum conditional risk of the minimax procedure [1]. The conditional risk when hypothesis $H_{j}$ is true, denoted by $R_{j}(\boldsymbol{\delta})$, is given by $R_{j}(\boldsymbol{\delta})=\sum_{i=0}^{M-1}c_{ij}p_{ij}$ and the Bayes risk is expressed as $r_{B}(\boldsymbol{\delta})=\sum_{j=0}^{M-1}\pi_{j}R_{j}(\boldsymbol{\delta})$, where $\pi_{j}$ denotes the a priori probability of hypothesis $\mathcal{H}_{j}$ and $c_{ij}$ is the cost incurred by choosing hypothesis $\mathcal{H}_{i}$ when in fact hypothesis $\mathcal{H}_{j}$ is true. Hence, (I) is a special case of (3).

In this letter, for the first time in the literature, we provide a unified characterization of optimal decision rules for simple hypothesis testing under a general criterion involving error probabilities.

Let $\boldsymbol{v}$ be a real (column) vector of length $M(M-1)$ whose elements are denoted as $v_{ij}$ for $0\leq i,j\leq M-1$ and $i\neq j$. Next, we present an optimal deterministic decision rule that minimizes the weighted sum of $p_{ij}$’s with arbitrary real weights $\boldsymbol{v}$.¹¹1In classical Bayesian $M-$ary hypothesis testing, $v_{ij}=\pi_{j}(c_{ij}-c_{jj})$.

In order to characterize the solution of (3), we first present the following lemma.

Lemma: Let $\boldsymbol{p}_{0}$ be a point on the boundary of $\mathsf{P}$ and $\{\boldsymbol{p}\,:\,\boldsymbol{v}^{T}\boldsymbol{p}=\boldsymbol{v}^{T}\boldsymbol{p}_{0}\}$ be a supporting hyperplane to $\mathsf{P}$ at the point $\boldsymbol{p}_{0}$.
Case 1: Any deterministic decision rule of the form given in (7) corresponding to the weights specified by $\boldsymbol{v}$ yields $\boldsymbol{p}_{0}$ if $\mathsf{B}(\boldsymbol{v})$, defined in (9), has zero probability under all hypotheses.
Case 2: $\boldsymbol{p}_{0}$ is achieved by a randomization among at most $M(M-1)$ deterministic decision rules of the form given in (7), all corresponding to the same weights specified by $\boldsymbol{v}$, if $\mathsf{B}(\boldsymbol{v})$, defined in (9), has nonzero probability under some hypotheses.

Proof: See Appendix A.

It should be noted that the condition in case 1 of the lemma, i.e., $\mathsf{B}(\boldsymbol{v})$ has zero probability under all hypotheses, is not difficult to satisfy. A simple example is when the observation under hypothesis $\mathcal{H}_{i}$ is Gaussian distributed with mean $\mu_{i}$ and variance $\sigma^{2}$ for all $0\leq i\leq M-1$. Furthermore, the lemma implies that any extreme point of the convex set $\mathsf{P}$, i.e., any point on the boundary of the convex set $\mathsf{P}$ that is not a convex combination of any other points in the set, can be achieved by a deterministic decision rule of the form (7) without any randomization. The points that are on the boundary but not extreme points can be obtained via randomization as stated in case 2.

Next, we present a unified characterization of the optimal decision rule for problems that are in the form of (3). We suppose that the problem in (3) is feasible and let $\boldsymbol{\delta}^{\ast}$ and $\boldsymbol{p}^{\ast}(\boldsymbol{\delta}^{\ast})$ denote an optimal decision rule and the corresponding pairwise error probabilities, respectively.

Theorem: An optimal decision rule that solves (3) can be obtained as
Case 1: a randomization among at most two deterministic decision rules of the form given in (7), each specified by some real $\boldsymbol{v}$, if $\mathsf{B}(\boldsymbol{v})$, defined in (9), has zero probability under all hypotheses for all real $\boldsymbol{v}$; otherwise
Case 2: a randomization among at most $M(M-1)+1$ deterministic decision rules of the form given in (7), one specified by some real $\boldsymbol{v}$ and the remaining $M(M-1)$ correspond to the same weights specified by another real $\boldsymbol{v}$.

Proof: If the optimal point $\boldsymbol{p}^{\ast}(\boldsymbol{\delta}^{\ast})$ is on the boundary of $\mathsf{P}$, then the lemma takes care of the proof. Here, we consider the case when $\boldsymbol{p}^{\ast}(\boldsymbol{\delta}^{\ast})$ is an interior point of $\mathsf{P}$. First, we pick an arbitrary $\boldsymbol{v}^{1}\in\mathbb{R}^{M(M-1)}$ and derive the optimal deterministic decision rule according to (7). Let $\boldsymbol{p}^{1}$ denote the pairwise error probability vector corresponding to the employed decision rule. Then, we move along the ray that originates from $\boldsymbol{p}^{1}$ and passes through $\boldsymbol{p}^{\ast}(\boldsymbol{\delta}^{\ast})$. Since $\boldsymbol{P}$ is bounded, this ray will intersect with the boundary of $\boldsymbol{P}$ at some point, say $\boldsymbol{p}^{2}$. If the condition in case 1 is satisfied, then by lemma-case 1, there exists a deterministic decision rule of the form given in (7) that yields $\boldsymbol{p}^{2}$. Otherwise, by lemma-case 2, $\boldsymbol{p}^{2}$ is achieved by a randomization among at most $M(M-1)$ deterministic decision rules of the form given in (7), all sharing the same weight vector $\boldsymbol{v}^{2}$. Since $\boldsymbol{p}^{\ast}(\boldsymbol{\delta}^{\ast})$ resides on the line segment that connects $\boldsymbol{p}^{1}$ to $\boldsymbol{p}^{2}$, it can be attained by appropriately randomizing among the decision rules that yield $\boldsymbol{p}^{1}$ and $\boldsymbol{p}^{2}$. $\blacksquare$

When the optimization problem in (3) possesses certain structure, the maximum number of deterministic decision rules required to achieve optimal performance may be reduced below those given in the theorem. For example, suppose that the objective is a concave function of $\boldsymbol{p}$ and there are a total of $n$ constraints in (3) which are all linear in $\boldsymbol{p}$ (i.e., the feasible set, denoted by $\mathsf{P}^{\prime}$, is the intersection of $\mathsf{P}$ with halfspaces and hyperplanes). It is well known that the minimum of a concave function over a closed bounded convex set is achieved at an extreme point [5]. Hence, in this case, the optimal point $\boldsymbol{p}^{\ast}$ is an extreme point of $\mathsf{P}^{\prime}$. By Dubin’s theorem [6], any extreme point of $\mathsf{P}^{\prime}$ can be written as a convex combination of $n+1$ or fewer extreme points of $\mathsf{P}$. Since any extreme point of $\mathsf{P}$ can be achieved by a deterministic decision rule of the form (7), the optimal decision rule is obtained as a randomization among at most $n+1$ deterministic decision rules of the form (7). If there are no constraints in (3), i.e., $n=0$, the deterministic decision rule given in (7) is optimal and no randomization is required with a concave objective function.

An immediate and important corollary of the theorem is given below.

Corollary: Likelihood ratios are sufficient statistics for simple $M-$ary hypothesis testing under any decision criterion that is expressed in terms of arbitrary functions of error probabilities as specified in (3).

Proof: It is stated in the theorem that a solution of the generic optimization problem in (3) can be expressed in terms of decision rules of the form given in (7). These decision rules only involve comparisons among $V_{i}(\boldsymbol{y})$’s, which are linear w.r.t. the density terms $f_{i}(\boldsymbol{y})$’s. Normalizing $f_{i}(\boldsymbol{y})$’s with $f_{0}(\boldsymbol{y})$ and defining $L_{i}(\boldsymbol{y})\vcentcolon=f_{i}(\boldsymbol{y})/f_{0}(\boldsymbol{y})$, we see that an optimal decision rule that solves the problem in (3) depends on the observation $\boldsymbol{y}$ only through the likelihood ratios. $\blacksquare$

In this section, numerical examples are presented by considering a binary hypothesis testing problem; i.e., $M=2$ in (1). Suppose that a bit ($0$ or $1$) is sent over two independent binary channels to a decision maker, which aims to make an optimal decision based on the binary channel outputs. The output of binary channel $k$ is denoted by $y_{k}\in\{0,1\},\ k=1,2$, and the decision maker declares its decision based on $\boldsymbol{y}=[y_{1},y_{2}]$. The probability that the output of binary channel $k$ is $i$ when bit $j$ is sent is denoted by $p_{ij}^{(k)}$ for $0\leq i,j\leq 1$ with $p_{0j}^{(k)}+p_{1j}^{(k)}=1$. Then, the pmf of $\boldsymbol{y}$ under $\mathcal{H}_{j}$ is given by

for $j\in\{0,1\}$. As in the previous sections, the pairwise error probability vector of the decision maker for a given decision rule $\boldsymbol{\delta}$ is represented by $\boldsymbol{p}(\boldsymbol{\delta})$, which is expressed as $\boldsymbol{p}(\boldsymbol{\delta})=[p_{10},p_{01}]^{T}$ in this case. It is assumed that the decision maker knows the conditional pdfs in (11).

In this section, a special case of (3) is considered based on prospect theory by focusing on a behavioral decision maker [4, 7, 8, 9]. In particular, there exist no constraints (i.e., $m=p=0$ in (3)) and the objective function in (3) is expressed as

The aim of the decision maker is to obtain a decision rule that minimizes (12). In the first example, $\kappa$ is set to $5$, and the parameters of the binary channels are selected as $p_{10}^{(1)}=p_{10}^{(2)}=0.4$ and $p_{01}^{(1)}=p_{01}^{(2)}=0.1$. In this case, it can be shown via (11) that there exist $6$ different deterministic decision rules in the form of (7), which achieve the pairwise error probability vectors marked with blue stars in Fig. 1. The convex hull of these pairwise error probability vectors is also illustrated in the figure. Over these deterministic decision rules (i.e., in the absence of randomization), the minimum achievable value of (12) becomes $0.1901$, which corresponds to the pairwise error probability vector shown with the green square in Fig. 1. If randomization between two deterministic decision rules in the form of (7) is considered, the resulting minimum objective value becomes $0.0422$, and the corresponding pairwise error probability vector is indicated with the red triangle in the figure. On the other hand, in compliance with the theorem (case 2), the minimum value of (12) is achieved via randomization of (at most) three deterministic decision rules in the form of (7) (since $M(M-1)+1=3$). In this case, the optimal decision rule randomizes among $\delta_{1}$, $\delta_{2}$, and $\delta_{3}$, with randomization coefficients of $0.41$, $0.51$, and $0.08$, respectively, as given below:

This optimal decision rule achieves the lowest objective value of $0.0400$, and the corresponding pairwise error probability vector is marked with the black circle in Fig. 1. Hence, this example shows that randomization among three deterministic decision rules may be required to obtain the solution of (3).

In the second example, the parameters are taken as $\kappa=1.5$, $p_{10}^{(1)}=0.3$, $p_{10}^{(2)}=0.2$, $p_{01}^{(1)}=0.4$, and $p_{01}^{(2)}=0.25$. In this case, there exist $8$ different deterministic decision rules in the form of (7), which achieve the pairwise error probability vectors marked with blue stars in Fig. 2. The minimum value of (12) among these deterministic decision rules is $3.9278$, which corresponds to the pairwise error probability vector shown with the green square in the figure. In addition, the pairwise error probability vectors corresponding to the solutions with randomization of two and three deterministic decision rules are marked with the red triangle and the black circle, respectively. In this scenario, the minimum objective value ($3.8432$) can be achieved via randomization of two deterministic decision rules, as well. This is again in compliance with the theorem (case 2), which states that an optimal decision rule can be obtained as a randomization among at most $M(M-1)+1$ deterministic decision rules of the form given in (7).

This letter presents a unified characterization of optimal decision rules for simple $M-$ary hypothesis testing under a generic performance criterion that depends on arbitrary functions of error probabilities. It is shown that optimal performance with respect to the design criterion can be achieved by randomizing among at most two deterministic decision rules of the form reminiscent (but not necessarily identical) to Bayes rule when points on the decision boundary do not contribute to the error probabilities. For the general case, the solution for an optimal decision rule is reduced to a search over two weight coefficient vectors, each of length $M(M-1)$. Likelihood ratios are shown to be sufficient statistics. Classical performance measures including Bayesian, minimax, Neyman-Pearson, generalized Neyman-Pearson, restricted Bayesian, and prospect theory based approaches all appear as special cases of the considered framework.

Finally, we point out that the form of optimal local sensor decision rules for the problem of distributed detection [10, 11, 12, 13] with conditionally independent observations at the sensors and an arbitrary fusion rule can be characterized using the proposed framework.