Asymmetric Totally-corrective Boosting for Real-time Object Detection

Due to its important applications in video surveillance, interactive human-machine interface etc, real-time object detection has attracted extensive research recently [1, 2, 3, 4, 5, 6]. Although it was introduced a decade ago, the boosted cascade classifier framework of Viola and Jones [2] is still considered as the most promising approach for object detection, and this framework is the basis which many papers have extended.

One difficulty in object detection is the problem is highly asymmetric. A common method to detect objects in an image is to exhaustively search all sub-windows at all possible scales and positions in the image, and use a trained model to detect target objects. Typically, there are only a few targets in millions of searched sub-windows. The cascade classifier framework partially solves the asymmetry problem by splitting the detection process into several nodes. Only those sub-windows passing through all nodes are classified as true targets. At each node, we want to train a classifier with a very high detection rate (e.g., $99.5\%$) and a moderate false positive rate (e.g., around $50\%$). The learning goal of each node should be asymmetric in order to achieve optimal detection performance. A drawback of standard boosting like AdaBoost in the context of the cascade framework is it is designed to minimize the overall false rate. The losses are equal for misclassifying a positive example and a negative example, which makes it not be able to build an optimal classifier for the asymmetric learning goal.

Many subsequent works attempt to improve the performance of object detectors by introducing asymmetric loss functions to boosting algorithms. Viola and Jones proposed asymmetric AdaBoost [3], which applies an asymmetric multiplier to one of the classes. However, this asymmetry is absorbed immediately by the first weak classifier because AdaBoost’s optimization strategy is greedy. In practice, they manually apply the $n$-th root of the multiplier on each iteration to keep the asymmetric effect throughout the entire training process. Here $n$ is the number of weak classifiers. This heuristic cannot guarantee the solution to be optimal and the number of weak classifiers need to be specified before training. AdaCost presented by Fan et al. [7] adds a cost adjustment function on the weight updating strategy of AdaBoost. They also pointed out that the weight updating rule should consider the cost not only on the initial weights but also at each iteration. Li and Zhang [8] proposed FloatBoost to reduce the redundancy of greedy search by incorporating floating search with AdaBoost. In FloatBoost, the poor weak classifiers are deleted when adding the new weak classifier. Xiao et al. [9] improved the backtrack technique in [8] and exploited the historical information of preceding nodes into successive node learning. Hou et al. [10] used varying asymmetric factors for training different weak classifiers. However, because the asymmetric factor changes during training, the loss function remains unclear. Pham et al. [11] presented a method which trains the asymmetric AdaBoost [3] classifiers under a new cascade structure, namely multi-exit cascade. Like soft cascade [12], boosting chain [9] and dynamic cascade [13], multi-exit cascade is a cascade structure which takes the historical information into consideration. In multi-exit cascade, the $n$-th node “inherits” weak classifiers selected at the preceding $n-1$ nodes. Wu et al. [14] stated that feature selection and ensemble classifier learning can be decoupled. They designed a linear asymmetric classifier (LAC) to adjust the linear coefficients of the selected weak classifiers. Kullback-Leibler Boosting [15] iteratively learns robust linear features by maximizing the Kullback-Leibler divergence.

Much of the previous work is based on AdaBoost and achieves the asymmetric learning goal by heuristic weights manipulations or post-processing techniques. It is not trivial to assess how these heuristics affect the original loss function of AdaBoost. In this work, we construct new boosting algorithms directly from asymmetric losses. The optimization process is implemented by column generation. Experiments on toy data and real data show that our algorithms indeed achieve the asymmetric learning goal without any heuristic manipulation, and can outperform previous methods.

Therefore, the main contributions of this work are as follows.

1. 1.

   We utilize a general and systematic framework (column generation) to construct new asymmetric boosting algorithms, which can be applied to a variety of asymmetric losses. There is no heuristic strategy in our algorithms which may cause suboptimal solutions. In contrast, the global optimal solution is guaranteed for our algorithms.

   Unlike Viola-Jones’ asymmetric AdaBoost [3], the asymmetric effect of our methods spreads over the entire training process. The coefficients of all weak classifiers are updated at each iteration, which prevents the first weak classifier from absorbing the asymmetry. The number of weak classifiers does not need to be specified before training.

2. 2.

   The asymmetric totally-corrective boosting algorithms introduce the asymmetric learning goal into both feature selection and ensemble classifier learning. Both the example weights and the linear classifier coefficients are learned in an asymmetric way.

3. 3.

   In practice, L-BFGS-B [16] is used to solve the primal problem, which runs much faster than solving the dual problem and also less memory is needed.

4. 4.

   We demonstrate that with the totally corrective optimization, the linear coefficients of some weak classifiers are set to zero by the algorithm such that fewer weak classifiers are needed. We present analysis on the theoretical condition and show how useful the historical information is for the training of successive nodes.

In this section, we propose two asymmetric losses, which are motivated by asymmetric AdaBoost [3] and cost-sensitive LogitBoost [17], respectively.

We first introduce an asymmetric cost in the following form:

Here ${\boldsymbol{x}}$ is the input data, $y$ is the label and $F({\boldsymbol{x}})$ is the learned classifier. Viola and Jones [3] directly take the product of $\rm ACost$ and the exponential loss $E_{{\boldsymbol{X}},Y}[\exp(-yF({\boldsymbol{x}})]$ as the asymmetric loss:

where ${\bf I}(\cdot)$ is the indicator function. In a similar manner, we can also form an asymmetric loss from the logistic loss $E_{{\boldsymbol{X}},Y}[\mathrm{logit}\big{(}-yF({\boldsymbol{x}})\big{)}]$:

where $\mathrm{logit}(x)=\log(1+\exp(-x))$ is the logistic loss function.

Masnadi-Shirazi and Vasconcelos [17] proposed cost-sensitive boosting algorithms which optimize different versions of cost-sensitive losses by the means of gradient descent. They proved that the optimal cost-sensitive predictor minimizes the expected loss:

With fixing $\gamma$ to $1$, the expected loss can be reformulated to

In this section, we construct asymmetric totally-corrective boosting algorithms (termed AsymBoost_TC here) from the losses (5) and (6) discussed previously. In contrast to the methods constructing boosting-like algorithms in [17], [18] and [19], we use column generation to design our totally corrective boosting algorithms, inspired by [20] and [5].

Suppose there are $M$ training examples ($M_{1}$ positives and $M_{2}$ negatives), and the sequence of examples are arranged according to the labels (positives first). The pool ${\mathcal{H}}$ contains $N$ available weak classifiers. The matrix $H\in\mathbb{Z}^{M\times N}$ contains binary outputs of weak classifiers in ${\mathcal{H}}$ for training examples, namely $H_{ij}=h_{j}(x_{i})$. We are aiming to learn a linear combination $F_{\boldsymbol{w}}(\cdot)=\sum_{j=1}^{N}w_{i}h_{j}(\cdot)$. $C_{1}$ and $C_{2}$ are costs for misclassifying positives and negatives, respectively. We assign the asymmetric factor $k=C_{2}/C_{1}$ and restrict $\gamma=(C_{1}+C_{2})/2$ to $1$, thus $C_{1}$ and $C_{2}$ are fixed for a given $k$.

The problems of the two AsymBoost_TC algorithms can be expressed as:

where ${\boldsymbol{l}}=[C_{1}/M_{1},\cdots,C_{2}/M_{2},\cdots]^{{\!\top}}$, and

where ${\boldsymbol{e}}=[1/M_{1},\cdots,1/M_{2},\cdots]^{{\!\top}}$. In both (7) and (8), $z_{i}$ stands for the margin of the $i$-th training example. We refer (7) as AsymBoost_TC1 and (8) as AsymBoost_TC2. Note that here the optimization problems are $\ell_{1}$-norm regularized. It is possible to use other format of regularization such as the $\ell_{2}$-norm.

First we introduce a fact that the Fenchel conjugate [21] of the logistic loss function $\mathrm{logit}(x)$ is

Now we derive the Lagrange dual [21] of AsymBoost_TC1. The Lagrangian of (7) is

The dual function

The dual problem is

Since the problem (7) is convex and the Slater’s conditions are satisfied [21], the duality gap between the primal (7) and the dual (3) is zero. Therefore, the solutions of (7) and (3) are the same. Through the KKT condition, the gradient of Lagrangian (3) over primal variable ${\boldsymbol{z}}$ and dual variable ${\boldsymbol{u}}$ should vanish at the optimum. Therefore, we can obtain the relationship between the optimal value of ${\boldsymbol{z}}$ and ${\boldsymbol{u}}$:

Similarly, we can get the dual problem of AsymBoost_TC2, which is expressed as:

with

In practice, the total number of weak classifiers, $N$, could be extremely large, so we can not solve the primal problems (7) and (8) directly. However equivalently, we can optimize the duals (3) and (3) iteratively using column generation [20]. In each round, we add the most violated constraint by finding a weak classifier satisfying:

This step is the same as training a weak classifier in AdaBoost and LPBoost, in which one tries to find a weak classifier with the maximal edge (i.e. the minimal weighted error). The edge of $h_{j}$ is defined as $\sum_{i=1}^{M}u_{i}y_{i}h_{j}(x_{i})$, which is the inverse of the weighted error. Then we solve the restricted dual problem with one more constraint than the previous round, and update the linear coefficients of weak classifiers (${\boldsymbol{w}}$) and the weights of training examples (${\boldsymbol{u}}$). Adding one constraint into the dual problem corresponds to adding one variable into the primal problem. Since the primal problem and dual problem are equivalent, we can either solve the restricted dual or the restricted primal in practice. The algorithms of AsymBoost_TC1 and AsymBoost_TC2 are summarized in Algorithm 1. Note that, in practice, in order to achieve specific false negative rate (FNR) or false positive rate (FPR), an offset $b$ is needed to be added into the final strong classifier: $F({\boldsymbol{x}})=\sum_{j=1}^{n}w_{i}h_{j}({\mathbf{x}})-b$, which can be obtained by a simple line search. The new weak classifier $h^{\prime}(\cdot)$ corresponds to an extra variable to the primal and an extra constraint to the dual. Thus, the minimal value of the primal decreases with growing variables, and the maximal value of the dual problem also decreases with growing constraints. Furthermore, as the optimization problems involved are convex, Algorithm 1 is guaranteed to converge to the global optimum.

Next we show how AsymBoost_TC introduces the asymmetric learning into feature selection and ensemble classifier learning. Decision stumps are the most commonly used type of weak classifiers, and each stump only uses one dimension of the features. So the process of training weak classifiers (decision stumps) is equivalent to feature selection. In our framework, the weak classifier with the maximum edge (i.e. the minimal weighted error) is selected. From (10) and (12), the weight of $i$-th example, namely $u_{i}$, is affected by two factors: the asymmetric factor $k$ and the current margin $z_{i}$. If we set $k=1$, the weighting strategy goes back to being symmetric. On the other hand, the coefficients of the linear classifier, namely ${\boldsymbol{w}}$, are updated by solving the restricted primal problem at each iteration. The asymmetric factor $k$ in the primal is absorbed by all the weak classifiers currently learned. So feature selection and ensemble classifier learning both consider the asymmetric factor $k$.

The number of variables of the primal problem is the number of weak classifiers, while for the dual problem, it is the number of training examples. In the cascade classifiers for face detection, the number of weak classifiers is usually much smaller than the number of training examples, so solving the primal is much cheaper than solving the dual. Since the primal problem has only simple box-bounding constraints, we can employ L-BFGS-B [16] to solve it. L-BFGS-B is a tool based on the quasi-Newton method for bound-constrained optimization.

Instead of maintaining the Hessian matrix, L-BFGS-B only needs the recent several updates of values and gradients for the cost function to approximate the Hessian matrix. Thus, L-BFGS-B requires less memory when running. In column generation, we can use the results from previous iteration as the starting point of current problem, which leads to further reductions in computation time.

The complementary slackness condition [21] suggests that $\lambda_{j}w_{j}=0$. So we can get the conditions of sparseness:

This means that, if the weak classifier $h_{j}(\cdot)$ is so “weak” that its edge is less than $\theta$ under the current distribution ${\boldsymbol{u}}$, its contribution to the ensemble classifier is “zero”. From another viewpoint, the $\ell_{1}$-norm regularization term in the primal (7) and (8), leads to a sparse result. The parameter $\theta$ controls the degree of the sparseness. The larger $\theta$ is, the sparser the result would be.

We have proposed two asymmetric totally-corrective boosting algorithms for object detection, which are implemented by the column generation technique in convex optimization. Our algorithms introduce asymmetry into both feature selection and ensemble classifier learning in a systematic way.

Both our algorithms achieve better results for face detection than AdaBoost and Viola-Jones’ asymmetric AdaBoost. An observation is that we can not see great differences on performances between AsymBoost_TC1 and AsymBoost_TC2 in our experiments. For the face detection task, AdaBoost already achieves a very promising result, so the improvements of our method are not very significant.

One drawback of our algorithms is there are two parameters to be tuned. For different nodes, the optimal parameters should not be the same. In this work, we have used the same parameters for all nodes. Nevertheless, since the probability of negative examples decreases with the node index, the degree of the asymmetry between positive and negative examples also deceases. The optimal $k$ may decline with the node index.

The framework for constructing totally-corrective boosting algorithms is general, so we can consider other asymmetric losses (e.g., asymmetric exponential loss) to form new asymmetric boosting algorithms. In column generation, there is no restriction that only one constraint is added at each iteration. Actually, we can add several violated constraints at each iteration, which means that we can produce multiple weak classifiers in one round. By doing this, we can speed up the learning process.

Motivated by the analysis of sparseness, we find that the very early nodes contribute little information for training the later nodes. Based on this, we can exclude some useless nodes when the node index grows, which will simplify the multi-exit structure and shorten the testing time.