Eliminating Multicollinearity Issues in Neural Network Ensembles: Incremental, Negatively Correlated, Optimal Convex Blending

Consider two zero-bias networks $N(x)$ and $\hat{N}(x)$, with corresponding misfits:

Then their convex blend may be defined as:

with misfit $\tilde{M}(x)=\beta M(x)+(1-\beta)\hat{M}(x)$, and MSE given by:

Note that $\displaystyle\langle\tilde{M}^{2}\rangle$ being quadratic in $\beta$, has a minimum at:

where $\beta_{u}$ is the unconstrained minimizer of $\langle\tilde{M}^{2}\rangle$. If $\beta_{u}\notin[0,1]$, then either $\beta^{*}=1$ or $\beta^{*}=0$ and there is no blending, as can be readily deduced by inspecting eq. (17).
Note from (19) that $\beta_{u}\in(0,1)$, implies:

and in this case $\beta^{*}=\beta_{u}$ leading to:

From eq. (21), one may deduce, after some manipulation, that $\displaystyle\langle\tilde{M}^{2}\rangle^{*}\ <\ \min\{\langle M^{2}\rangle,\langle\hat{M}^{2}\rangle\}$.
The important conclusion is that a convex combination of two zero-bias networks may be arranged so as to yield a zero-bias network with a reduced MSE.

The idea is to construct recursively an aggregate network, by combining each time the current aggregate with a newly trained network. Namely, if by $N^{(a)}_{k}(x)$ we denote the $k^{th}$ aggregate network, then the $(k+1)^{th}$ aggregate will be given by the convex blend:

Let $M_{k}(x)\equiv N^{(a)}_{k}(x)-y(x)$ be the aggregate misfit. Then $N_{k+1}(x)$ is trained so that its misfit $m_{k+1}(x)$ satisfies:

in order to guarantee convex blending (i.e. $\beta_{k+1}\in[0,1]$). This is a constrained optimization problem stated formally as:

Note that since the positive quantity: $\min\{\langle M_{k}^{2}\rangle,\langle m^{2}_{k+1}\rangle\}$, is expected to be small, condition (25) may be satisfied by imposing that the aggregate and the new network misfits are negatively correlated or uncorrelated, i.e. $\langle M_{k}m_{k+1}\rangle\leq 0$, which is a stronger requirement than (25), and which in addition prevents $\beta_{k+1}$ from assuming the limiting no-blend values of $0$ and $1$³³3For different blending restrictions see Appendix B.
The next aggregate misfit is then given by:

Setting $M_{1}(x)=m_{1}(x)$ (and hence $\beta_{1}=0$), and noting that $M_{k}$ is a linear combination of $m_{1}(x)$, $m_{2}(x),\cdots,m_{k}(x)$, one may express $M_{k}$ as:

with the coefficients $a^{(k)}_{i}$ given by:

One may verify that $\displaystyle\sum_{i=1}^{k}a^{(k)}_{i}=1\hbox{\ and \ }a^{(k)}_{i}\geq 0,$ and the aggregate network is then given by:

Note that the aggregate network $N^{(a)}_{k}(x)$ is not the simple average of the member networks as is the case in Ahmad and Zhang (2009) and in the negative correlation approaches Liu and Yao (1999); Chan and Kasabov (2005); Chen and Yao (2009); Brown et al. (2005). The combination coefficients $a^{(k)}_{i}$ given by eq. (29 ), do take in account the relative importance of the different network contributions, and are being built iteratively by optimal pairwise convex blending, avoiding so problems due to multicollinearity. This is in contrast to the approach described by eq. (15) which had been adopted by several authors in the past.

The above results and ideas are employed to design an algorithm for practical use.

Initialization

• •

  Create the empty list, MList, to store the aggregate misfit in each step.

• •

  Create the empty list, BList, to store $\beta$’s.

• •

  Create the empty list, ModelList to store the trained neural network model.

• •

  Set $b_{L},\ b_{U}$ both $\in(0,1)$, the lower and upper bound for the $\beta$’s.

• •

  Set $K_{e}$, the number of networks to be contained in the ensemble.

Main Part

1. 1.

   Train the first zero bias network, $N_{1}(x)$ and push it to the ModelList.

2. 2.

   Set $M_{1}(x_{i})=N_{1}(x_{i})-y_{i},\forall\{x_{i},y_{i}\}\in T_{r}$ and push it to MList.

3. 3.

   Set $\beta_{1}=0$ and push it to the BList.

4. 4.

   $k\leftarrow 1$

5. 5.

   Read $M_{k}(x),\ \forall x\in T_{r}$ from MList.

6. 6.

   Train the zero bias network $N_{k+1}(x)$ s.t. $\langle M_{k}m_{k+1}\rangle\ \leq\ \min\{\langle M_{k}^{2}\rangle,\langle m^{2}_{k+1}\rangle\}$.
   This can be facilitated via a penalty method, i.e. by optimizing:

   $$\langle m^{2}_{k+1}\rangle+\lambda\max\{\langle m_{k+1}M_{k}\rangle,0\}$$

   for a sequence of increasing $\lambda$ values, until $\beta_{k+1}\in[b_{L},b_{U}]$^***$\beta_{k+1}$ is calculated according to eq. (27).

7. 7.

   Calculate $M_{k+1}$ from eq. (26) and $\langle M^{2}_{k+1}\rangle$.

8. 8.

   Push $M_{k+1}$ to MList, $\beta_{k+1}$ to BList, and $N_{k+1}$ to ModelList.

9. 9.

   $k\leftarrow k+1$

10. 10.

    If $k<K_{e}$ repeat from .

11. 11.

    Training Task Completed.

Final Network Assembly

From BList and ModelList, recover the $\beta$’s and the trained networks respectively.

1. (a)

   From $\beta_{1},\cdots,\beta_{K_{e}}$, calculate the member coefficients $a_{i}^{(K_{e})}$ using eq. (29).

2. (b)

   Construct the final zero-bias aggregate network $N^{(a)}_{K_{e}}(x)$, according to eq. (30).

Model function: $f_{1}(x),x\in[-4,4]$. We construct a set $D$ containing $M_{D}=1000$ equidistant points, $x_{i}=-4+(i-1)\frac{8}{999}$. The training set $T_{r}$ contains $M_{r}=38$ equidistant points, $z_{j}=-4+(j-1)\frac{216}{999}=x_{27j-26}$. Note that $T_{r}\subset D$ and the test set is $T_{s}=D-T_{r}$. The ensemble contained six networks. Details of this experiment are listed in table 1. Among the 6 models, model No. 5 has the minimum training MSE, which is 0.38498. The training MSE of the final aggregate is 0.14035, which is $63\%$ lower than that of model No. 5. The plots of the six member-networks $N_{i}(x)$ together with the target function $f_{1}(x)$ are depicted in fig. 2. Likewise, plots of the corresponding aggregate networks are depicted in fig. 3.

To escalate the difficulty of the task, in this case we have experimented with $f_{1}(x)$ in a wider range $x\in[-6,6]$, which introduces intense oscillatory behavior illustrated in fig. 4.

We construct a set $D$ containing $M_{D}=1201$ equidistant points, $x_{i}=-6+\frac{i-1}{100}$. The training set $T_{r}$ contains $M_{r}=121$ equidistant points, $z_{j}=-6+\frac{j-1}{10}=x_{10j-9}$. Note that $T_{r}\subset D$ and the test set is $T_{s}=D-T_{r}$. The generalization performance is quite satisfactory but not of the quality of the first example, as can be seen by inspecting table 2. This was expected since the rapid oscillations as $|x|$ grows, render the task harder.

We have noticed that if one of the member networks $N_{i}(x)$ overfits the data, then a number of additional networks are necessary in the ensemble in order to eliminate the overfit effect. So it seems that a preferred tactic would be to use small networks, that are less prone to over-fitting, rather than large ones. This also serves the need of the so called “few-shot learning”, that refers to problems with limited training examples Wang et al. (2020).

In fig. 5, the member networks are plotted. Note that networks $N_{2}(x)$, $N_{4}(x)$, $N_{5}(x)$ and $N_{13}(x)$, clearly overfit $f_{1}(x)$. However the aggregate networks, plotted in figure 6, eliminate this effect gradually, by appropriate weighting. Note that the corresponding coefficients $a_{2},a_{4},a_{5}$ and $a_{13}$ are among the lowest ones in the linear combination composing the final aggregate network. Among the 15 models, model No. 11 has the minimum training MSE, which is 0.98583. The training MSE of final aggregate model is 0.31371, which is $68\%$ lower than that of model No. 11.

We have also experimented with the multidimensional function, $f_{2}(\mathbf{x})$, where we have set $d=4$. Again, we use MLP’s with one hidden layer. For this function, we have used a grid of 15 equidistant points along each dimension, producing so a set $D$ containing a total of 50,625 points. We have chosen 2,000 of these points at random (seed is set to 12345) for the training set $T_{r}$, and the rest were used for the test set $T_{s}$. Detailed results of the experiments with $f_{2}(\mathbf{x})$, are listed in Table 3. Model No. 6 has a training MSE of 0.04537, which is the lowest among the 10 trained models. The training MSE of the final aggregate is 0.02547, which is $43\%$ lower than that of model No. 6.

In this case we have experimented with noisy data. Model function: $f_{1}(x)$ with $x\in[-5,5]$. We construct a set $D$ containing $M_{D}=1001$ equidistant points, $x_{i}=-5+\frac{i-1}{100}$. The training set $T_{r}$ contains $M_{r}=101$ equidistant points, $z_{j}=-5+\frac{j-1}{10}=x_{10j-9}$ and the target values are contaminated with white normal noise ($0.3N(0,1)$). The test set is left intact, i.e. without any added noise. The ensemble comprises 15 different networks. In fig. 8, the plots of $f_{1}(x)$ and of the trained member networks are displayed. The aggregate ANN’s are plotted in fig. 9. The noisy data are depicted in fig. 7, where the dots correspond to the noisy training observations, while the solid line is a plot of $f_{1}(x)$.

Detailed results are listed in table 4. As expected, the aggregate training error decreases by adding more networks and similar is the behavior of the test error. From column “AG. MSE/TE” of table 4, we note that the aggregate network generalizes extremely well. Note, model No. 12 has the lowest training MSE among all models which is 1.43211. The training MSE of the final aggregate is 0.45040, which is $68\%$ lower than that of model No. 12.