Fuzzy Knowledge Distillation from
High-Order TSK to Low-Order TSK

TSK fuzzy classifier can be described by fuzzy IF-THEN rules, which indicate the input-output relationship of the classifier. For TSK fuzzy classifiers, the fuzzy rules can be expressed as follows:

where $k=1,2,\ldots,K$. $K$ is the total number of fuzzy rules, the input is denoted as $\mathbf{x}=(x_{1},x_{2},\ldots,x_{i},\ldots,x_{m})^{T}$, $A_{i}^{k}$ is the fuzzy set of the $k$-th rule of $x_{i}$, $y^{k}$ is the output of the $k$-th rule, $\wedge$ is the fuzzy conjunction operation. If $f^{k}(\mathbf{x})=p_{0}^{k}$, we term the TSK fuzzy classifier as the Zero-order TSK fuzzy classifier [9]. If $f^{k}(\mathbf{x})=p_{0}^{k}+x_{1}^{k}p_{1}^{k}+x_{2}^{k}p_{2}^{k}+\ldots+x_{m}^{k}p_{m}^{k}$, the TSK fuzzy classifier is termed as the First-order TSK fuzzy classifier [10]. If $f^{k}(\mathbf{x})$ is described as:

It is termed as High-order TSK fuzzy classifier [11]. Where $n(n\geq 2)$ is the order of the highest polynomial of TSK fuzzy classifier, $j_{i}$ is the order of $x_{i}$, $a_{j_{1},j_{2},\ldots,j_{m}}^{k}$ represents the coefficient of the highest polynomial with $m$ independent variables in the linear combination constituting the $k$-th rule.

It is clearly that the consequent parameters of Zero-order and First-order TSK fuzzy classifier are relatively simple, they are the coefficients of the fuzzy membership degree and input variables, which ensure the high interpretability but need more fuzzy rules to achieve satisfactory performance. In contrast, High-order TSK fuzzy classifier can preform better with fewer fuzzy rules, but the consequent part of fuzzy rule are extremely complex and sharply reduce the interpretability. As a result, how to combine the strong performance of High-order TSK fuzzy classifier with the high interpretability of Low-order TSK fuzzy classifier becomes a very valuable research direction.

Knowledge distillation [14] transfers the dark knowledge from complex/large model, namely the teacher model, to simple/small model, namely the student model, and hence improve the performance of student model. Specifically, knowledge distillation introduces the hyper-parameter temperature in softmax function to obtain soft labels at first, then calculates the KL divergence of soft labels and the cross-entropy of student model output and the ground-truth label, finally transfers the dark knowledge via soft labels from teacher model to student model, improves the performance of student model, as shown in Fig. 1.

There are many strategies to construct the loss function in knowledge distillation, such as the KL divergence [14], the mean squared error [37] and the Jensen–Shannon divergence [38], etc. Traditional knowledge distillation transfers dark knowledge in a highly coupled way, which limits the flexibility for knowledge transfer. Zhao et al. [32] pointed out that dark knowledge can be decoupled into target class knowledge and non-target class knowledge and to be transfered to student model in a more flexible way by reconstructing the KL divergence. Furlanello et al. [33] demonstrated that the decoupled dark knowledge of teacher model can guide student model to have stronger generalization ability than that of teacher model. In this paper, we attempt to distill fuzzy dark knowledge from High-order TSK fuzzy classifier, and propose a novel born-again TSK fuzzy classifier endowed with the powerful classification performance as well as high interpretability.

Here, we summarize the learning algorithm of teacher model and student model in Algorithm 1. The proposed HTSK-LLM-DKD is given in Algorithm 2.

Table I demonstrates the experiment results of the proposed HTSK-LLM-DKD and HTSK-LLM-KD compared with nine adopted fuzzy classifiers. We conclude the following results:

1. 1.

   Our HTSK-LLM-DKD and HTSK-LLM-KD all achieve the first and the second best performance on eleven out of sixteen datasets in terms of $Accuracy$ and $Weighted-F$, respectively, which shows that knowledge distillation can improve the performance of TSK fuzzy classifier very well and HTSK-LLM-DKD is much better than HTSK-LLM-KD. Additionally, HTSK-LLM-DKD keeps a comparable performance on TIT, WIL, PHO, MAG and ADU datasets.

2. 2.

   Evidently, HTSK-LLM-DKD performs better with fewer fuzzy rules than TSK${}_{v2}^{1}$, which is the student model of HTSK-LLM-DKD. At the same time, HTSK-LLM-DKD performs better with shorter running time than High-order TSK fuzzy classifier on most datasets. It shows that the fuzzy dark knowledge improves the generalization ability of HTSK-LLM-DKD greatly, and the proposed LLM-DKD can speed up the running time of HTSK-LLM-DKD.

Table II displays the experiment results of HTSK-LLM-DKD, HTSK-LLM-KD, CNN-TSK-DKD and CNN-TSK-KD compared with its corresponding student model in terms of $Accuracy$ and $Weighted-F$, respectively. We conclude the following results:

1. 1.

   Our HTSK-LLM-DKD has achieved the best performance improvement, with 1.41% and 1.56% in terms of Accuracy and Weighted-F, which indicates that knowledge distillation can effectively improve the performance of student model by transferring fuzzy dark knowledge from teacher model.

2. 2.

   The comparative experiments in Table II can be divided into two parts. The first part is the comparison between decoupled knowledge distillation and traditional knowledge distillation. It can be seen that HTSK-LLM-DKD and CNN-TSK-DKD perform better than HTSK-LLM-KD and CNN-TSK-KD, we believe the reason lie in that: in traditional knowledge distillation, the transfer efficiency of non-target knowledge is negatively correlated with the confidence on the target class of teacher model, and decoupled knowledge distillation can better transfer non-target knowledge by re-weighting, thus improving the generalization ability of student model.

3. 3.

   The second part is the comparison between CNN and TSK fuzzy classifier as teacher models. We can observe that the average $Accuracy$ and $Weighted-F$ of HTSK-LLM-KD and HTSK-LLM-DKD are usually higher than CNN-TSK-KD and CNN-TSK-DKD, which indicates that the fuzzy dark knowledge transfered from High-order TSK fuzzy classifier is more conducive to improving the performance of low-order TSK fuzzy classifier than CNN.

From the experiments results above we can conclude that knowledge distillation can improve the generalization ability of model. Next, we will explain how knowledge distillation can affect the performance of student model on the real-world dataset Cleveland heart disease.

We use a real-world dataset Cleveland heart disease (CLE) to demonstrate the performance of our model in detail. CLE is about heart disease in Cleveland city of USA, which is built by University Hospital Zurich and Cleveland Clinic Foundation. CLE is composed of 303 instances with 13 features, including chest pain type, resting blood pressure, resting electrocardiographic results, etc., and the output indicates what risk does the patient have with cardiac disease, which is an integer number ranging from 0 (not present) to 4: 0 for nil risk; 1 for low risk; 2 for potential risk; 3 for high risk; 4 for very high risk [42]. In order to clearly demonstrate the effect of knowledge distillation on the proposed model, we divide CLE into three classes: 0 for zero-risk (nil risk), 1 for low-risk (low risk, potential risk and high risk) and 2 for high-risk (very high risk). The experiments in Table III shows that HTSK-LLM-DKD and HTSK-LLM-KD defeat other comparative methods and obtain the first best and the second best performance on CLE, respectively. In addition, HTSK-LLM-DKD and HTSK-LLM-KD require fewer fuzzy rules (8.1 & 8.6) and less running time (0.3168 & 0.2782), which shows once again that the fuzzy dark knowledge greatly improves the generalization ability of model. In Table IV, HTSK-LLM-DKD achieves the best performance improvement with 3.50% in $Accuracy$ and 3.46% in $Weighted-F$, which are better than HTSK-LLM-KD with 3.42% in $Accuracy$ and 3.40% in $Weighted-F$, illustrating the great performance improvement brought by decoupled knowledge distillation once again, which is in accordance with the conclusions obtained in Section IV-B.

Fig. 3 shows the effects of distillation parameters of the proposed HTSK-LLM-DKD on dataset CLE. All distillation parameters are selected from $\{1,2,5,10,20,100\}$, and it can be clearly seen that in the process of most parameter selection, the classification performance improves first and then degrades. The distillation temperature $\tau$ has an important effect on the softness of class label. Fig. 3(a) shows that distillation temperature $\tau$ in the range $\{2,20\}$ may be a better choice. It means that the lower temperature can’t effectively distill the similar information between classes, while the higher temperature destroys the prediction of various classes, so the temperature in the middle is the most appropriate. $\zeta$ and $\lambda$ show the proportion of target class knowledge and non-target class knowledge, respectively, as shown in Figs. 3(b) & (c). We use $\lambda/\zeta$ to show the proportion of them. Fig. 3(d) reveals that $\lambda/\zeta$ within $\{2,5\}$ may be a better option, which means that non-target class knowledge can better improve the efficiency of knowledge distillation, thus improving the performance of the model, but too much non-target class knowledge will also reduce the classification performance of the model, because the target class knowledge transfers the fuzzy dark knowledge about the sample difficulty. $\varphi$ in Fig. 3(e) exhibits the proportion of the knowledge contained in the ground-truth label, we use $(\lambda+\zeta)/\varphi$ to show the proportion of fuzzy dark knowledge and knowledge contained in the ground-truth label. Fig. 3(f) displays that $(\lambda+\zeta)/\varphi$ within $\{0.3,3\}$ may be a better choice, which means that a appropriate amount of fuzzy dark knowledge can improve the classification performance. We can conclude that when transferring too little fuzzy dark knowledge, student model cannot be effectively guided by teacher model, and when transferring too much fuzzy dark knowledge, student model will be misled by the mistakes made by teacher model.

The benefit of fuzzy classifiers is that they may be articulated in terms of linguistic explanation. Here, we choose one trial with the best $Accuracy$ on CLE among all experiment trials, and then, list in Table V the corresponding antecedent parts and consequent parts of all fuzzy rules. Due to space limitation, we only present three features of CLE dataset, i.e., the first, the second, and the last (13th) features of all three fuzzy rules. For a random given datum $\mathbf{x}=(-0.2730,-0.6799,\ldots,-16.4620)^{T}$, we can observed from Table V that, each fuzzy set A${}_{i}^{k}$ for the rule $k$ and feature $i$ in the If-Part of HTSK-LLM-DKD can be interpreted with a possible linguistic phrase. In the Then-Part of HTSK-LLM-DKD, the absolute value of the outputs in Rules 2 & 3 are much larger than that of Rule 1, which means that Rules 2 & 3 play a more important role in HTSK-LLM-DKD. In Rules 2 & 3, the 3rd output value is obviously the largest, after comprehensive calculation, we can conclude that the final prediction of HTSK-LLM-DKD is 3rd class, which means high-risk.