Adaptive Data-Free Quantization

To measure the sample adaptability to Q, we focus primarily on: 1) the dependence on Q; 2) the advantage of P over Q; and 3) the disagreement between P and Q. Following [17], we first define the disagreement sample below:

As inspired, we revisit the DFQ as: G generates the sample with large adaptability to Q, to be equivalent to maximizing $\mathcal{H}_{nor}$; upon the generated samples, Q is updated to recover the performance with decreasing the sample adaptability, which is equivalent to minimizing $\mathcal{H}_{nor}$, adversarial to maximizing $\mathcal{H}_{nor}$. Such fact is in line with the principle of zero-sum game^2††footnotetext: ² For AdaDFQ, G aims to enlarge the disagreement, i.e., adaptability to Q, between P and Q, while Q decreases it by learning from P, where they cancel each other out, making the overall changing (summation) of the disagreement ($\mathcal{H}_{info}(p_{ds})$) close to 0 (see Fig.3 for such intuition), which is consistent to the intuition of zero-sum game.[20], as discussed in the next section.

On top of [17], we revisit the DFQ from a zero-sum game perspective over sample adaptability between two players — a generator and a quantized network, as follows:

$$\mathop{min}_{\theta_{q}\in\Theta_{q}}\mathop{max}_{\theta_{g}\in\Theta_{g}}\mathcal{H}(\theta_{g},\theta_{q})=\mathop{min}_{\theta_{q}\in\Theta_{q}}\mathop{max}_{\theta_{g}\in\Theta_{g}}\mathbb{E}_{z,y}[1-\mathcal{H}^{{}^{\prime}}_{info}(p_{ds})],$$

(4)

where $\theta_{g}\in\Theta_{g}$ and $\theta_{q}\in\Theta_{q}$ are the weight parameters of G and Q, respectively. Particularly, Eq.(4) is alternatively optimized via gradient descent during each iteration: $\theta_{q}$ is fixed while $\theta_{g}$ is updated to generate the sample with large adaptability by maximizing Eq.(4); alternatively, $\theta_{g}$ is fixed while $\theta_{q}$ is updated to calibrate Q over the generated sample by minimizing Eq.(4). The optimization process will reach a Nash equilibrium [2] $(\theta_{g}^{*},\theta_{q}^{*})$ when the following inequality

$$\mathcal{H}(\theta_{g},\theta_{q}^{*})\leq\mathcal{H}(\theta_{g}^{*},\theta_{q}^{*})\leq\mathcal{H}(\theta_{g}^{*},\theta_{q})$$

(5)

holds for all $\theta_{g}\in\Theta_{g}$ and $\theta_{q}\in\Theta_{q}$, where $\theta_{g}^{*}$ and $\theta_{q}^{*}$ are the parameters of G and Q under an equilibrium state. Maximizing Eq.(4) is equivalent to maximizing $\mathcal{H}(\cdot,\cdot)$ throughout learning G to generate the sample with largest adaptability (i.e., smallest $\mathcal{H}^{{}^{\prime}}_{info}(p_{ds})$).

However, such fact may incur:

The above facts indicate that the sample with either largest or lowest adaptability generated by maximizing Eq.(4) is not necessarily the best, incurring over-and-under fitting issues, which fail to be resolved by the above disagreement sample (Definition 1) since it focuses on the larger adaptability (i.e., smaller $\mathcal{H}^{{}^{\prime}}_{info}(p_{ds})$). To address the issues, we refine the maximization of Eq.(4) during the zero-sum game by proposing to balance disagreement sample with agreement sample, as discussed in the next.

One may wonder why AdaDFQ can generate the sample with adaptive adaptability. We answer the question by studying the generalization of Q trained on the generated samples from a statistic view. According to the VC theory [21, 12, 13], the classification error of a quantized network (Q) learning from the ground truth label (y) on the generated samples by G can be decomposed as

$\displaystyle R(f_{q})-R(f_{r})\leq O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qr}}})+\varepsilon_{qr},$

(14)

where $R(\cdot)$ is the error of a specific function. $f_{q}\in\mathcal{F}_{q}$ is the quantized network (Q) function and $f_{r}\in\mathcal{F}_{r}$ is the ground truth label ($y$) function. $\varepsilon_{qr}$ denotes the approximation error of the quantized network function class $\mathcal{F}_{q}$ (considering all possible acquired $f_{q}$ by optimizing Q) w.r.t. $f_{r}\in\mathcal{F}_{r}$ (a fixed learning target) on the generated samples during the training phase. $\alpha_{qr}$ denotes the learning rate for the given generated samples, particularly: when $\alpha_{qr}$ approaches $\frac{1}{2}$ (a slow rate) for the generated samples with the excessive information to Q; while approaching $1$ (a fast rate) for the generated samples with the limited information to Q. $O(\cdot)$ represents the estimation error of a network function over the real data during the testing phase. $|\cdot|_{C}$ measures the capacity of a function class and $n$ is the number of the generated samples from G. Similarly, let $f_{p}\in\mathcal{F}_{p}$ be the full-precision network (P) function, then

$\displaystyle R(f_{p})-R(f_{r})\leq O(\frac{|\mathcal{F}_{p}|_{C}}{n^{\alpha_{pr}}})+\varepsilon_{pr},$

(15)

where $\alpha_{pr}$ is related to the learning rate of P upon the ground truth label ($y$); $\varepsilon_{pr}$ denotes the approximation error of the full-precision network (P) function class $\mathcal{F}_{p}$ w.r.t. $f_{r}\in\mathcal{F}_{r}$. For data-free setting, Q is required to learn from P with the generated samples, which yields the following:

$\displaystyle R(f_{q})-R(f_{p})\leq O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})+\varepsilon_{qp},$

(16)

where $\alpha_{qp}$ is related to the learning rate of Q upon P, while $\varepsilon_{qp}$ is the approximation error of the quantized network (Q) function class $\mathcal{F}_{q}$ w.r.t. $f_{p}\in\mathcal{F}_{p}$. For such case, to study the classification error of Q learning from the ground truth label $y$ on the generated samples, we combine Eq.(15) and (16) [12, 13], leading to

	$\displaystyle R(f_{q})-R(f_{r})$	$\displaystyle=R(f_{q})-R(f_{p})+R(f_{p})-R(f_{r})$		(17)
		$\displaystyle\leq O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})+\varepsilon_{qp}+O(\frac{|\mathcal{F}_{p}|_{C}}{n^{\alpha_{pr}}})+\varepsilon_{pr}$
		$\displaystyle=\underbrace{O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})+O(\frac{|\mathcal{F}_{p}|_{C}}{n^{\alpha_{pr}}})}_{\text{Estimation error}}+\underbrace{\varepsilon_{qp}+\varepsilon_{pr}.}_{\text{Approximation error}}$

Evidently, Eq.(17) shows that, to benefit the generalization of Q, both estimation and approximation error should be reduced, so that a tighter upper bound can be captured, where we claim that to be consistent with the insights on the optimization of Eq.(12): first, as aforementioned, the reasons for the overflow of generalization error stem from: 1) the generated sample with too large $\mathcal{H}^{{}^{\prime}}_{info}(p_{ds})$ is not informative to calibrate Q, i.e., the small $\varepsilon_{qp}$ during the training stage and large $O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})$ during the testing stage, leading to overfitting issue for Q; 2) for the generated sample with too small $\mathcal{H}^{{}^{\prime}}_{info}(p_{ds})$, Q has no sufficient ability to learn informative knowledge from P, i.e., both $\varepsilon_{qp}$ during the training stage and $O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})$ during the testing stage are large, leading to underfitting issue for Q. Based on that, $\lambda_{l}$ and $\lambda_{u}$ in Eq.(10) along with the balance process ($\mathcal{L}_{bal}$) in Eq.(9) aim to avoid too large or small $\varepsilon_{qp}$ by optimizing the margin, thus the estimation error $O(\frac{|\mathcal{F}_{q}|_{C}}{n^{\alpha_{qp}}})$ can be reduced, i.e., overcoming the over-and-under fitting issues. Second, the above fact discussed the adaptability of single sample to Q, as aforementioned, BNS distribution information (Eq.(11)) of the training data extracted from P further facilitates calibrating Q for better generalization via the generated samples within a batch, which ensures the generated samples to be informative to P, i.e., decreasing $O(\frac{|\mathcal{F}_{p}|_{C}}{n^{\alpha_{pr}}})+\varepsilon_{pr}$ in Eq.(15) or (17), along with the category information from P regarding the disagreement and agreement samples.

Based on the above, the sample with adaptive adaptability to Q can be generated by optimizing Eq.(12), reducing both estimation and approximation error (obtaining a tighter upper bound) in Eq.(17), so as to generate the samples by G with adaptive adaptability which is beneficial to calibrating Q for better generalization by optimizing Eq.(13), where they are alternatively optimized during a zero-sum game until reaching a Nash equilibrium.

We validate AdaDFQ over three typical image classification datasets, including: CIFAR-10 and CIFAR-100 [8] contain 10 and 100 classes of images, which are split into 50K training images and 10K testing images; ImageNet (ILSVRC2012) [19] consists of 1.2M samples for training and 50k samples for validation with 1000 categories. For data-free setting, only validation sets are adopted to evaluate the performance of the quantized models (Q). We quantize pre-trained full-precision networks (P) including ResNet-20 for CIFAR, and ResNet-18, ResNet-50 and MobileNetV2 for ImageNet, via the following quantizer to yield Q:

For generation process, we construct the architecture of the generator G following ACGAN [15], which is trained via Eq.(12) using Adam [7] as an optimizer with a momentum of 0.9 and a learning rate of 1e-3. For calibration process, Q is optimized by minimizing Eq.(13), where SGD with Nesterov [14] is adopted as an optimizer with a momentum of 0.9 and weight decay of 1e-4. For CIFAR, the learning rate is initialized to 1e-4 and decayed by 0.1 for every 100 epochs; while it is 1e-5 (1e-4 for ResNet-50) and divided by 10 at epoch 350 on ImageNet. G and Q are alternatively trained for 400 epochs. The batch size is set to 16. For hyper-parameters, $\alpha_{ds}$ and $\alpha_{as}$ in Eq.(9); $\lambda_{l}$, $\lambda_{u}$, $\beta$ and $\gamma$ in Eq.(12) are empirically set to 0.2, 0.1, 0.1, 0.8, 1 and 1 (see our supplementary material for more parameter studies). All experiments are implemented with pytorch [16] via the code of GDFQ [22] and run on an NVIDIA GeForce GTX 1080 Ti GPU and an Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz.

To evaluate AdaDFQ, we offer practical insights into “why” AdaDFQ works, including the comparisons with the state-of-the-arts, ablation studies apart from visual analysis.

We verify the core idea of AdaDFQ — optimizing the margin to generate the sample with adaptive adaptability for better Q’s generalization under varied bit widths. We perform the experiments with ResNet-18 (Fig.3(a)) and ResNet-50 (Fig.3(b)) serving as both P and Q on ImageNet. Fig.3(a) illustrates that, compared to GDFQ [22] and Qimera [3], the disagreement (computed by Eq.(2)) between P and Q for AdaDFQ performs stably within a small range, i.e., the overall changing summation ($\Delta_{G}^{i}+\Delta_{Q}^{i}$) of the disagreement is close to 0, following the principle of zero-sum game (Sec.2.2), which confirms that the generated sample with adaptive adaptability by AdaDFQ is fully exploited to benefit Q, and the lower and upper bound constraints ($\lambda_{l}$ and $\lambda_{u}$ in Eq.(12)) avoid the generated sample with too large or small adaptability, which results in ridiculously large disagreement or agreement. Fig.3(b) reveals that AdaDFQ achieves better Q’s generalization with 3-bit and 5-bit precision unlike GDFQ, where the generated sample with adaptive adaptability succeeds in overcoming the underfitting (both training and testing loss are large) and overfitting (small training loss but large testing loss), confirming the analysis in Sec.2.4.

To verify the superiority of AdaDFG, we compare it with typical DFQ methods, i.e., GDFQ [22], ARC [25] and Qimera [3]: reconstructing the original data from P; ZAQ [11] focuses primarily on the adversarial sample generation rather than adversarial game process for AdaDFG; IntraQ [24] optimizes the noise to obtain fake sample without a generator; AIT [4] improves the loss function and gradients for ARC to generate better sample, denoted as ARC+AIT; AdaSG [17] focuses on the zero-sum game framework, serving as a special case of AdaDFQ.

Table 2 summarizes our following findings: 1) AdaDFQ obtains a significant and consistent accuracy gain over the state-of-the-arts, in line with our purpose of optimizing the margin to generate the sample with adaptive adaptability to Q (Sec.2.3). Impressively, AdaDFQ achieves at most 10.46%, 12.59% and 36.93% accuracy gains on CIFAR-10, CIFAR-100 and ImageNet. Notably, compared with GDFQ, ARC and Qimera where Q is independent of the generation process, AdaDFQ obtains accuracy improvement with a large margin, e.g., at least 0.35% gain (ResNet-20 with 5w5a on CIFAR-10), confirming the necessity of AdaDFQ over the sample adaptability to Q in Sec.1. Specifically, without regard for the sample adaptability, ZAQ suffers from a large performance loss caused by many unexpected generated samples, which are harmful to the calibration process of Q. AdaDFQ upgrades beyond AIT despite of the combination with ARC. As expected, AdaDFQ exhibits the obvious advantages over AdaSG, implying the benefits of optimizing the margin upon the zero-sum game. 2) AdaDFQ delivers the substantial gains for Q under varied bit-widths, confirming the importance of adaptive adaptability to varied Q (Sec.2.3). Especially for 3-bit situation, most of the competitors suffer from the poor accuracy or convergence, while AdaDFQ obtains at most 36.93% (ResNet-18 with 3w3a) accuracy gains.

To further show the intuition of the generated samples with adaptive adaptability to Q, we perform the visual analysis over MobileNetV2 serving as both P and Q on ImageNet by the similarity matrix (calculated as the $\ell_{1}$ norm between $p_{ds}$ of generated samples), along with the examples of generated samples (10 images per category) from 10 categories. Fig.5(c) illustrates that the generated samples by AdaDFQ exhibit a much larger similarity (the darker, the larger) than those by GDFQ [22], implying that the substantial samples with undesirbale adaptability by GDFQ exist against AdaDFQ. Fig.6 shows that the generated samples for different bit widths (i.e., 3 bit, 4 bit and 5bit) vary greatly, confirming the intuition of AdaDFQ — generating the sample with adaptive adaptability to varied Q (Sec.2.3); while the samples from varied categories differ greatly from each other, confirming that the category information is fully exploited (Sec.2.3.1); due to page limitation, see supplementary material for higher resolution.