Joint Data Deepening-and-Prefetching for Energy-Efficient Edge Learning

Consider an edge learning network comprising a pair of the edge server and the IoT sensor as a data collector (see Fig. 1). We aim at training a binary classifier using local data with two classes collected by the IoT sensor. Due to the IoT sensor’s limited computation capability, the edge server is requested to train a classifier with the local data offloaded from IoT sensor instead of training the classifier on the IoT sensor.

Consider $M$ samples measured at the sensor, denoted by $\mathbf{y}_{m}\in\mathbb{R}^{D}$, where $m$ is the index of data sample, i.e., $m=1,2,\cdots,M$. We assume that the class of each sample is known, denoted by ${c}_{m}\in\{0,1\}$. Each raw data sample’s dimension, say $D$, is assumed to be equivalent. The dimension $D$ is in general sufficiently high to reflect complex environments, which is known as an obstacle to achieve high-accuracy classification [8]. Besides, a large amount of energy is required to offload these raw data to the edge server. To overcome these limitations, these high dimensional raw data can be embedded into a low-dimensional space using data embedding techniques [9], such as principle component analysis (PCA) [10] and auto-encoder [11]. Specifically, given $F$ less than $D$, there exists a mapping function $\mathcal{F}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{F}$ such that

where $\mathbf{x}_{m}=\left[x_{m,1},\cdots,x_{m,F}\right]^{T}$ represents the embedded data with $F$ features. We assume that the edge device knows the embedding function $\mathcal{F}$, which has been trained by the edge server using the historical data set. We use PCA as a primary feature embedding technique due to its low computational overhead, while other techniques are straightforwardly applicable. Partial or all features of each embedded data are offloaded depending on the offloading and learning designs introduced in the sequel.

The entire offloading duration is slotted into $K$ rounds with $t_{0}$ seconds. The channel gain in round $k$ is denoted as $g_{k}$ with $g_{k}>0$. We assume that channel gains are constant over one time slot and independently and identically distributed (i.i.d.) over different rounds. Following the models in [12] and [13], the transmission power required to transmit $b$ bits in round $k$, denoted as $e_{k}$, is modeled by a monomial function and is given as $e_{k}=\lambda\frac{{(b/t)}^{\ell}}{g_{k}}$ where $\lambda$ is the energy coefficient, $\ell$ represents the monomial order, and $t$ is an allowable transmission duration for $b$ bits. The typical range for a monomial order is $2\leq\ell\leq 5$ because this order depends on the specific modulation and coding scheme. Then, the energy consumption in round $k$, which is the product of $e_{k}$ and $t$, is given as

It is shown that energy consumption is proportional to the transmitted data size $b$, and inversely proportional to the transmission time $t$. For energy-efficient edge learning, it is necessary to decrease the amount of transmitted data and increase the transmission time.

The proposed JD2P is a feature-by-feature offloading control for energy-efficient classifier training, built on the following definition.

By Definition 1, we can offload less amount of data required to train the classifier and the resultant energy consumption can be reduced if depths of all data are known in advance. On the other hand, each data sample’s depth can be determined after the concerned classifier is trained. Eventually, it is required to process each data’s depth identification and classifier training simultaneously to cope with the above recursive relation, which is technically challenging. To this end, we propose two key techniques summarized below.

This subsection deals with the threshold design $\bar{d}^{(k)}$ to categorize whether the concerned data sample $\mathbf{x}_{m}^{(k)}$ is ACI or CCI based on the $k$-depth classifier. The stochastic distribution of each class can be approximated in a form of $k$-variate Gaussian processes using the Gaussian mixture model (GMM) [14]. As shown in Fig. 4, the overlapped area between two distributions is observed. The data samples in the area are likely to be misclassified. We aim at setting the threshold $\bar{d}^{(k)}$ in such a way that most data samples in the overlapped area are included except a few outliners located in each tail.

To this end, we introduce the Mahalanobis distance (MD) [15] as a metric representing the distance from each instant to the concerned distribution. Given class $c\in\{0,1\}$, the MD is defined as

where $\mathbf{x}^{(k)}\sim\mathcal{N}(\boldsymbol{\mu}_{c}^{(k)},\boldsymbol{\Sigma}_{c}^{(k)})$ with $\boldsymbol{\mu}_{c}^{(k)}\in\mathbb{R}^{k}$ and $\mathbf{\Sigma}_{c}^{(k)}\in\mathbb{R}^{k\times k}$ being the distribution’s mean vector and covariance matrix respectively, which are obtainable through the GMM process. It is obvious that $\delta_{c}^{(k)}$ is a scale-free random variable and we attempt to set the threshold as the value whose cumulative distribution function (CDF) of $\delta_{c}^{(k)}$ becomes $p_{\mathrm{th}}$, namely,

Noting that the square of $\delta_{c}^{(k)}$ follows a chi-square distribution with $k$ degree-of-freedom, the CDF of this distribution for $r>0$ is defined as :

where $\Gamma$ is gamma function defined as $\Gamma(k)=\int_{0}^{\infty}t^{k-1}e^{-t}dt$ and $\gamma$ is the lower incomplete gamma function defined as $\gamma(k,r)=\int_{0}^{r}t^{k-1}e^{-t}dt$. In a closed-form, the threshold $\bar{\delta}_{c}^{(k)}$ can be given as

where $\mathscr{G}^{-1}$ represents the inverse CDF of chi-square distribution with $k$ degree-of-freedom. Due to the scale-free property, the threshold $\bar{\delta}_{c}^{(k)}$ is identically set regardless of the concerned class; thus, the index of class can be omitted, namely, $\bar{\delta}_{0}^{(k)}=\bar{\delta}_{1}^{(k)}=\bar{\delta}^{(k)}$. Given $\bar{\delta}^{(k)}$, the each distribution can be truncated as

Last, the threshold $\bar{d}^{(k)}$ is set by the maximum distance from the hyperplane in (3) to an arbitrary $k$-dimensional point $\mathbf{x}^{(k)}$ in the overlapped area of $\mathcal{R}_{0}$ and $\mathcal{R}_{1}$, given as

The process to obtain the threshold $\bar{d}^{(k)}$ is summarized in Algorithm 2.

After $K$ rounds, the entire classifier has a hierarchical structure comprising from $1$-depth to $K$-depth classifiers. Consider that a mobile device sends an unlabeled data sample to the edge server, which is initially set as an ACI. Starting from the $1$-depth classifier, the data sample passes through different depth classifiers in sequence until it is changed to an CCI. The last classifier’s depth is referred to as the data sample’s depth. In other words, its classification result becomes the final one.

Consider the prefetching duration in round $k$, say $\tau_{k}$, which is equivalent to the training duration of the $k$-depth classifier, as shown in Fig. 3. The number of data samples in $\mathbb{S}^{(k)}$ is denoted by $s_{k}$. Among them, $p_{k}$ data samples are randomly selected and their $(k+1)$-th features are prefetched. The prefetched data size is $\alpha p_{k}$, where $\alpha$ represents the number of bits required to quantize feature data²²2The quantization bit rate depends on the value of intensity. For example, one pixel of MNIST data has $255$ intensities and can be quantized 8 bits, i.e., $\alpha=8$.. Given the channel gain $g_{k}$, the resultant energy consumption for prefetching is

Here, the number of prefetched data $p_{k}$ is a discrete control parameter ranging from $0$ to $s_{k}$. For tractable optimization in the sequel, we regard $p_{k}$ as a continuous variable within the range, which is rounded to the nearest integer in practice.

Next, consider the offloading duration in round $(k+1)$, say $t_{k+1}=t_{0}-\tau_{k+1}$. Among the data samples in $\mathbb{S}^{(k+1)}$, a few number of data, denoted by $n_{k+1}$, remain after the prefetching. Given the channel gain $g_{k+1}$, the resultant energy consumption is

Note that $n_{k+1}$ is determined after the $k$-depth classifier is trained. In other words, $n_{k+1}$ is random at the instant of the prefetching decision. Denote $\rho_{k}$ as the ratio of a data sample in $\mathbb{S}^{(k)}$ being included in $\mathbb{S}^{(k+1)}$. Then, $n_{k+1}$ follows a binomial distribution with parameters $(s_{k}-p_{k})$ and $\rho_{k}$, whose probability mass function is $P(j)=\bigl{(}{{s_{k}-p_{k}}\atop j}\bigr{)}\rho_{k}^{j}(1-\rho_{k})^{s_{k}-p_{k}-j}$ for $j=0,\cdots,{s_{k}-p_{k}}$. Given $p_{k}$, the expected energy consumption is

where $\nu=\mathbb{E}[\frac{1}{g_{k+1}}]$ is the expectation of the inverse channel gain, which can be known a priori due to its i.i.d. property.

Last, summing up (12) and (14) is the expected energy consumption for the $(k+1)$-th feature when $p_{k}$ data samples are prefetched, leading to the following two-stage stochastic optimization:

The optimal prefetching policy can be designed by solving IV-A explained in the following subsection.

This subsection aims at deriving the closed-form expression of the optimal prefetching number $p_{k}^{*}$ by solving IV-A. The main difficulty lies in addressing the $\ell$-th moment $\mathbb{E}_{n_{k+1}}[n_{k+1}^{\ell}]$, of which the simple form is unknown for general $\ell$. To address it, we refer to the upper bound of the $\ell$-th moment in [16],

where $\mu_{n_{k+1}}=\left(s_{k}-p_{k}\right)\rho_{k}$ is the mean of the binomial distribution with parameters $(s_{k}-p_{k})$ and $\rho_{k}$. It is proved in [16] that the above upper bound is tight when the order $\ell$ is less than the mean $\mu_{n_{k+1}}$. Therefore, instead of solving IV-A directly, the problem of minimizing the upper bound of the objective function can be formulated as

Note that P2 is a convex optimization, enabling us to derive the closed-form solution. The main result is shown in the following proposition.