Edge Learning for Large-Scale Internet of Things With Task-Oriented Efficient Communication

The initial attempts of task-oriented communications were to design task-aware transmission phases rather than end-to-end data reconstruction, see, e.g., [10, 11, 12] for designing task-aware reporting phases in the case of distributed inference tasks. Unlike task-aware efficient transmissions, several pioneering works [13, 14, 15, 16, 17] have also studied task-oriented schemes in edge learning systems. The work [13] designed a task-oriented communication scheme to realize a trade-off between preserving the relevant information and fitting with bandwidth-limited edge inference nicely. In [14, 15], task-oriented methods were proposed to maximize learning accuracy by jointly designing sensing, communication, and computation. The work [16] proposed a task-oriented transmission scheme to accelerate learning processes efficiently by capturing the semantic features from the correlated multimodal data of the IoT users. Nevertheless, these task-oriented communication schemes are mostly heuristics based, and it is necessary to improve their learning performance via additional optimization.

To overcome the drawback of heuristic methods, the work [17] proposed a learning-centric power allocation (LCPA) model to guide power allocation efficiently so as to optimize the limited network resources under the task-oriented principle. First, the learning performance was approximated by parameter fitting to capture the shape of learning models. Then, a majorization minimization (MM) algorithm was designed to allocate transmit power efficiently. However, the MM algorithm is inefficient for large-scale IoT networks due to its high computational complexity and lack of co-channel interference (CCI) management. In particular, for massive IoT users, it is necessary to collect multi-modal datasets and process heterogeneous learning tasks concurrently, making it imperative for designing parallel and low-complexity task-oriented communication algorithms. Furthermore, concurrent transmissions of massive IoTs will inevitably yield severe CCI, thus degrading the performance of task-oriented communications [18]. But due to the highly-coupled CCIs, the associated power allocation problem is non-separable and non-convex, which is non-trivial to realize the inference management and algorithm parallelization.

To fill the gap, this paper designs a task-oriented power allocation model for efficient communications in large-scale IoT networks with edge learning. On the one hand, as a task-oriented learning system involves heterogeneous learning tasks, it is necessary to predict the required resources for training different tasks. Therefore, our method is designed as an offline learning procedure that fits historical datasets to a performance prediction model and an online inference procedure that guides the IoT-edge communications with the pre-trained performance model. Note that this performance model can be fine-tuned by exploiting a small amount of real-time data from active IoT users. On the other hand, we formulate a task-oriented power allocation problem to guide communication-efficient data collection for large-scale IoT networks. To alleviate CCI, multi-user scheduling is first performed before power allocation. Then, a highly parallel algorithm is designed for different learning tasks. Lastly, we develop an accelerated algorithm to make the parallel algorithm more efficient. In brevity, Table I compares the existing and proposed schemes.

Aiming at efficient communication for task-oriented edge learning, this paper starts with a multi-user scheduling strategy to mitigate CCI. In particular, a relaxation-and-rounding algorithm is exploited to identify scheduled users efficiently, and an approximate closed-form solution is obtained. Secondly, a parallel algorithm with Gauss-Seidel methods is developed. By a set of variable decompositions, we realize a highly parallel iteration. Thirdly, we design an accelerated algorithm to speed up this parallel algorithm. Finally, extensive experimental results demonstrate the efficiency of our design. In summary, the main contributions are as follows:

• 1)

  A task-oriented power allocation model is proposed to process multiple distinct datasets at the edge. Moreover, a multi-user scheduling strategy is performed before power allocation to mitigate CCI for large-scale IoT networks efficiently.

• 2)

  A highly parallel algorithm is designed for the task-oriented power allocation problem. By variable decompositions and eliminating auxiliary variables, the power allocation in the presence of CCI is realized efficiently in parallel.

• 3)

  An accelerated algorithm is developed to make the parallel algorithm more efficient for large-scale IoT networks. Specifically, this algorithm utilizes the Lipschitz continuous property of the learning error and the identity mapping of the gain matrix to improve the convergence.

• 4)

  Extensive experimental results show that the multi-user scheduling strategy can mitigate CCI in large-scale IoT networks. Moreover, our parallel and accelerated algorithms efficiently solve task-oriented power allocation problems with a significantly shorter computation time than the existing algorithms.

The rest of this paper is organized as follows. Section \@slowromancapii@ describes the system model and formulates a task-oriented power allocation problem. Section \@slowromancapiii@ performs multi-user scheduling to mitigate CCI and designs a parallel algorithm for solving the task-oriented power allocation problem. Section \@slowromancapiv@ develops an accelerated algorithm for large-scale IoT networks. Section \@slowromancapv@ discusses the experimental results, and finally, Section \@slowromancapvi@ concludes the paper.

Notation: Scalars, column vectors, and matrices are denoted by regular italic letters, lower- and upper-case letters in bold typeface, respectively. The symbol $\bm{\mathsf{1}}$ indicates a column vector with all entries being unity. The superscripts $(\cdot)^{T}$ and $(\cdot)^{H}$ denote the transpose and Hermitian transpose of a vector or matrix, respectively, and the subscript $\|\bm{x}\|_{2}$ denotes the two-norm of $\bm{x}$. The abbreviation $\mathcal{CN}(\bm{0},\,\varrho\bm{I})$ stands for a multi-variable complex Gaussian distribution with mean vector $\bm{0}$ and variance matrix $\varrho\bm{I}$. The notation $\left|\mathcal{X}\right|$ denotes the cardinality of the set $\mathcal{X}$, and $\mathcal{Y}\setminus\mathcal{X}$ denotes the complement of set $\mathcal{Y}$ except for $\mathcal{X}$. The matrix operation $\bm{A}(\mathcal{K}_{i},\,\mathcal{K}_{j})$ denotes a sub-matrix of size $|\mathcal{K}_{i}|\times|\mathcal{K}_{j}|$ that includes the rows and columns in $\bm{A}$ specified by the sets of indices $\mathcal{K}_{i}$ and $\mathcal{K}_{j}$, respectively. The arithmetic operations $\bm{x}\succeq\bm{y}$, $\bm{x}\circ\bm{y}$, and $\left\langle\bm{x},\,\bm{y}\right\rangle$ denote that each element of $\bm{x}$ is greater than or equal to the counterpart of $\bm{y}$, the Hadamard product, and the inner product of two vectors, respectively. The Landau notation $\mathcal{O}(\cdot)$ denotes the order of arithmetic operations. Further, we define a binary function $\lfloor w\rceil=1$ if $w\geq 0.5$, and $\lfloor w\rceil=0$ otherwise, and $\lfloor\bm{w}\rceil\triangleq\left[\lfloor w_{1}\rceil,\,\lfloor w_{2}\rceil,\,\cdots,\,\lfloor w_{K}\rceil\right]^{T}\in\mathbb{R}^{K\times 1}$ is computed for each element of $\bm{w}$. Finally, the floor function $\left\lfloor x\right\rfloor\triangleq\max\{n\in\mathbb{Z}:n\leq x\}$, where $\mathbb{Z}$ is the set of integers.

Figure 1 illustrates a task-oriented edge learning system consisting of an edge server equipped with $N$ antennas, $I$ different learning tasks $\{\mathcal{T}_{1},\,\mathcal{T}_{2},\,\cdots,\,\mathcal{T}_{I}\}$ with corresponding user sets $\mathcal{K}\triangleq\{\mathcal{K}_{1},\,\mathcal{K}_{2},\,\cdots,\,\mathcal{K}_{I}\}$ and power allocation parameters $\bm{p}\triangleq\left[\bm{p}_{1}^{T},\,\bm{p}_{2}^{T},\,\cdots,\,\bm{p}_{I}^{T}\right]^{T}\in\mathbb{R}^{K}$, where $\bm{p}_{i}\triangleq\left[p_{i_{1}},\,p_{i_{2}},\cdots,\,p_{|\mathcal{K}_{i}|}\right]^{T}$, $i\in\mathcal{I}\triangleq\{1,\,2,\,\cdots,\,I\}$, $i_{j}\in\mathcal{K}_{i}$, $j\in\{1,\,2,\cdots,|\mathcal{K}_{i}|\}$, $K\triangleq|\mathcal{K}|$, and $p_{k}$, $k\in\mathcal{K}$, denotes the transmit power of the $k^{\rm th}$ user. As for the $I$ different learning tasks in Fig. 1, each of them concerns a set of transmitted data, a multi-user scheduling algorithm, a learning model, a process of parameter fitting for a learning model, and a task-oriented power allocation problem.

To improve the performance of edge learning, multi-user scheduling is adopted to alleviate CCI in large-scale IoT networks, and task-oriented power allocation is performed to implement efficient communications. To maximize the network utility function of long-term average data rates, by recalling the seminal Shannon formula, the achievable data rate of user $k$ in the presence of multi-user scheduling can be expressed as [8]

where $\Pi_{\mathcal{S}}(\cdot)$ denotes a projection function of multi-user scheduling; $\sigma^{2}$ is the variance of additive white Gaussian noise; $G_{k,\ell}$ represents the composite channel gain from the $\ell^{\rm th}$ user to the edge server when decoding data of the $k^{\rm th}$ user, computed as $G_{k,k}=\rho_{k}\|\bm{h}_{k}\|_{2}^{2}$ if $\ell=k$, and $G_{k,\ell}={\rho_{\ell}|\bm{h}_{k}^{H}\bm{h}_{\ell}|^{2}}/{\|\bm{h}_{k}\|^{2}_{2}}$ if $\ell\neq k$, with $\bm{h}_{k}\in\mathbb{C}^{N\times 1}$ being the complex-valued channel fast-fading vector from the $k^{\rm th}$ user to the edge server and $\rho_{k}$ being the path loss of the $k^{\rm th}$ user. By (1), the number of samples transmitted by user $k$ for the learning task $\mathcal{T}_{i}$ at the edge server can be computed as

where $B$ is the total bandwidth in Hz; $T$ is the transmission time in seconds; $V_{i}$ is the number of bits for each data, and $A_{i}$ is the initial number of historical data for the $i^{\rm th}$ pre-trained task.

This paper considers the average channel over a long transmission period instead of assuming a static channel. The reason is twofold. On the one hand, to fine-tune diverse learning models, it is essential to require a relatively long transmission time with tens or hundreds of seconds to obtain a large number of datasets. On the other hand, the effect of multi-user scheduling can only be disclosed in the context of a long-term channel average rather than an instantaneous channel realization. Assume that the transmission period consists of different time slots. The channels are quasi-static during each time slot and vary in consecutive time slots. Therefore, $G_{k,k}$ and $G_{k,\ell}$ in (1) could denote the average channels gain during these slots.

To establish a connection between wireless resource allocation and the performance of machine learning, the work [17] conceived a non-linear exponential function $\Theta_{i}(D_{i}|a_{i},\,b_{i})\triangleq a_{i}D_{i}^{-b_{i}}$ to capture the shape of the learning error function, where $a_{i}$ and $b_{i}$ are tuning parameters that denote the model complexity and account for the non-independent and identically distributed (n.i.i.d.) parallel datasets, respectively. In practice, the values of $a_{i}$ and $b_{i}$ are obtained by fitting the learning error function from the historical dataset. This fitted function matches the experimental data of the machine learning model very well. In line with this idea and multi-user scheduling, we formulate a task-oriented power allocation problem:

where $\lambda_{i}\triangleq A_{i}V_{i}/(\sum_{j\in\mathcal{I}}A_{j}V_{j})$ is a weight of diverse datasets. In general, the power allocation of all scheduled users shall satisfy $\sum_{k\in\mathcal{K}}p_{k}\leq P$, i.e., not exceeding the total power budget $P$. As a larger value of $\sum_{k\in\mathcal{K}}p_{k}$ always improves the learning performance, (3b) is obtained [17]. (3c) means to schedule a subset of users, and (3d) implies no power is allocated to inactive users.

Compared to the conventional min-max objective function used in [17], it only focuses on the worst learning task, even if the task is not critical for real-world application. Thus, it is not suitable for the multi-task multi-modal scenario considered in this paper. Instead, the weighted sum model in (3a) can optimize multiple tasks simultaneously. In particular, the objective function can adapt to different learning tasks by adjusting the weight factors $\lambda_{i},i\in\mathcal{I}$ in (3a).

In the task-oriented learning system, multi-user scheduling is an effective strategy for solving the massive-connectivity problem. Traditional approaches to multi-user scheduling are almost non-convex algorithms, e.g., greedy heuristic search. In particular, as the number of scheduling cases increases exponentially with the number of users, it is hard to enumerate all possible subsets of users explicitly [20]. To deal with this problem, we introduce binary variables $w_{k}$, $k\in\mathcal{K}$, to replace $\Pi_{\mathcal{S}}$ defined immediately after (1). More specifically, $w_{k}=1$ if $k\in\Pi_{\mathcal{S}}\left(\mathcal{K}\right)$, and $w_{k}=0$ otherwise. As a result, given $p_{k}$, inserting (1)-(2) into $\mathcal{P}1$, the multi-user scheduling problem in the $i^{\rm th}$ group, $i\in\mathcal{I}$, is formulated as

where $R_{k}=\log_{2}(1+{G_{k,k}p_{k}}/(\sum_{\ell\in\mathcal{K}\setminus k}w_{\ell}G_{k,\ell}p_{\ell}+\sigma^{2}))$ as per (1), $\bm{w}\triangleq\left[w_{1},\,w_{2},\,\cdots,\,w_{K}\right]^{T}$, and (5c) is derived from (3c) with $N_{i}$ being the maximal allowed number of active users for the $i^{\rm th}$ learning task.

To solve $\mathcal{P}2$, we adopt a relaxation-and-rounding algorithm [21]. First, we relax the binary constraint (5b) as the real-valued constraint $0<w_{k}\leq 1$. Then, we provide the following Proposition 1 to obtain an approximate closed-form solution to the relaxed version of $\mathcal{P}2$.

Proposition 1 shows that the multi-user scheduling decision is analytically determined, with extremely low computational complexity proportional to the number of users, i.e., $\mathcal{O}(K)$. Also, it is noted that our multi-user scheduling strategy is fair with respect to different learning tasks, but the fairness among users is not accounted for since it is beyond the scope of this paper.

After the multi-user scheduling is performed by Proposition 1, the task-oriented power allocation problem can be rewritten as

where $\tilde{R}_{k}\triangleq\log_{2}(1+{G_{k,k}p_{k}}/(\sum_{\ell\in\mathcal{K}\setminus k}\tilde{w}_{\ell}G_{k,\ell}p_{\ell}+\sigma^{2}))$; (7b) is the relaxation of (3d), which means little power is reserved for inactive users.

The optimization problem $\mathcal{P}3$ is non-convex; even worse, its computational complexity rises with the number of users and tasks. To address these issues, we propose a parallel first-order algorithm. As there is a dependency on the power and CCI terms amongst different learning tasks, it is hard to realize parallelization for various tasks in a straightforward manner. An efficient strategy is to introduce auxiliary variables to separate these terms independently. The resulting problem involves a set of sub-problems by variable decompositions, and these sub-problems are easier to be solved in parallel [22, 23].

Now, we begin to extract the relevant sub-problems. First, to divide the interference term, we introduce additional variables $\bm{\delta}$, defined as

where $\bm{\delta}=\left[\bm{\delta}_{1}^{T},\,\bm{\delta}_{2}^{T},\,\cdots,\,\bm{\delta}_{I}^{T}\right]^{T}$ with $\bm{\delta}_{i}\triangleq[\delta_{i_{1}},\,\delta_{i_{2}},\,\cdots,\,\delta_{i_{|\mathcal{K}_{i}|}}]^{T}$, and $\bm{\Delta}\triangleq\tilde{\bm{G}}-\tilde{\bm{D}}$ with $\tilde{\bm{G}}\triangleq\left[\begin{array}[]{ccc}\bm{G}(1,\,:)^{T}\circ\tilde{\bm{w}}&\cdots&\bm{G}(K,\,:)^{T}\circ\tilde{\bm{w}}\end{array}\right]^{T}$ and $\tilde{\bm{D}}\triangleq\left[\begin{array}[]{ccc}\bm{D}(:,\,1)\circ\tilde{\bm{w}}&\cdots&\bm{D}(:,\,K)\circ\tilde{\bm{w}}\end{array}\right]$. Here, the $(k,\,\ell)^{\rm th}$ element of $\bm{G}$ is made up of $G_{k,\,\ell}$, and the $k^{\rm th}$ diagonal element of the diagonal matrix $\bm{D}$ is made up of $G_{k,\,k}$. By partitioning users $\mathcal{K}$ into $I$ groups of users $\{\mathcal{K}_{i}\}_{i=1}^{I}$ and introducing a set of variables $\{\bm{z}_{i}\}_{i=1}^{I}$ and $\{P_{i}\}_{i=1}^{I}$, we have

where $\bm{z}_{i}\triangleq\left[z_{1,i},\,z_{2,i},\,\cdots,\,z_{K,i}\right]^{T}$, and $\bm{p}_{i}\triangleq[p_{i_{1}},\,p_{i_{2}},\,\cdots,\,p_{i_{\left|\mathcal{K}_{i}\right|}}]^{T}$. It is noteworthy that $\bm{z}_{i}$ in (9a)-(9b) and $P_{i}$ in (9c)-(9d) are auxiliary variables. As a result, $\mathcal{P}3$ can be transformed into:

where (10b) is naturally satisfied as $\sum_{\ell\in\mathcal{K}\setminus k}\tilde{G}_{k,\ell}p_{\ell}\geq 0$ and $p_{\ell}\geq 0$. Specially, $\Phi_{i}(\bm{p}_{i}|\bm{\delta}_{i})$ in (10a) is explicitly given by

It is clear that $\mathcal{P}4$ separates the interference term and power constraint by introducing auxiliary variables; thus, it is beneficial to the parallelization of algorithm design. However, as there are multiple auxiliary variables and constraints in $\mathcal{P}4$, they will linearize the augmented terms and slow down the convergence. Even worse, convergence may not be guaranteed if there are more than two variables.

To address this issue, we propose a method to eliminate auxiliary variables [24, pp. 249-251]. First, the augmented Lagrangian function (ALF) of $\mathcal{P}4$ can be written as

where $\mu$ is an increasing positive sequence $\{\mu(t)\}$ about the iteration. From (11), it is observed that (9b) and (9d) are not directly considered in the ALF since different tasks are correlated. As will be shown shortly, this new ALF term allows the sub-problems to be solved in parallel. It is noteworthy that this algorithm differs from the conventional alternating direction method of multipliers (ADMM) algorithms, and its convergence is guaranteed [24, p. 255]. Given (11), we have the following proposition.

By Proposition 2, it is evident that we have eliminated auxiliary variables and decreased the dimension of dual variables. Next, we split the ALF given by (11) with respect to $\bm{p}$ and $\bm{\delta}$.

We have derived the ALF of $\mathcal{P}4$ and obtained a set of sub-functions to realize a parallel algorithm for different learning tasks. Now, we compute partial derivatives of relative variables and then apply the gradient descent algorithms in parallel.

In principle, the essence of our accelerated algorithm is to use an identical transform of matrices to split variable blocks. Using a fast proximal linearized ADMM algorithm with parallelizable splitting [26], we derive ALFs associated with variable blocks $\bm{p}_{i}$ and $\bm{\delta}_{i}$, respectively.