Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications

Motivated by the previous problem formulation, the purpose of this work is to find an answer to the central question:

Is it possible to devise an algorithm for online, distributed non-convex optimization problems with compressed exchange of zeroth-order information, and with provable convergence guarantees for both single-agent and multi-agent settings?

To address this question, we focus on a general stochastic nonconvex optimization problem, taking into account the following factors: i) access to the stochastic cost function is limited to zeroth-order oracle, meaning only function values at current locations and times are available, ii) due to communication constraints, only compressed or quantized gradients are exchanged between the agents and the server, iii) multiple agents use zeroth-order information to track their targets, and iv) the objective functions are time-varying in nature, resulting in an online optimization problem.

We prove the existence of a first-order solution in $\mathbb{R}^{d}$ that is $\xi$-accurate with $T=\mathcal{O}\left(\frac{d\sigma^{2}ML(\Delta+\bar{\omega})}{\xi^{2}}\right)$ in the dominant term, where $\sigma^{2}$, $L$, $M$, $\Delta$, and $\bar{\omega}$ denote the variance of stochastic gradients, smoothness constant in Assumption 3, bound constant on the stochastic gradients’ second moment in Assumption 2, the difference between averages of loss functions for the first and last iterates, and the summation of drift bounds from Assumption 4 respectively. Hence the dominant term in the convergence error is not dependent on the compression ratio. This is achieved while using an EF mechanism and a ZO gradient estimator which uses two function evaluations. In the derivation of this result, we also relax the assumption of bounded second moment commonly found in related literature [6]. Instead of assuming that the second moment of the stochastic gradients are upper-bounded by a constant term greater than or equal to their variance, we adopt the relaxed assumptıon that it is upper-bounded by the variance plus a term that is proportional to the square of its expected value. In other words, we relax the assumptions on the value of $M$ in Assumption 2, whereas it is commonly assumed in other literature that $M=0$, uniformly. That is, our upper bound depends on the current sample rather than a uniform bound. Whereas the previous work deals with a single-agent scenario [11], we examine the effectiveness of the proposed approach in a multi-agent target tracking scenario with limited communication where collision avoidance is of paramount importance. The problem of reducing collisions among agents is addressed by incorporating the FL paradigm and a new regularization term. This task is formulated as an online, distributed nonconvex optimization problem that can be solved by a multi-agent variation of the proposed scheme. Theoretical analysis shows that a $\xi$-accurate first-order solution in $\mathbb{R}^{Nd}$ with $T=\mathcal{O}\left(\dfrac{\sigma^{2}dMQ\left(\Delta^{2}+\bar{\omega}^{2}\right)+M\left(\sigma^{2}+Z^{4}\right)}{\xi^{2}}\right)$ in the dominant term can be found in a scenario with $N$ agents, where $Z^{2}$ and $Q$ are constants that arise from Assumption 6, which effectively places a bound on the norm of the gradients of each client in terms of the average of these gradients over all of the clients [12]. The results of the study are further supported by experimental results

Our preliminary work on single-agent convergence analysis and experiments was accepted to and will be presented at 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing[13]. The current work presents a significantly more thorough analysis of the subject, with an additional part detailing the multi-agent algorithm and its analysis, presented in Theorem 2. The complete proofs of the two theorems are also provided. The experimental section is ameliorated with more descriptive results, and an additional experiment involving an area coverage problem.

Communication-efficient FL. FedAvg is the seminal FL paper in which the central server takes the average of the local gradients transmitted by the clients and distributes the updated parameters to corresponding clients [1]. The crux of this work is the locality of data, in that data is acquired and trained on locally by a multitude of clients, without ever transporting it to a central server. Several difficulties, including privacy concerns [14], heterogeneity of client data [15], and high communication costs in agent-to-server links [16] arise in relation to this paradigm. Variations of FedAvg have been developed to mitigate these problems. For instance, to deal with high communication costs, [17, 18] propose a sparsification algorithm in communication for time-varying decentralized learning and optimization. Reference [19] proposes utilizing adaptive learning rate for aggregation, which is relevant to both the client data heterogeneity and communication efficiency issues. Reference [20] suggests using a novel aggregation technique which first quantizes gradients, and then skips communicating less impactful quantized gradients in favor of reusing previous ones. Reference [21] proposes using a momentum-based global update at the server, which promotes communication efficiency through variance reduction. Reference [22] proposes a derivative-free federated ZO optimization (FedZO) algorithm, and to improve its communication efficiency over wireless networks, they propose an over-air computation assisted variant. Reference [23] proposes a multiple local update strategy and a decentralized ZO algorithm to improve the communication efficiency and convergence rate in the decentralized FL scenario, in which there is no access to first-order derivatives. Reference [24] promotes the use of multiple gossip steps for communication efficiency. Various compression schemes such as Top-k [6], Rand-k [5], Biased and Unbiased Dropout-p [4], Quantized SGD (QSGD) [3], and their variants/generalizations are used to achieve the communication efficiency of FL algorithms. Compression schemes can be divided into contractive and non-contractive methods. With contractive compression schemes, which are our focus in this work, it is common to introduce an EF mechanism to compensate for the error due to compression by accumulating compression error in memory and adding it back as feedback for subsequent rounds. In [6], it is shown that such a method used in conjunction with SGD has a comparable rate of convergence to non-compressed SGD. In this study, we relax the assumption of having stochastic-first order oracle with bounded noise required in [6] by meticulously characterizing the impact of such relaxation on convergence. Furthermore, instead of the single-agent case as investigated in [6], we consider multiple agents with the additional contingency of preventing their collisions, which make the theoretical side more challenging.

Multi-agent target tracking. In our setting, agents are limited to ZO information, since they are assumed to only be able to sense their distance to their targets and other nearby agents. As a result of this consideration, our method is applicable to different practical scenarios such as [25, 26]. In these kinds of scenarios, gradients of the loss function can still be estimated by finite differences [27] but doing so in a multi-agent setting under communication constraints still remains an open challenge. Reference [11] describes a setup comparable to online optimization employing ZO oracles, applied to a target tracking problem. In that work, the authors focus on the case where there is a single source pursued by a single agent, which we generalize to the multi-agent setting as part of our contribution. We further investigate an effective approach via nonconvex regularization for collision avoidance. The $\lambda$ parameter we refer to as the regularization parameter is in essence similar to the penalty and augmented Lagrangian methods used in functional constrained optimization[28, 29]. However, these methods aim to adaptively tune the $\lambda$ parameter on-the-go, which is out of the scope of our work. It should be noted also that this line of research is very relevant to the area of safe reinforcement learning, see e.g. [30, 31, 32, 33].

In [34], a cooperative, mobile multi-agent source localization problem is tackled via using a distributed algorithm. Compared to our setting, the agents sense first-order information and their neighboring agents benefit from collaboration between agents to avoid collision. Reference [35] deals with a source localization problem in a single-agent and single-source setting, where the source is stationary or near-stationary. Reference [36] studies the problem of OL using ZO information with convex cost functions. They extend the problem out of the conventional Euclidean setting onto Riemannian manifolds. In [37] a ZO source localization problem is considered using distance information, where the agents are essentially multiple sensors of known position. References [38] and [39] deal with an online optimization problem using a decentralized network of multiple agents which have access to ZO information, and propose the usage of local information and information from neighboring agents in the network. In [38], an iterative algorithm with guarantees is proposed for time-varying online loss functions. The theoretical result there is established by assuming a certain bounded drift in time assumption which is standard in the literature, see, e.g., [11].

Online optimization and online target tracking. In the general online optimization setting, we focus on literature within or similar to the online convex optimization framework, which we can consider as a sequential decision making game in the presence of time-varying loss functions [40]. In [41], a distributed online optimization problem with multiple agents is considered, and the local loss function of each agent is convex and time-varying. The authors propose a randomized gradient-free distributed projected gradient descent procedure, where agents estimate the gradient of their local loss functions in a random direction using information from a locally-built ZO oracle. In [42], a similar setting is considered, and a multi-agent distributed optimization problem is studied in continuous-time, with time-varying convex loss functions. Reference [11] deals with a setting where zeroth-order oracles are used for optimization in the presence of time-varying cost functions. Besides the general online optimization setting, there is an abundance of literature focusing on the online optimization aspect of the target tracking problem. A large number of them also involve a swarm of multiple agents working in coordination. Usually, literature on this area tends to consider the problem in the context of unmanned aerial vehicles (UAV), or unmanned surface vehicles (USV). As pointed out in [43], the approaches to the problem may be separated into three broad categories: those using filtering based, control theory based and machine learning based approaches. For instance, [43] examines the problem within the domain of reinforcement learning by formulating it as a constrained Markov decision process, with application to autonomous target tracking using a swarm of UAVs. The authors provide an algorithm with provable guarantees. In [44], the authors again consider a multi-agent multi-target pursuit evasion scenario, where they propose the usage of a recurrent neural net work for target trajectory prediction, in conjunction with a multi-agent deep deterministic policy gradient formulation for decision making. Reference [45] deals with a robust formulation of a similar scenario in the domain of supervised learning, using a game theoretic approach. Reference [46] deals with a multi-target following scenario with consideration of external threats. The authors treat the problem as an online path planning problem and adopt a control-theoretic approach. In [47], an online adaptive Kalman filter is ussed in a target tracking problem where the sensor signals of the agents are assumed to have unknown noise statistics, to formulate a solution that is robust to noise. Lastly, [48] considers a decentralized control problem involving multiple agents with multiple control objectives, among which target tracking is one. The authors make use of a scheme based on adaptive dynamic programming, and feedback from a critic neural network which approximates the control objectives in online fashion.

Our work is focused on a nonconvex online distributed optimization problem with compressed exchange of zeroth-order information, along with the error feedback mechanism. Although these concepts were investigated individually in prior works [6, 11, 22, 27, 34] to the best of our knowledge, we are the first to combine them in a single framework and propose an algorithm with its theoretical analysis and convergence guarantee.

Furthermore, we may compare our theoretical results in Section 3 with the result of the analysis in [49], which derives an upper bound for the offline, first-order case with contractive compressors and the error feedback mechanism for the optimization of a smooth, nonconvex function in the FL paradigm. Their result establishes, in this setting, an iteration complexity of $\mathcal{O}(1/N\xi^{2})$ to produce a $\xi-$accurate first-order solution. Our result agrees with this result on the $\xi-$dependency. However, the analysis in this result is obtained in an offline setting with access to the first-order derivatives and is hence able to derive a convergence rate that is inversely proportional to the number of agents $N$. As opposed to that setting, we consider a more challenging online setting in which agents lack access to first-order derivatives but have access to finite differences. Thus, the convergence rate we establish is independent of $N$. Moreover, reference [49] assumes a uniform bound on the second moment of the gradients. We adopt a more relaxed version of this assumption, which does not assume a uniform bound on the second moment (Assumption 2), and this makes our analysis more involved. We make a similar relaxation of a standard assumption used in distributed optimization [50, 51, 52, 53, 54, 55, 56] in the same vein, by lifting the assumption of a uniform bound (Assumption 6).

The rest of the paper is organized as follows: In Section 2, we present the related background on stochastic gradient descent in zeroth-order oracle setting and necessary assumptions for our theoretical analysis. In Section 3, we propose the EF-ZO-SGD and FED-EF-ZO-SGD algorithms and present two theorems for their convergence along with sketches of proofs. Experimental results for two different settings are presented in Section 4 followed by the conclusion in Section 5. The complete proofs of the theorems along with the statements of relevant lemmas are presented in the Appendix.

We now present EF-ZO-SGD, an algorithm which uses compression along with the EF mechanism in addition to the ZO estimator in (2) to achieve a communication-efficient method of approaching the presented problem in the single-agent scenario. The complete algorithm is demonstrated in EF-ZO-SGD. Given an initial solution $x_{0}\in\mathbb{R}^{d}$, which for our problem represents the initial position of the agent, the algorithm works iteratively to construct subsequent solutions to the sequence of optimization problems. It first samples a random vector in $\mathbb{R}^{d}$ from the standard Gaussian distribution and uses this to construct a ZO estimator to the gradient (steps 3 and 4). Then, the error feedback vector, which keeps track of information discarded during compression in previous communication rounds (step 7) is added to this ZO estimator to produce the augmented estimator (step 5). In this manner, information previously lost to compression is re-utilized. The augmented estimator is the quantity used in the update rule to produce the subsequent solution (step 6), and it is further used to update the error feedback vector (step 7). This process is repeated for $t=1,...,T$ to produce solutions to all terms of the sequence of optimization problems.

The convergence properties of EF-ZO-SGD are analyzed next. For the convergence of EF-ZO-SGD in a single-agent setting, we establish Theorem 1.

Note that although the EF-ZO-SGD algorithm can be thought of as a SGD-type scheme, the analysis – due the interaction of EF and ZO estimation – is involved. In the proof, we leverage a new intertwined perturbation analysis, wherein we analyze the convergence of a virtual solution sequence to the smoothed functions $\ell_{\mu,t}$ and tie that to the performance of the real iterates $x_{t}$ to $\ell_{t}$, while utilizing the relaxed bounded stochastic gradient assumption.

We further note that (14) demonstrates that the dominant term in the complexity is independent of the compression parameter $\delta$. Therefore, for long sequences of time-varying optimization problems where $T$ is very large, the contribution of compression to the convergence error is negligible. Also notable is the fact that the complexity scales with dimension $d$. While this dependence is undesirable, in the worst case, it is unavoidable even without compression as shown in [62].

We may discuss the implication of our results to the setting of learning parameters of an overparameterized model, e.g., a deep learning predictor. It has been argued, see, e.g. [63, 64], such models typically satisfy a so-called strong growth condition which implies $\sigma=0$ in Assumption 2. That is, as the EF-ZO-SGD algorithm converges to a stationary solution, it enters into a virtuous cycle wherein the noise in the stochastic gradient reduces. As our analysis demonstrates, in such settings we can modify $\eta$ and $\mu$ accordingly (in particular set $\eta$ independent of T) to improve the complexity of the proposed algorithm to $T=\mathcal{O}(\frac{1}{\xi})$.

FED-EF-ZO-SGD algorithm is a generalization of EF-ZO-SGD to multi-agent and multi-target setting. In addition to the compression, EF mechanism, and ZO estimator, agents are coordinated with the central server and their compressed gradients are averaged in the server as in [1]. The complete algorithm is shown in FED-EF-ZO-SGD. Given an initial solution $x_{0}^{1:N}\in\mathbb{R}^{Nd}$, which in our problem represents the concatenation of the initial position of the agents, the FED-EF-ZO-SGD algorithm works iteratively on both the agent side and the server side to generate the consecutive solutions to the sequence of optimization problems. The agent side is similar to EF-ZO-SGD except for the content of the solution vectors. In our setting, without loss of generality, we consider agents which can sense the position of nearby agents called "neighbors" and merge the position vectors with their current position to obtain $x_{t}^{1:N}$. Entries that correspond to the other agents which are not neighbors are set to 0. The same algorithm can be implemented for the agents having no knowledge of the nearby agents’ positions. For every agent, the algorithm first samples a random vector in $\mathbb{R}^{Nd}$ from the standard Gaussian distribution and the entries that do not correspond to $i^{th}$ agent’s position are set to 0 (step 6). Thus, only $i^{th}$ agents position vector is perturbed to approximate the noisy gradient with finite differences (step 7). Steps 8 and 9 are the same as EF-ZO-SGD. Lastly, each agent sends its compressed augmented estimator to the central server. After the server collects all the estimators from every agent, it takes their average (step 12). Then, this average is used in the update (step 13) and the new positions are transmitted to the agents. This procedure is followed for $t=1,...,T$ to produce solutions to all terms of the sequence of optimization problems.

Now, we proceed with the analysis extended to the multi-agent case, which involves FED-EF-ZO-SGD. We state the following theorem:

Much like in the single-agent analysis, we note that the dominant term in the complexity is independent of the compression ratio $\delta$.

We begin with the application of our proposed FED-EF-ZO-SGD multi-agent target tracking scenario detailed in the previous sections. In all experiments, we instantiate a central server, $N$ agents $\{\mathcal{A}_{i}\}_{i=1}^{N}$ and $N$ sources $\{\mathcal{S}_{i}\}_{i=1}^{N}$. The initial location of each agent is chosen uniformly at random from $[-100,100]^{2}$ and each source from $[200,400]^{2}$. Hence, $d=2$, i.e., we consider the target tracking problem on a $2-$dimensional plane, which is reasonable for the motivating example of delivery robots. Also, we instantiate the agents and sources in two separate clusters, with some initial distance between them. Each agent $\mathcal{A}_{i}$ aims to track source $\mathcal{S}_{i}$ and each $\mathcal{S}_{i}$ actively evades its tracker with maximum speed. This setting generalizes that of [11] to a scenario with multiple agents and sources. We use $x^{i}_{t}$, $z^{i}_{t}$ to denote the positions of $\mathcal{A}_{i}$ and $\mathcal{S}_{i}$ at time step $t$. Each $\mathcal{S}_{i}$ aims to maximize its distance to $\mathcal{A}_{i}$, by setting its velocity at each step to $\zeta^{i}_{t}=\beta(z^{i}_{t}-x^{i}_{t})/\lVert z^{i}_{t}-x^{i}_{t}\rVert$, i.e., moving directly away from $\mathcal{A}_{i}$ with speed $\beta=0.1$.

An illustration of the movements of agents and sources is given in Fig. 2.

As explained in the previous sections, an additional contingency we introduce to the above setting is the requirement of a collision avoidance mechanism to prevent agents from unsafe maneuvers. To this end, we propose a two-level approach: i) on the local level within each agent, by means of local neighbor detection leveraging a judicious regularization term, and ii) coordination via the FL paradigm. With regard to i), at every step of the simulation, we calculate the set of neighbors of each $\mathcal{A}_{i}$ as $D^{i}_{t}\coloneqq\{j\neq i:\lVert x_{t}^{i}-x_{t}^{j}\rVert\leq r\}$, where we set $r=10$. These neighbor sets determine the local loss function $\ell^{i}_{t}$ of $\mathcal{A}_{i}$ at time step $t$, which we define as:

where $x^{1:N}_{t}=[(x^{1}_{t})^{T}\cdots(x^{N}_{t})^{T}]^{T}\in\mathbb{R}^{(Nd)}$ and $\lambda$ is the predetermined regularization parameter. We note that the time-varying nature of these neighbor sets introduce time-variance to the loss functions, which is exactly the setting we examine in the theoretical analysis. We divide the local loss function into two terms in order to simplify the notation in the subsequent calculation of the local ZO gradient estimator $g^{i}_{\mu,t}$:

where the loss due to source $s_{t}^{i}$ is given by $s_{t}^{i}(x_{t}^{i},z^{i}_{t})=\frac{1}{2}\lVert x_{t}^{i}-z_{t}^{i}\rVert^{2}$ and the loss due to regularization between agents $\mathcal{A}_{i}$ and $\mathcal{A}_{j}$, $r_{t}^{i,j}$, by $r_{t}^{i,j}(x_{t}^{i},x_{t}^{j})=\lambda(\lVert x_{t}^{i}-x_{t}^{j}\rVert^{2}-r^{2})$. In terms of the scenario, one could see the regularization term as agents being able to sense other agents within a radius $r$ around its position. With regard to ii), collision avoidance is ensured by means of federated aggregation of the local gradient estimators. The global loss function at $t$ is defined as $\bar{\ell}_{t}(x_{t}^{1:N},z_{t}^{1:N})=\frac{1}{N}\sum_{i=1}^{N}\ell_{t}^{i}(x_{t}^{1:N},z^{i}_{t})$, where $z^{1:N}_{t}$ is defined similarly to $x^{1:N}_{t}$.

Defining the neighborhood of two agents $\mathcal{A}_{i}$ and $\mathcal{A}_{j}$ in the above manner results in a symmetric relation. To make the setting more interesting, we also introduce the concept of neighbor dropout which aims to capture practical considerations such as imperfection in communication links and sensing capabilities. At each $t$, if $\mathcal{A}_{i}$ is to be added to $\mathcal{D}^{j}_{t}$, a random number $X$ is sampled from $U[0,1]$. If $X>p$, $i$ is added to $\mathcal{D}_{t}^{j}$, otherwise, it is dropped out. This leads to a more realistic scenario and opens up room for more meaningful collaboration between agents by breaking the symmetricity of the relation. If, for example, $\mathcal{A}_{i}$ is a neighbor of $\mathcal{A}_{j}$ but fails to detect it, we would expect $\mathcal{A}_{j}$ to compensate for this. Or worse, if both $\mathcal{A}_{i}$ and $\mathcal{A}_{j}$ fail to detect each other, we would expect $\mathcal{A}_{k}$ such that $k\in\mathcal{D}_{t}^{i}\cap\mathcal{D}_{t}^{j}$ to compensate for these detection failures. With the local loss function defined in (21), every agent $\mathcal{A}_{i}$ calculates a ZO gradient estimator $g^{i}_{\mu,t}$. Following the setup in [11], we slightly modify the computation of the ZO estimator, by introducing a small change in the argument of the first function evaluation. Let

where $u^{i,j}_{t}$ for all $j\in\mathcal{D}_{t}^{i}$ are drawn from $\mathcal{N}(0,I_{d})$ at time $t$ and $\zeta^{i}_{t},\xi^{i}_{t}$ denote the velocities of agent $\mathcal{A}_{i}$ and source $\mathcal{S}_{i}$ at time $t$, respectively. Similar to $\ell_{t}^{i}$ and (22), we divide $\ell^{i}_{t^{+}}$ into two terms:

where $r_{t^{+}}^{i,j}(x_{t}^{i},x_{t}^{j})=\lambda(\lVert x^{i}_{t}+\mu u^{i,j}_{t}-(x^{j}_{t}+0.5\xi^{j}_{t})\rVert^{2}-r^{2})$ and $s_{t^{+}}^{i}(x_{t}^{i},z^{i}_{t})=\frac{1}{2}\lVert x^{i}_{t}+\mu u^{i,i}_{t}-(z^{i}_{t}+0.5\zeta^{i}_{t})\rVert^{2}$.

Now, we define $g^{i}_{\mu,t}=[(g^{i,1}_{\mu,t})^{T}\cdots(g^{i,N}_{\mu,t})^{T}]^{T}\in\mathbb{R}^{(Nd)}$ where