Network-Aware Optimization of
Distributed Learning for Fog Computing

We use $D_{i}(t)$ to denote the set of data collected by device $i\in V(t)$ for the ML task at time $t$; $d\in D_{i}(t)$ denotes each datapoint. Note, $D_{i}(t)=\emptyset$ if a device does not collect data at time $t$. $G_{i}(t)$, by contrast, denotes the set of datapoints processed by each device at time $t$, for the ML task. In conventional distributed learning frameworks, $D_{i}(t)=G_{i}(t)$, as all devices process the data they collect [5]; separating these variables is one of our main contributions. We suppose that each device $i$ can process up to $C_{i}(t)$ datapoints at each time $t$, incurring a cost of $c_{i}(t)$ for each point. For example, devices with low battery will have lower capacities $C_{i}(t)$ and higher costs $c_{i}(t)$.

The devices $V$ are connected to each other via a set $E$ of directed links $(i,j)$ between devices $i$ and $j$; $E(t)\subseteq E$ denotes the set of functioning links at time $t$. The overall system can then be described as a directed graph $(\left\{s,V\right\},E)$ with vertices $V$ representing the devices and edges $E$ the links between them. We suppose that $(\left\{s,V(t)\right\},E(t))$ is connected at each time $t$ and that links between devices are single-hop, i.e., devices do not use each other as relays except possibly to communicate with the server. The scenarios outlined in Section I-A each possess such an architecture: in smart factories, for example, a subset of the floor sensors connect to each controller. Each link $(i,j)\in E(t)$ is characterized by a capacity $C_{ij}(t)$, i.e., the maximum datapoints transferable in a time interval, and a per-unit “cost of connectivity” $c_{ij}(t)$. This cost may reflect network conditions or a desire for privacy, and will be high if sending from $i$ to $j$ is less desirable at $t$.

Each datapoint $d$ takes the form $(x_{d},y_{d})$, where $x_{d}$ is an attribute/feature vector and $y_{d}$ is an associated label.¹¹1For unsupervised ML tasks, there are no labels $y_{d}$. However, we can still minimize a loss defined in terms of the $x_{d}$, similar to (1). We use $D_{V}=\cup_{i,t}D_{i}(t)$ to denote the full set of datapoints collected by all devices over all time. Following prior work [37, 38], we model data collection as each device $i$ sampling points uniformly at random from a (usually unknown) distribution $\mathcal{D}_{i}$. Temporal changes in $\mathcal{D}_{i}$ are assumed to be slow compared to the time horizon $T$. We use $\mathcal{D}=\cup_{i}\mathcal{D}_{i}$ to denote the global distribution induced by these $\mathcal{D}_{i}$. Our model implies that the relationship between $x_{d}$ and $y_{d}$ is temporally invariant, which is common in the applications discussed in Section I-A, e.g., image recognition from road cameras at fixed locations or AR users with random mobility patterns. We use such a dataset for evaluation in Section V.

To distributedly solve (1), we first decompose it into a weighted sum of local loss functions

where $G_{i}\equiv\cup_{t\leq T}G_{i}(t)$ denotes the set of datapoints processed by device $i$ over all times. The global loss (1) is then $L(w|D_{V})=\sum_{i}L_{i}(w|G_{i})\left|G_{i}\right|/\left|D_{V}\right|$ if $\cup_{i}G_{i}=D_{V}$, i.e., all datapoints $d\in D_{V}$ are eventually processed.

Loss functions such as (2) are typically minimized using gradient descent techniques [10]. Specifically, the devices update their local parameter estimates at $t$ according to

where $\eta(t)>0$ is the step size or learning rate, and $\nabla L_{i}(w_{i}(t-1)|G_{i}(t))=\sum_{d\in G_{i}(t)}\nabla l(w_{i}(t-1),x_{d},y_{d})/|G_{i}(t)|$ is the gradient with respect to $w$ of the average loss of points in the current dataset $G_{i}(t)$ at the parameter value $w_{i}(t-1)$. We define the loss only on $G_{i}(t)$ since future data in $G_{i}$ has not yet been revealed; since we assume each node’s data is i.i.d. over time, $L_{i}(w_{i}(t-1)|G_{i}(t))$ approximates the local loss $L_{i}(w_{i}|G_{i})$. The computational cost $c_{i}(t)$ of processing datapoint $d$ is then the cost of computing the gradient $\nabla l(w_{i}(t-1),x_{d},y_{d})$. If the local data distributions $\mathcal{D}_{i}$ are all the same, then all datapoints are i.i.d. samples of this distribution, and this process is similar to stochastic gradient descent with batch size $|G_{i}(t)|$.

The aggregation server $s$ will periodically receive the local estimates $w_{i}(t)$ from the devices, compute a global update based on these models, and synchronize the devices with the global update. Formally, the $k$th aggregation is computed as

where $\tau$ is the fixed aggregation period and $H_{i}(k\tau)=\sum_{t=(k-1)\tau+1}^{k\tau}|G_{i}(t)|$ is the number of datapoints node $i$ processed since the last aggregation. Thus, the update is a weighted average factoring in the sample size $H_{i}(t)$ on which each $w_{i}(t)$ is based. Intuitively, since the objective in (1) minimizes the empirical loss [5, 10], nodes that process more data should be weighted more, as in (1)-(4).

Once this is computed, we synchronize local estimates, i.e., $w_{i}(t)\leftarrow w(t/\tau)$ $\forall i$. Using a smaller $\tau$ generally results in faster convergence of $w$, while larger $\tau$ requires less network resources. Prior work [5] considered how to optimize $\tau$; we analyze its effect experimentally in Section V.

We define $s_{ij}(t)\in[0,1]$ as the fraction of data collected at device $i$ that is offloaded to device $j\neq i$ at time $t$. Thus, at time $t$, device $i$ offloads $D_{i}(t)s_{ij}(t)$ amount of data to $j$. Similarly, $s_{ii}(t)$ will denote the fraction of data collected at time $t$ that device $i$ also processes at time $t$. We suppose that as long as $D_{i}(t)s_{ij}(t)\leq C_{ij}(t)$, the capacity of the link between $i$ and $j\neq i$, then all offloaded data will reach $j$ within one time interval and can be processed at device $j$ in time interval $t+1$. Since devices must have a link between them to offload data, $s_{ij}(t)=0$ if $(i,j)\notin E(t)$.

We also define $r_{i}(t)\in[0,1]$ as the fraction of data collected by device $i$ at time $t$ that will be discarded. In doing so, we assume that device $j$ will not discard data that has been offloaded to it by others, since that has already incurred an offloading cost $D_{i}(t)s_{ij}(t)c_{ij}(t)$. The amount of data collected by device $i$ at time $t$ and discarded is then $D_{i}(t)r_{i}(t)$, and the amount of data processed by each device $i$ at time $t$ is

In defining the variables $s_{ij}(t)$ and $r_{i}(t)$, we have implicitly specified the constraint $r_{i}(t)+\sum_{j}s_{ij}(t)=1$: all data collected by device $i$ at time $t$ must either be processed by device $i$ at this time, offloaded to another device $j$, or discarded. We assume that devices will not store data for future processing, which would add another cost component to the model.

To quantify the cost of discarding relative to processing/offloading, we also define $f_{i}(t)$ as the cost per unit model loss $L(w_{i}(t)|D_{V})$. The dependence on $i$ can capture the fact that certain devices in the network may weight their model error costs differently depending on the importance of the application. The dependence on $t$ can capture the fact that the loss may become less important relative to the resource utilization as the model approaches convergence.

We formulate the following cost minimization problem for determining the data movement variables $s_{ij}(t)$ and $r_{i}(t)$ over the time period $T$:

Constraints (6–8) were introduced above and ensure that the solution is feasible. The capacity constraints in (9) ensure that the amounts of data transferred and processed are within link and node capacities, respectively.

The three terms in the objective (5) correspond to the processing, offloading, and error costs, respectively. We do not include the cost of communicating parameter updates to/from the server in our model; unless a device processes no data, the number of updates stays constant. By including these three cost terms, our formulation accounts for the tradeoff between model performance and resource utilization:

Since $w_{i}(t)$ is computed as in (3), the model error term is an implicit function of $G_{i}(t)$. We include the error from each device $i$’s local model at each time $t$, instead of considering the error of the final model, since devices may need to make use of their local models as they are updated (e.g., if aggregations are infrequent due to resource constraints [5]).

If the local datasets are i.i.d. (i.e., each is representative of the global distribution), then discarding more data clearly increases the global loss, since less data is used to train the ML model. Offloading may also skew the local model if it is updated over a small $G_{i}(t)$. We can, however, upper bound the loss function $L(w_{i}(t))$ regardless of the data movement:

In Section IV, we will consider how to use Theorem 1’s result to find tractable forms of the loss expression that allow us to solve the optimization (5–9) efficiently. Moreover, without perfect information on the device costs, capacities, and error statistics over the time period $T$, we cannot solve (5–9) exactly, so we will propose methods for estimating them.

Overestimating $C_{i}(t)$ leads to the deferral of some data processing until future time periods, which may cause a cascade of processing delays. Misestimations of the link capacities have similar effects. Here, to limit delays due to overestimation, we formalize guidelines for the capacities in (9)’s constraints. As commonly done [40], we assume that processing times on stragglers follow an exponential distribution $\exp(\mu)$ for parameter $\mu$.

For device capacities, we obtain the following result:

For instance, $\sigma=1$ guarantees an average processing time of less than one time slot, as assumed in Section III’s model. Thus, Theorem 2 shows that we can still (probabilistically) bound the data processing time when stragglers are present. As long as $G_{i}(t)\leq C_{i}(t)$, where $G_{i}(t)$ is defined based on any values of $s_{ij}(t)$ and $r_{i}(t)$, Theorem 2 holds for any data offloading and discarding policy, including the one provided by the solution to our optimization (5)-(9).

Network link congestion encountered in transferring data may also delay its processing, which can be handled by choosing the network capacity $C_{ij}(t)$ analogously to Theorem 2.

To solve the optimization (5)–(9), we also need an expression of the error objective $f_{i}(t)L(w_{i}(t)|D_{V})$ in terms of our decision variables. As shown in Theorem 1, we can bound the local loss at time $t$ in terms of a gradient divergence constant $\delta_{i}\geq\left\|\nabla L_{i}(w)-\nabla L(w)\right\|$. The following in turn provides an upper bound for $\delta_{i}$ in terms of $G_{i}(t)$:

The bound in Lemma 1 is likely to be a loose bound for non-i.i.d data, since, with data movement, the resulting distribution of data at device $i$ will be a mixture of the original distributions at each device. The value of $\Delta$ quantifies the degree to which local datasets $\mathcal{D}_{i}$ are non-i.i.d. across devices. When the local datasets are i.i.d., $\mathcal{D}_{i}=\mathcal{D}_{j}$ $\forall i,j$ and $\Delta=0$. Since we take the generated device datasets $D_{i}(t)$ as given in the optimization, $\Delta$ is independent of our decision variables $s_{ij}(t)$ and $r_{i}(t)$, and thus independent of $G_{i}(t)$. As a result, only the term $\gamma_{i}/\sqrt{G_{i}(t)}$ from (11) is dependent on our decision variables. By substituting Lemma 1 into $g_{i}(t)$ in Theorem 1, we see that $L(w_{i}(t))-L(w^{\star})\propto\sqrt{G_{i}(t)^{-1}}$. Since the other time-dependent terms in $g_{i}(t)$ can be absorbed into $f_{i}(t)$, one choice of the error cost $f_{i}(t)L(w_{i}(t)|D_{V})$ in (5) is $f_{i}(t)\sqrt{G_{i}(t)^{-1}}$, capturing diminishing marginal returns in $G_{i}(t)$.

Since $\sqrt{G_{i}(t)^{-1}}$ is a convex function of $G_{i}(t)$, with this choice of error cost, (5–9) becomes a convex optimization problem and can be solved relatively easily in theory. When the number of variables is large, however – e.g., if the number of devices $n>100$ with $T>100$ time periods, as could be the case in the applications discussed in Section I-A – standard interior point solvers will be prohibitively slow [41]. In such cases, we may wish to approximate the error term with a linear function and leverage faster linear optimization techniques, i.e., to take the error cost as $-f_{i}(t)G_{i}(t)$ so the error decreases when $G_{i}(t)$ increases. This choice eliminates the diminishing marginal returns in $G_{i}(t)$. For learning tasks that are of a critical nature (e.g., object detection in smart vehicles [2]), this may actually be more desirable in order to prioritize the trained model accuracy. Ultimately, in practice, the desired relationship between the resource and error costs in the optimization depends on the particular application.

We can further express $-f_{i}(t)G_{i}(t)=-f_{i}(t)(1-r_{i}(t))D_{i}(t)+f_{i}(t)D_{i}(t)\sum_{j\neq i}s_{ij}(t)-f_{i}(t)\sum_{j\neq i}s_{ji}(t-1)D_{j}(t-1)$. Since these last two terms are linear functions of $s_{ij}(t)$, they can alternatively be viewed as part of the transmission cost $\sum_{(i,j)\in E(t)}D_{i}(t)s_{ij}(t)c_{ij}(t)$ in (5). Specifically, if we redefine $c_{ij}(t)\leftarrow c_{ij}(t)+f_{i}(t)-f_{j}(t+1)$ in the objective, we can treat the error cost as $-f_{i}(t)D_{i}(t)[1-r_{i}(t)]$, which is equivalent to minimizing $f_{i}(t)D_{i}(t)r_{i}(t)$, i.e., a cost proportional to the amount of data discarded at node $i$. The analytical results we present in Sec. IV-B will use this form.

We can quantify the difference imposed by the linear approximation by considering the Taylor expansion of $h(x)=x^{-1/2}$, where $x$ represents the quantity of processed data, i.e., $G_{i}(t)$, at a device. Our linear approximation is equivalent to the first order Taylor approximation of $h(x)$, i.e. $h(x)\approx h(x_{0})-\frac{1}{2}x_{0}^{-3/2}(x-x_{0})$, with $x_{0}=0$. Taylor’s theorem shows that the error in this first order expansion is proportional to $c^{-5/2}x^{2}$ for some $0<c<x$, which is increasing in $x$ [42, 43]. Thus, the approximation will have a larger difference when devices are processing more data. Later in Sec. V-B, our experiments will show that there can be a practical advantage to modeling the discard cost as $f_{i}(t)D_{i}(t)r_{i}(t)$ without modifying the $c_{ij}(t)$.

First, we consider the case of a general network topologies. In this case, we can derive a closed-form solution for the data processing variables under the linear error term $f_{i}(t)r_{i}(t)D_{i}(t)$:

This theorem implies that with a linear discard cost, in the absence of resource constraints, all data will either be processed, offloaded to the lowest cost neighbor, or discarded. Additionally, Theorem 3 quantifies how the link costs $c_{ij}(t)$ affect the amount of data offloaded, allowing us to write the fraction of data offloaded by device $i$ as $1-s_{ii}(t)-r_{i}(t)$, which is $1$ if $c_{ik}(t)+c_{k}(t+1)\leq\min\{f_{i}(t),c_{i}(t)\}$ and $0$ otherwise. We later use this point in the proofs for Theorems 5 and 6.

While the connected vehicles and AR settings require the generic topology treatment from Theorem 3, we can derive additional results for static hierarchical and social topologies:

In hierarchical scenarios, more powerful edge servers will likely always have sufficient capacity to handle all offloaded data, and they will likely have lower computing costs when compared to other devices. If the cost of discarding is linear, then from Theorem 3, sensors offload their data to the edge servers, unless the cost of offloading exceeds the difference in computing costs. We show in Section V that the network cost can exceed the savings from offloading to more powerful devices; in such cases, devices process or discard their data instead.

When the cost of discarding is nonlinear, the optimal solution is less intuitive: it may be optimal to discard data if the additional processing cost outweighs the expected error reduction. If we consider multiple heterogeneous devices with static processing costs and data generation rates, each connected to a more powerful edge server over the same wireless network (and thus assumed with the same connectivity costs), we can find a closed form solution to our original optimization:

Intuitively, as the costs $c_{i}$ or $c_{n+1}$ increase, so should the amount of data discarded, as in Theorem 4. Theorem 4 also shows that data is neither fully discarded nor offloaded, in contrast with Theorem 3, implying that convex error bounds lead to a more balanced distribution of data across nodes in the network. We do not include resource constraints in Theorem 4 in order to focus on the effects of the compute and transmission costs $c_{i}$, $c_{n+1}$, and $c_{t}$ on the optimal solution. In practice, an edge server would have nearly unlimited capacity $C_{n+1}$ to process the datapoints generated by the devices, as it is much more powerful, and often has access to a more reliable power source, than typical mobile devices. Intuitively, including the capacities $C_{i}$ and $C_{t}$ on the devices and links would further increase the amount of data discarded.

When networks are larger and have more complex topologies, we extrapolate from Theorem 3’s characterization of device behaviors to understand data movement in the network as a whole. Specifically, we consider a social topology in which edges between devices are defined by willingness to share data (Figure 1(b)). In such cases, we assume $c_{ij}(t)=0$ as nodes either trust each other to share data or do not; nodes that do not trust each other simply will not have an edge between them. We can find the fraction of devices that offload data, which allows us to determine the cost savings from offloading, when the error cost is linear:

Thus, the reduction in cost from enabling device offloading in such scenarios is approximately linear in $C$: as the range of computing costs increases, there is greater benefit from offloading, since devices are more likely to find a neighbor with lower cost. The processing cost model may for instance represent device battery levels drawn uniformly at random from $0$ (full charge) to $C$ (low charge). The expected reduction from offloading, however, may be less than the average computing cost $C/2$, as offloading data to another device does not entirely eliminate the computing cost. Note we can use a similar technique to analyze the scenario where $c_{ij}$ is either nonzero or follows a given distribution; however, the expression for cost savings is then more complex due to the need to compare $c_{j}(t)$ (drawn from $U(0,C)$) with $c_{jk}(t)+c_{k}(t)$ (drawn from a convolution of $U(0,C)$ with the distribution of $c_{ij}(t)$). Similarly, we do not generalize Theorem 5’s results to a non-uniform $c_{i}(t)$ distribution due to the complexity in writing a closed-form solution as in (15). However, we expect similar observations to hold, i.e., offloading becomes more beneficial as the cost range increases.

Next, we consider the case in which resource constraints are present, e.g., for less powerful edge devices. We find the expected devices that will have tight resource constraints:

Theorem 6 makes two assumptions: $D_{i}(t)=D$ and resource capacities being i.i.d. across devices. The former assumption models a constant rate of data generation (e.g., from regular monitoring of user activity). If $D$ is stochastic (e.g., from event-driven sensor readings with random event sequences), we may generalize (16) by taking an additional expectation over $D$. The latter assumption is reasonable if we have no information about specific devices in the network, but know a range of possible hardware specifications that determine the compute capacities. This is likely to occur when there are a large number of devices; as we explain below, such a scenario is precisely where Theorem 6 is most useful.

Theorem 6 allows us to quantify the complexity of solving the data movement optimization problem when resource constraints are in effect. We observe that it depends on not just the resource constraints, but also on the distribution of computing costs (through $P_{o}(k)$), since these costs influence the probability devices will want to offload in the first place. Additionally, Theorem 6 provides a guide for the most efficient way to solve optimization (5)–(9). If the expected number of resource violations is low, then following the procedure in Theorem 3 will produce a near-optimal solution that violates only a few resource constraints. We can then ensure these constraints are satisfied with minimal adjustments to the solution, e.g., optimizing over only the $s_{ij}(t)$ and $r_{i}(t)$ variables for the affected nodes and their neighbors while fixing all other optimization variables, or even increasing the $r_{i}(t)$ until the capacity constraints are satisfied. While this solution will not be optimal, (12) can be implemented distributedly, if each device $j$ sends each of its neighbors $i$ (i) its processing cost $c_{j}(t)$ and (ii) estimates of $c_{ij}(t)$, e.g., channel conditions at the receiver. Thus, it is significantly more efficient than solving (5)–(9) via a generic linear solver. On the other hand, if (16) is large, then we should solve (5)–(9) using a generic optimizer.