Repair Strategies for Storage on Mobile Clouds

We consider a distributed storage system (DSS) consisting of mobile storage nodes that enter and exit a geographically-limited area $\mathcal{A}$. When a node departs from $\mathcal{A}$, its data is lost. The nodes that store file fragments within area $\mathcal{A}$ are said to be live nodes. New nodes that are used to store repaired fragments are said to be newcomer nodes or newcomers. We assume that there are always sufficient newcomers to perform repairs. Moreover, as we are interested in the system performance due to network dynamics, we do not consider data loss due to hardware failures. Such failures occurs orders of magnitude less frequently than node departures. Following the network dynamics model of prior works [8, 23], we model the time $X_{i}$ spent by each node within $\mathcal{A}$ as an exponentially distributed random variable with parameter $\lambda$ (i.e., $X_{i}\sim\textrm{Exp}(\lambda),~\forall i)$. Random variables $\{X_{i}\}$ are assumed independent and identically distributed.

The repair time is modeled by an exponentially distributed random variable with parameter $\mu$. For ease of analysis, we initially assume that $\mu$ is independent of the number of fragments that need to be repaired. We later revise our analysis and consider a more realistic model in which repairs proceed in parallel at different nodes with the same rate $\mu$. This corresponds to the distributed nature of mobile DSS. Finally, we define $\rho=\frac{\lambda}{\mu}$ as the ratio of the departure-to-repair rate.

A file $\mathcal{F}$ of size $\mathcal{M}$ bits is stored in $n$ storage nodes using a regenerating code with parameters $(n,k,d,\alpha,\beta)$ (see Fig. 2(b)). We focus on the two most popular types of regenerating codes, namely Minimum Storage Regenerating (MSR) codes and Minimum Bandwidth Regenerating (MBR) codes. These two classes of codes operate at the end points of the tradeoff between per node storage and repair bandwidth, as introduced in [9]. MSR codes achieve minimum storage by setting $\alpha=\mathcal{M}/k$ and minimize the repair bandwidth under this constraint. Their operating point is given by:

Note that, for MSR codes, $\alpha_{\textrm{MSR}}\leq\gamma_{\textrm{MSR}}$ and hence, the per-node storage is smaller than the repair bandwidth. MBR codes, on the other hand, minimize the repair bandwidth (achieved when $\gamma=\alpha$), and operate at:

Instances of these codes can be found in [12, 10, 11].

In our model, the system continuously monitors the redundancy level and initiates a repair when $\tau$ live nodes remain within $\mathcal{A}$. The determination of $\tau$, the type of repair (regeneration, reconstruction, or both) and the communication model for fragment retrieval (centralized or distributed) form a file maintenance strategy. We note that the practical implementation details of the redundancy monitoring mechanism and of the communication protocols for retrieving various fragments are beyond the scope of the present work. We focus on the theoretical aspects of the maintenance process. Since repairs are initiated only when the number of remaining nodes reaches threshold $\tau$, a repair strategy can be viewed as an i.i.d. system recovery process occurring every $\Delta$ seconds, where $\Delta$ is a random variable denoting the time elapsed between two instances of a fully repaired system. For this recovery process, we define the following costs.

We determine the optimal file maintenance strategy for different node departure rates, code parameters, and communication models for fragment retrieval.

In distributed repair, newcomers recover lost fragments by independently downloading relevant symbols from live nodes. The repair process is initiated when $\tau$ live nodes remain within $\mathcal{A}$, where $k\leq\tau<n-1$ (when $\tau<k$, the data is irrecoverably lost). If $\tau\geq d$, fragment recovery can be performed through regeneration. Each of the $n-\tau$ newcomers downloads $\beta$ symbols from $d$ live nodes and independently regenerates a lost fragment. Fig. 3(a) demonstrates the distributed repair process for a file $\mathcal{F}$ stored with a $(n=4,k=2,d=3,\alpha=2,\beta=1)$ regenerating code. One fragment of $\mathcal{F}$ is lost because node $s_{9}$ departed from $\mathcal{A}$. The lost fragment is regenerated at $s_{1}$ by independently downloading $\beta=1$ symbol from three nodes. The total repair bandwidth is equal to 3 symbols.

If $\tau<d,$ regeneration cannot be directly applied. To reduce the repair cost, we consider a hybrid scheme consisting of regeneration and reconstruction. First, $d-\tau$ nodes are repaired by downloading $\alpha$ symbols from $k$ live nodes and reconstructing $\mathcal{F}$. When $d$ fragments become available, regeneration is applied to repair the remaining $n-d$ newcomers. Accordingly, the repair cost is expressed by:

The subscript $D$ in $c_{D}(\tau)$ is used to denote the cost of distributed repair and $\gamma$ denotes the regeneration cost of a single fragment which depends on the underlying regeneration code (see eqs. (1) and (2) for MSR and MBR codes, respectively). From (3), it is evident that $c_{D}(\tau)$ monotonically decreases with $\tau.$ Moreover, the rate of cost change (with respect to $\tau$) is higher when $\tau<d.$ To determine the optimal threshold $\tau^{*}$, we are interested in minimizing $r_{D}(\tau)$, which captures the repair cost for maintaining $n$ fragments per unit of time.

To calculate $r_{D}(\tau)$, we use the continuous-time Markov chain (CTMC) model shown in Fig. 4. This model captures the periodic repair process when node departures occur independently, the time spent by each node in $\mathcal{A}$ is exponentially distributed with parameter $\lambda$, and the system recovery process is exponentially distributed with parameter $\mu$.

The CTMC consists of $n-\tau+1$ states representing the number of fragments that remain within $\mathcal{A}$ after each node departure, until a repair at state $\tau$ is initiated. Note that we have omitted states after $\tau$ in the CTMC model, because we are interested in optimizing the periodic cost of repairing the DSS at threshold $\tau$. Moreover, the transition probability to state $\tau-1$ is negligible for most realistic scenarios in which $\mu\gg\tau\lambda.$ For cases when $\mu\not\gg\tau\lambda$, we compute the mean time it takes to depart from the optimal repair strategy of repairing at state $\tau$ and interpret this event as a form of system error which leads to data loss (see Section V-C).

For the CTMC in Fig 4, the departure rate from a state $i$ equals the node departure rate $\lambda$, times the number of nodes which store fragments at state $i$. When the repair process is initiated, the system transitions from state $\tau$ to state $n$ because all fragment repairs nodes proceed in parallel. For the CTMC, we define the expected average cost $r_{D}(\tau)$ per unit of time as

where $E[\Delta]$ is the average time between two transitions through the $n^{th}$ state in the periodic repair process¹¹1The alternative definition of $r_{D}(\tau)=E\left[\frac{c_{D}(\tau)}{\Delta}\right]$ is not useful because the expectation is infinite. This is due to the infinitesimally small values that can be obtained by $\Delta$, whereas $c_{D}(\tau)$ remains lower bounded.. For $\Delta$,

where $T_{i}$ denotes the time that the system stays at state $i$ (inter-departure time) and $T_{\tau}$ is the expected time for completing repairs so that $n-\tau$ fragments are recovered (return to state $n$). The random variables $T_{i}$ are independent and exponentially distributed with parameter $i\lambda$, whereas $T_{\tau}$ is exponentially distributed with parameter $\mu$. In particular, $\textrm{E}[T_{i}]=\frac{1}{i\lambda}$ and $\textrm{E}[T_{\tau}]=\frac{1}{\mu}$. Therefore, $\textrm{E}[\Delta]$ is the sum expectation of independent exponential random variables.

where $H_{n,\tau}=\sum_{i=\tau+1}^{n}\frac{1}{i}$. Combining (4) and (6), we obtain the average repair cost per unit of time as follows.

We use (7) to determine the optimal threshold $\tau^{*}$ which minimizes $r_{D}(\tau).$ This is given by Propositions 1 and 2.

Proposition 1 determines the $\rho$ regime for which repairs at $\tau=d$, an instance of lazy repair, is more efficient than initiating repairs at $\tau=n-1$, referred to as eager repair. In the following Lemma, we show that there is always a positive $\rho$ for which lazy repair is more efficient that eager repair.

We now examine if there is a $\rho$ regime for which the hybrid scheme, i.e., reconstruction plus regeneration results in a lower expected cost per unit of time compared to regeneration only. This rate regime is given by the following proposition.

Similar to Lemma 1, we investigate if the highest departure-to-repair rate for which reconstruction at $k$ is more efficient than regeneration is always positive independent of the code parameters. Unlike the case of Lemma 1, we show that for a certain relationship between $n,k,\gamma,$ and $\alpha,$ regeneration is strictly more efficient than regeneration plus reconstruction, independent of $\rho$. For any other code parameters, the most efficient strategy depends on $\rho$.

We further explore the condition in Lemma 2 for MSR and MBR codes. For MSR codes, we obtain that $dn<k^{2}(d-k+1)$ by substituting the operation points of MSR from (1). Similarly, for MBR codes, we obtain that $n<k^{2}$ by substituting the operation points of MBR from (2). Note that Lemma 2 does not enumerate all possible codes for which regeneration is strictly more efficient than regeneration plus reconstruction for any $\lambda.$ This is because we have used bounds on the harmonic function to derive the analytic formulas. Numerical bounds could provide a more accurate range of code parameters for which Lemma 2 is true.

In the centralized strategy, repairs are performed by a leader node in two stages. In the first stage, the leader newcomer node downloads $\alpha$ symbols from $k$ live nodes and reconstructs $\mathcal{F}$. In the second stage, the leader node transmits $\alpha$ bits to each of the remaining $(n-\tau-1)$ newcomers to restore the remaining $(n-\tau-1)$ fragments. Fig. 3(b) shows an example of centralized repair for a $(n=4,k=2,d=3,\alpha=2,\beta=1)$ regenerating code. Nodes $s_{2}$ and $s_{9}$ have departed from area $\mathcal{A}$, leading to the loss of their respective fragments. Node $s_{5},$ who acts as a leader, downloads $\alpha=2$ symbols from $k=2$ other nodes to reconstruct $\mathcal{F}$. It then distributes $\alpha=2$ symbols to $s_{1}$ and $s_{7}$ to restore the system reliability. The repair cost of centralized repair is given by:

In (10), the subscript $C$ in $c_{C}(\tau)$ is used to denote the cost of centralized repair. The node departure process does not vary with the repair strategy. Therefore, the same CTMC model shown in Fig. 4 applies for the centralized repair. According to (4), the average repair cost $r_{C}(\tau)$ is given by:

The optimal threshold $\tau^{*}$ which minimizes $r(\tau)$ is obtained in Proposition 3.

Using Proposition 3, we can determine the optimal repair strategy for any $\rho$, when centralized repair is employed. We note that according to Lemma 1, the value $\frac{kH_{n-1,k}}{(n-k-1)}-\frac{1}{n}$ is strictly positive for any code parameters. Therefore, there is always a departure-to-repair ratio for which lazy repair is more efficient than eager repair, independent of the code used for regeneration and reconstruction.

According to the results of Propositions 1, 2, and 3, we classify the departure-to-repair ratios into a low departure-to-repair rate regime $(\rho_{\textrm{low}}$) and a high departure-to-repair rate regime $(\rho_{\textrm{high}})$. The two regimes are defined by finding the lowest and highest rates, based on the bounds stated in the three propositions.

Noting that $\frac{H_{n-1,d}}{n-d-1}-\frac{1}{n}<\frac{kH_{n-1,k}}{(n-k-1)}-\frac{1}{n}$ for $k<d,$ the two regime expressions can be simplified to

For any $\rho\leq\rho_{\textrm{low}}$, the repair cost per unit of time is minimized when lazy repair is applied since that choice of $\rho$ would be lower than the bounds found in (8), (9) and (12) and the corresponding repair thresholds are the lowest possible. On the other hand, for any $\rho\geq\rho_{\textrm{high}}$, eager repair (i.e., repair at $\tau^{*}=n-1$) yields the lowest $r(\tau)$. These findings hold for both distributed and centralized repair. If the departure-to-repair rates do not lie in either of the $\rho$ regimes, then the optimal repair policy (eager vs. lazy) depends on the relationship of the code parameters and the repair strategy (centralized or distributed).

We now fix the departure rate $\lambda$ and repair rate $\mu$ to compare the repair cost of centralized vs. distributed repair per unit of time, as a function of the code parameters. Specifically, we determine relationships between $n,k,d$ and the code type (MSR vs. MBR) for which an optimal strategy can be derived. Our results are stated in the following two propositions.

We now prove that if $\tau^{*}$ lies between $k$ and $d$, using MSR codes with centralized repair is optimal.

We now examine the Mean Time to Data Loss (MTTDL) for the periodic threshold repair process. For our purposes, we consider that data is lost if the DSS transitions from state $\tau$ to state $\tau-1$ instead of state $n$. That is, if a node leaves the system before repairs are completed when initiated at state $\tau$, the repair process is abandoned and the system eventually reaches state $k-1$, at which data is lost. In this case, the file $F$ is reinstated at the mobile nodes by a central entity. Note that when $\tau>k$ repairs could be re-initiated at state $\tau-1$, because at least $k$ fragments remain available. We opted not to consider this option for the MTTDL calculation to capture the periodic nature of the threshold repair strategy. The MTTDL reflects the period of time at which the DSS oscillates between states $n$ and $\tau$. The time to reach state $k-1$ assuming no repairs are attempted after state $\tau$ is given by:

The MTTDL is a decreasing function of $\tau$. This is intuitive considering that the number of nodes that need to depart for reaching state $k-1$ increases with $\tau$. Moreover, the average time it takes to reach state $\tau$ from state $n$ increases with $\tau$. This indicates that the periodic repair of the DSS will on average last longer if a lazy repair strategy is adopted.

In this section, we validate our theoretical results be providing numerical examples. Fig.  5(a) shows $r(\tau)$ when $d>\frac{n+k-1}{3}$ and $\rho=10^{-4}$ . According to Proposition 4, for this combination of code parameters, a distributed repair strategy with MBR codes (D-MBR) achieves the minimum $r(\tau)$ for all $d\leq\tau^{*}\leq n-1$. The minimum occurs at $\tau^{*}=d$. Moreover, according to Proposition 5, centralized MSR codes (C-MSR) minimize $r(\tau)$ for $k\leq\tau<d.$ This is verified in all plots of Fig. 5, for which the cost is minimized by the C-MSR strategy when $\tau^{*}=k$, if $\tau<d$. In Fig. 5(b), we show $r(\tau)$ when $d<\frac{n+k-1}{3}$ and $\rho=10^{-4}$. For this case, there is no one scheme with optimal cost for any value of $d\leq\tau\leq n-1$. For $\tau>16,$ D-MBR is optimal, whereas for $10\leq\tau\leq 15$, C-MSR becomes optimal. C-MSR achieves the lowest overall cost at $\tau=k.$

We also studied the impact of $\rho$, when the code parameters are fixed to $(n=30,k=20,d=25)$. Fig. 5(c) shows the average cost per unit of time ($r(\tau)$) when $\rho=10^{-4}.$ For this $\rho$ regime, a lazy repair strategy with $\tau^{\ast}=d$ minimizes $r(\tau)$, with D-MBR codes achieving the lowest cost. On the other hand, eager repair becomes optimal for any $\rho>\rho_{\textrm{high}}$. This is observed in Fig. 5(d), in which the value of $\rho$ has been increased to one. D-MBR codes still remain the optimal option, however, the optimal repair threshold is now shifted to $\tau^{*}=n-1$. Note that at the high $\rho$ regime, all codes exhibit the same behavior. The average cost per unit of time becomes a decreasing function of $\tau$.

Finally, on the right $y$-axis of the plots in Fig. 5, we show the MTTDL values for the given set of parameters. As expected, the MTTDL is an decreasing function of $\tau$ due to the corresponding increase in departure rate from state $\tau$ with the value of $\tau$. The MTTDL becomes impractical in the high $\rho$ regime, because nodes frequently leave area $\mathcal{A}$ before repairs can be completed.

When multiple nodes are to be repaired simultaneously, in addition to contacting live nodes and downloading symbols from those, newcomers can also communicate between each other to complete the recovery process. Formally, assume that $t$ nodes are to be repaired. Each of the $t$ newcomer nodes can contact $d$ live nodes and download $\beta$ symbols as well as download $\beta^{\prime}$ from each other. In this scenario, the repair bandwidth can be calculated as $\gamma=d\beta+(t-1)\beta^{\prime}$. Such codes are studied in [19] (referred to also as coordinated regenerating codes) and the tradeoff between per node storage $\alpha$ and repair bandwidth $\gamma$ is analyzed. Two ends of tradeoff curve is named Minimum Storage Cooperative Regenerating (MSCR) and Minimum Bandwidth Cooperative Regenerating (MBCR). Accordingly, operating points are given as follows:

For MSCR codes, $\gamma_{\textrm{MSCR}}=d\beta_{\textrm{MSCR}}+(t-1)\beta_{\textrm{MSCR}}^{\prime}=\frac{\mathcal{M}(d+t-1)}{k\left(d-k+t\right)}\geq\frac{\mathcal{M}}{k}$, where the per-node storage is smaller than the repair bandwidth. On the other hand, MBCR codes provide the minimum repair bandwidth which operates at:

Note that for MBCR codes, we have $\alpha_{\textrm{MBCR}}=\gamma_{\textrm{MBCR}}=d\beta_{\textrm{MBCR}}+(t-1)\beta_{\textrm{MBCR}}^{\prime}$.

We define $r_{t}(\tau)$ as the average repair cost for a system with repair threshold $\tau$ under cooperative repair using groups of nodes of size $t$ (similarly $c_{t}(\tau)$ for the cost). Since $n-\tau$ nodes need to be repaired, any cooperative regenerating codes with $t$ such that $t|n-\tau$ can be used in practice. In the following proposition, we compare the performance of cooperative regenerating codes at all possible $t$ values to find the value of $t$ that minimizes the average repair cost.

In Fig. 6, we show the average repair cost for cooperative regenerating codes at all possible $t$ values for a given $n-\tau$. We observe that for the same $n-\tau$, if $t_{1}<t_{2}$, then $r_{t_{1}}(\tau)>r_{t_{2}}(\tau)$ and the minimum is achieved when $t=n-\tau$.

In the remaining of this section, we suppress the subindex $t$ from $r_{t}(\tau)$ since we established that $t=n-\tau$ minimizes the cost, i.e., $r(\tau)=r_{n-\tau}(\tau)$ and $c(\tau)=c_{n-\tau}(\tau)$. However, we may still need to distinguish $\gamma$ and $\alpha$ values under different cooperative regenerating codes. We denote the per-node storage for cooperative repairs with $n-\tau$, which results in the minimum average repair cost for the system with threshold $\tau$, by $\alpha_{\tau}$. Similarly we use $\gamma_{\tau}$ to denote the repair bandwidth at $\tau$.

Note that, cooperative regenerating codes, it is required that $d+n-\tau\leq n$. In other words, there should be at least $d$ live nodes when repairs are initiated. Otherwise, it is not possible to regenerate $n-\tau$ nodes from d live nodes. Henceforth, the repair cost $c(\tau)$ and the repair cost per time for cooperative codes are as follows:

The minimum $r(\tau)$ with respect to $\tau$ does not have a closed-form analytical expression. In Section VI-C, we present numerical results to study the change of $r(\tau)$ with $\tau$ and determine the optimal repair threshold $\tau$ that minimizes $r(\tau)$ for distributed cooperative repair.

Centralized repair of multiple node repairs is introduced in [22]. Using this model, a dedicated node among the $t$ newcomers downloads $\beta$ from any $d$ live nodes such that it can repair multiple node failures of size $t$. Such codes can be used in the centralized repair process that is proposed in Section IV when we set $t=n-\tau$. Rawat et al. in [22] characterize the tradeoff between per-node storage and repair bandwidth for this centralized repair model. Accordingly, the following operation points are derived for minimum storage multi-node regeneration (MSMR) and minimum bandwidth multi-node regeneration (MBMR):

Let $k\textrm{ mod}(n-\tau)=b$. If $H_{b}\geq\binom{\beta}{n-\tau}\left[b(\frac{2d+n-\tau-1}{2})-\binom{b}{2}\right]$ (where $H_{b}$ denotes entropy of information stored on $b$ nodes), then

Under centralized repair model discussed here, a dedicated node first downloads $\gamma=d\beta$ and then distributes $\alpha$ to remaining $n-\tau-1$. The difference between these codes and the earlier proposed method for centralized repair in Section IV-B is that the dedicated node may not need to download the whole file. Therefore, we have the following repair cost

from which one can obtain

In order to find the optimal threshold that minimizes the average repair cost, we need to find the minimum value of $r_{C}(\tau)$. We can replace $H_{n,\tau}$ with its approximation, $\ln(\frac{n}{\tau})$, and take the derivative with respect to $\tau$. Note that both $\gamma$ and $\alpha$ depend on $\tau$ and there is no tractable analytical solution for $\tau$ that minimizes $r_{C}(\tau)$. However, we can still analyze (27) numerically with respect to $\tau$ and observe the optimal threshold from numerical results.

We study the performance of cooperative codes with $n=30,d=25$ and $k=19$ under different $\rho$ regimes. In Fig. 7, we compare the codes studied in Sections IV and VI for different $\rho$ regimes. We first compare regenerating codes vs. cooperative regenerating codes for the distributed repair scenario. Note that we focus only on $d\leq\tau\leq n-1$ because at least $d$ live nodes must exist for cooperation (see Section VI-A). We observe that cooperative regenerating codes always have lower cost than regenerating codes for all values of $\rho$. Additionally, the gap between the cost of D-MBR and D-MBCR is much smaller than the gap between the cost of D-MSR and D-MSCR. Furthermore, we can observe two opposing regimes: In Fig. 7(a), the optimal cost is at $\tau=d$, whereas in Fig. 7(b), the cost is minimized at $\tau=n-1$. We also compare the centralized regenerating codes to the centralized repair of multiple node departures. As expected, since in the latter scheme one does not need the whole file for file reconstruction at the dedicated node, centralized repair of multiple node departures results in lower repair cost. At $\tau=n-1$, in Fig. 7(d) average repair cost is minimized for all schemes, on the other hand we observe different optimum $\tau$ values for different coding schemes in Fig. 7(c). Finally, we compare all schemes in Fig. 7(e)-(f) for different values of $\rho$ for completeness. It is observed that the centralized repair of multiple node departures model discussed in this section approaches to the distributed repair model in Section IV as $\tau$ approaches $n$ and diverges from the centralized repair model in Section IV. The reason for this behavior is that the dedicated node does not need to download the whole file now (as opposed to the centralized repair model in Section IV, which incurs high average repair cost for large $\tau$).