Fault-tolerant Network Design for Bounded Delay Data Transfer from PMUs to Control Center

Accurate estimation of the operational state of the electric power grid is vital for its stable operation. For state estimation of the grid, power transmission control data, generated by the Phasor Measurement Units (PMUs) located in some of the Substations (SS), has to arrive at the Controls Center(s) (CC) within an acceptable delay threshold $\delta$. Accordingly, communication network topology design for electric power grid has received significant attention in recent times [1, 2, 3]. The authors in [1] study the communication network design problem for the power grid to analyze possible topologies, and provide guidance to power utilities for choosing the most appropriate communication network for their PMU based real-time applications.

Many researchers have studied the optimal PMU placement problem with specific objectives, such as, full network observability [4]. However, optimal PMU placement without taking into account the cost of Communication Infrastructure (CI), may lead to the design of a very expensive Wide Area Monitoring (WAM) system, where the cost of CI may dominate the cost of the PMUs. Mohammadi et.al. [3] studied the joint optimization of PMU placement and associated CI cost. This line of research is continued in [2], where the authors study the optimal PMU-communication link placement (OPLP) problem that simultaneously considers the placement of PMUs and communication links for full observability.

Motivated by the studies undertaken by the power engineers [1, 2, 3], our earlier papers formalized the design problem as the Rooted Delay Constrained Minimum Spanning Tree (RDCMST) problem [5, 6]. However, this design does not provide any fault-tolerance capability: failure of just one communication link will prevent the PMU data from some of the substations from reaching the CC. In this paper, we overcome this limitation by studying the optimal cost network design problem with fault tolerance capability. That is, PMU data from the substations will have at least one path to reach the CC in spite of the failure of at most $R$ links, within the acceptable delay threshold. Although our formulation allows for the design of a $R$-fault-tolerant network for any $R$, our primary focus in this paper is the case where $R=1$. When $R=1$, the problem may be viewed as a variation of the well-studied cycle cover and vehicle routing problems [7, 8]. We define the problem as a Rooted Delay Constrained Minimum Cost Cycle Cover (RDCMCCC) problem.

The contributions of this paper are as follows:

• •

  We introduce the delay bounded optimal cost fault-tolerant PMU to CC data transfer network design problem.

• •

  We prove that the RDCMCCC problem is NP-complete and provide an approximation algorithm for it.

• •

  We provide an Integer Linear Program for finding the optimal solution for the RDCMCCC problem.

• •

  We show that a variation of the RDCMCCC problem can be solved in polynomial time.

• •

  We provide a Sweeping Technique based Heuristic solution for the RDCMCCC problem.

• •

  We conduct extensive experimental evaluation of our heuristic with real Arizona substation data.

We formalize the RDCMCCC problem by viewing the locations of the substations as a set $P$ of $n$ points in the Euclidean plane. The input to the problems includes a distinguished point, $p_{0}\in P$, corresponding to the location of the CC, and an acceptable delay threshold parameter $\delta$. We construct a weighted, complete graph $G=(V,E)$, where each node $v_{i}\in V$ corresponds to a point $p_{i}\in P$. Since the node $v_{i}$ and the point $p_{i}$ have a one-to-one correspondence, we use the terms node and point interchangeably. The weight $w(v_{i},v_{j})$ of an edge $(v_{i},v_{j})\in E$, is the Euclidean distance between the corresponding points $p_{i},p_{j}\in P$. We assume that $w(v_{i},v_{j})$ represents both the cost and the delay of the link $(v_{i},v_{j})$. We note that this is a simplified assumption, as in a realistic communication system, the link length is only one of the factors that will determine the cost and the delay associated with that link. The cost of a link may be determined by factors such as wired/wireless, optical fiber/coaxial cable, etc, whereas the delay on a link may be determined by factors such as transmission rate, link bandwidth, queuing delay, etc. The consideration of only the link length (which determines the propagation delay) as the only factor makes the model somewhat simplistic. However, this simplified model is widely used in literature for the evaluation of communications networks [9].

Note that in RDCMC-(R = 1)-FTNDP the directed edge $e_{i,j},(v_{i}\rightarrow v_{j})$ may be a part of the several paths, say, from $v_{a}$ to $v_{0}$, $v_{b}$ to $v_{0}$, $v_{c}$ to $v_{0}$ etc. In this environment, as the cost $C_{i,j}$ is associated with each edge $e_{i,j}$. if the edge $e_{i,j}$ is a part of three paths, $v_{a}$ to $v_{0}$, $v_{b}$ to $v_{0}$, $v_{c}$ to $v_{0}$, unless one is careful, the cost $C_{i,j}$ may be counted three times. However, in our model we view that this cost should not be counted three times, as an optical fiber line connecting the nodes $v_{i}$ and $v_{j}$ will be able to carry network traffic for multiple paths. Moreover, in our model if both the directed edges $e_{i,j},(v_{i}\rightarrow v_{j})$ and $e_{j,i},(v_{j}\rightarrow v_{i})$ are used for path construction, the two costs $C_{i,j}$ and $C_{j,i}$, associated with edges $e_{i,j}$ and $e_{j,i}$ respectively, should not be counted as two separate costs. In such a situation the cost of the link connecting the nodes $v_{i}$ and $v_{j}$ should be counted only one time, for the same reason given earlier.

In the mathematical domain, it implies that once the solution of the RDCMC-R-FTNDP is found, where all the edges are directed, the orientation of the edges may be ignored and the resulting graph be treated as an undirected graph. Since in this undirected graph version of the solution, each node $v_{i},1\leq i\leq n$, will have two undirected paths to the node $v_{0}$ of length at most $\delta$. In other words, each node $v_{i},1\leq i\leq n$ will be on at least one cycle with the node $v_{0}$ where cycle length is at most $2\delta$.

From this perspective the RDCMC-(R = 1)-FTNDP is similar to the well-studied Cycle Cover and Vehicle Routing problems [7, 8]. However, important differences remain, primarily in the way the cost of the design is measured. To the best of our knowledge this version has not been studied either in Cycle Cover or Vehicle Routing domain. Because of the similarity of the undirected version solution of the RDCMC-(F = 1)-FTNDP, we refer to it as the Rooted Delay Constrained Minimum Cost Cycle Cover (RDCMCCC) Problem.

We illustrate this by giving a transformation from the Hamiltonian Cycle (HC) problem [10]. We first establish the following lemma. A collection of cycles $\mathcal{C}=\{C_{1},C_{2},\ldots C_{m}\}$ with $|\mathcal{C}|>1$ is said to “cover” a graph $G$ if every node in $G$ is part of at least one cycle in $\mathcal{C}$.

Observe $\left|E\left(\bigcup_{C\in\mathcal{C}}C\right)\right|=|V|$ happens when (i) $m=1$, i.e., when the graph has a Hamiltonian Cycle or the cycles are all disjoint.

Clearly, RDCMCC is in NP, as a Turing Machine can easily verify whether the cost of the subgraph $G^{\prime}$ given as the output of the RDCMCC problem covers all the nodes and the cost of the edges in $G^{\prime}$ doesn’t exceed $\gamma$ [10].

From an instance of the HC problem, we create an instance of the RDCMCC problem and establish that the input graph of the HC problem has a Hamiltonian Cycle if and only if the instance of the RDCMCC has a subgraph that covers all the nodes at a cost that doesn’t exceed $\gamma$.

Consider an instance $G=(V,E)$ of the HC problem. Construct an instance of the RDCMCC problem as follows: (i) create a complete edge-weighted undirected graph $G^{\prime}=(V^{\prime},E^{\prime})$, where $V^{\prime}=V$, (ii) if two nodes $v_{i}$ and $v_{j}$ have an edge between them in $G$, set $w(v_{i},v_{j})=1$ in $G^{\prime}$, (iii) if two nodes $v_{i}$ and $v_{j}$ do not have an edge between them in $G$, set $w(v_{i},v_{j})=2$ in $G^{\prime}$, (iv) set the cost parameter $\gamma=|V|$.

If $G$ has a Hamiltonian Cycle, clearly $G^{\prime}$ has a subgraph that covers all the nodes in $G^{\prime}$ in one cycle (including the node $v_{1}$), and the cost of this cycle is $\gamma=n$. Furthermore, if $G^{\prime}$ has a subgraph that covers all the nodes in $G^{\prime}$ using cycle(s) (each one which includes the node $v_{1}$) and the cost of this cycle is $\gamma=n$, then it must be done with only one cycle since if multiple cycles were used, we know from Lemma 2 that the number of edges belonging to these cycles would have exceeded $|V|=n$ and would have exceeded the cost as the parameter $\gamma$ is set equal to $n$.

We model the power grid as a graph $G=(\{v_{0}\}\cup V,E)$ where the node $v_{0}$ corresponds to the CC, the nodes $V=\{v_{1},\ldots,v_{n}\}$ correspond to the SSs, and the edges in $E$ correspond to communication links (e.g., fiber optic lines). Let $|E|=m$. We view the physical location of the CC, as well as all the substations, as points in a two-dimensional plane. In the graph $G$, each node corresponds to a point in the plane, and the weight $w_{i,j}$ on the edge $e_{i,j}$, is taken to be the Euclidean distance between the corresponding points on the plane. The weight $w_{i,j}$ is taken to be a measure of both (i) the cost $C_{i,j}$ of the edge $e_{i,j}$ and also (ii) the delay $D_{i,j}$ on the edge $d_{i,j}$. We assume that the PMUs are placed only in a subset of the substations and denote this subset as $U\subseteq V$, $U=\{u_{1},\ldots,u_{s}\}$. In some grid environments, $U$ may be equal to $V$, in which case $s=n$ and $\forall i,1\leq i\leq n,u_{i}=v_{i}$. As PMU data has to flow from the PMUs at the substations (nodes $u_{1}$ through $u_{s}$) to the CC (node $v_{0}$) within the specified delay threshold $\delta$, there will be $s$ flows (or paths) from the nodes $u_{i},1\leq i\leq s$ to the node $v_{0}$, such that the length of each one of these paths must be upper bounded by $\delta$.

The following points should be noted:

• •

  In order to have a $R$-fault-tolerant communication network, each node $u_{i},1\leq i\leq s$ must have $R+1$ edge disjoint flows (or paths) from $u_{i}$ to $v_{0}$. Thus, the total number of paths will be $s\times(R+1)$.

• •

  We use the notation $F^{k,r}$ to indicate the $r-$th flow (or path) from node $u_{k}$ to node $v_{0}$, $1\leq k\leq s,1\leq r\leq R+1$.

• •

  We use a binary variable $f^{k,r}_{i,j}$ defined as follows:

  $$f^{k,r}_{i,j}=\left\{\begin{array}[]{ll}1,&\mbox{if the flow $F^{k,r}$ uses the directed edge $e_{i,j}$}\\ 0,&\mbox{otherwise}\end{array}\right.$$

In this section we provide a heuristic solution for the RDCMCCC problem. The locations of the substations and the Control Center are considered as $n$ points on a two dimensional plane. The rectangular (i.e., $(x,y)$) coordinates of the Control Center is taken to be (0, 0). The location of the $i$-th substation (i.e., point $p_{i}$) is expressed in polar coordinate as $(r_{i},\theta_{i})$, where $r_{i}$ denotes its distance from the Control Center, and $\theta_{i}$ denotes its polar coordinate angle. The substation locations points, $P=\{p_{1},p_{2},p_{3},\ldots,p_{n}\}$ are numbered in non-decreasing order of their polar angle, i.e., $\forall i,1\leq i\leq{n-1},\theta_{i}\leq\theta_{i+1}$. The heuristic is based on the idea of a vector first connecting the point (0,0) to point $(r_{1},\theta_{1})$ and then rotating in an anti-clockwise fashion sequentially sweeping the rest of the set of points {$(r_{2},\theta_{2}),(r_{3},\theta_{3}),,\ldots,(r_{n},\theta_{n})\}$. In other words, the vector sweeps the points one after another in non-decreasing order of their polar angle. Because of the nature of its working, we refer to it as the SWEEP heuristic. It attempts to construct the first cycle $C_{1}$, starting and ending at (0, 0), and including the largest number of points {$(r_{1},\theta_{1}),(r_{2},\theta_{2}),\ldots,(r_{k},\theta_{a})\}$, as long as the length of the cycle with points $\{(0,0),(r_{1},\theta_{1}),(r_{2},\theta_{2}),\ldots,(r_{k},\theta_{a})\}$ does not exceed the acceptable delay threshold $\delta$. It then repeats the process by trying to create a second cycle $C_{2}$ with points $\{(0,0),(r_{a+1},\theta_{a+1}),(r_{a+2},\theta_{a+2}),\ldots,(r_{b},\theta_{b})\}$ without exceeding the delay threshold $\delta$. This repetition of cycle creation process continues till all the points are included in at least one cycle.

During the creation of cycles $C_{2}$ and beyond, some refinements are done to reduce the total design cost of the cycles. The design cost is the sum of the cost of the edges of that make up the cycles. In the following, we discuss the refinement scheme in detail.
$\cal C$: The set of cycles $\{C_{1},C_{2},\ldots\}$ created by the Sweep heuristic
$P$: The set of points $\{p_{0},p_{1},p_{2},\ldots p_{n}\}$ representing the locations of the Control Center and the substations. The point $p_{0}$ represent the location of the CC and the rest of the points represent the locations of the substations

The results of our experimental evaluation of the SWEEP heuristic with substation location data of Arizona are presented here. The latitude-longitude locations of substations in Arizona were collected from the U.S. Dept. of Homeland Security website. The total number of substations in Arizona is 892, of which, 653 are operational. As PMUs are expensive, not all substations have them. It has been reported in [4] that PMUs installed in 20%-30% of the substations are sufficient for full observability. We conduct our experiments with substation locations in Phoenix and Tucson, the two largest cities in Arizona. The number of substations in Phoenix and Tucson are 132 and 25, respectively. The results of the experiments (where R=1) are presented in Tables I and II. The data sets $DS_{1}$ through $DS_{5}$ correspond to differing percentages of the substations with PMUs. In Table I, $35\%$ next to $DS_{1}$ indicates that in that data set, $35\%$ of 132 substations (46) are assumed to have PMUs. The delay threshold value $\delta$ was varied from 100 to 250. 46 substation locations from 132 were selected randomly three times. These three data sets may be viewed as $DS_{1,1},DS_{1,2},DS_{1.3}$. The design costs with SWEEP, Optimal, and the ratio between the two were computed for these three data sets for a specific $\delta$ value, and the corresponding average and standard deviation are presented in Table I. The values for SWEEP, Optimal, Ratio for $DS_{5}(15\%)$ and $\delta=100$ are (251.54, 18.53), (144.18, 3.92) and (1.74, 0.09) respectively. The other entries in tables I and II were similarly computed. As the number of substations in Tucson was fewer, we varied the percentage from $40\%$ to $60\%$, instead of $15\%$ to $35\%$ as was done for Phoenix. The ratios of the SWEEP to Optimal range from 1.5 to 3.44 for Phoenix dataset and from 1.2 to 1.39 for the Tucson dataset. The computational times for the optimal solution was multiple hours, whereas the SWEEP only took a few seconds. Accordingly, one can conclude that SWEEP is quite effective. Please note that some of the standard deviation values in the tables are zeroes. This is due to the fact that for some datasets, SWEEP produced degenerate solutions and the Optimal found only one solution after 24 hours of computation.

We have investigated a generalization of minimum delay constrained spanning trees: instead of considering trees, we considered subgraphs where removal of $R$ edges does not disconnect any vertex from a designated root node. In particular, we focused on the special case where $R=1$, which we refer to as the RDCMCCC problem, that is, removal of a single edge does not disconnect the edges from the source node while still satisfying the delay constraints. We have shown that even when $R=1$ the problem is NP-complete. Despite the difficulty of the problem, we have presented a novel ILP formulation based on network flows, and shown that in some special cases, the resulting constraint matrix is likely unimodular. We have also created a polynomial time heuristic which has been used to solve the Euclidean version of RDCMCCC.

In the future we hope to extend this work by considering other heuristics that may perform better. In addition, we hope that there may be fast approximation algorithms that grant sharp theoretical bounds on the cost of such structures.