PPCU: Proportional Per-packet Consistent Updates for Software Defined Networks - A Technical Report

A network is composed of a data plane, which examines packet headers and takes a forwarding decision by matching the forwarding tables in switches, and a control plane, which builds the forwarding tables. A Software Defined Network (SDN) simplifies network management by separating the data and control planes and providing a programmatic interface to the control plane.

SDN applications running on one or more controllers can program the network control plane dynamically by updating the rules of the forwarding tables of switches to alter the behaviour of the network. Every Rules Update consists of a set of updates in a subset $S$ of the switches $B$ in a network. Every update in switch $s_{i}\in S$ consists of two sets of rules $R_{1}$ and $R_{0}$. Rule set $R_{0}$ is to be deleted and rule set $R_{1}$ is to be inserted. One of $R_{0}$ or $R_{1}$ may be NULL. The rules are denoted by $s_{i}(R_{1},R_{0})$. Every switch $s_{i}$, $s_{i}\in S$, is called an affected switch. All other switches $(B-S)$ are unaffected switches. The set of rules that is neither inserted nor deleted is an unaffected set of rules and is denoted by $R_{u}$.

Packets that traverse the network during the course of an update must not be dropped (drop-freedom) and must not loop (loop-freedom) on account of the update. In Figure 1(a), when switches are updated to change a route from the old path (solid lines), to the new path (dotted lines), if $s_{f}$ is updated first, packets arriving at $s_{1}$ will get dropped, while if $s_{3}$ is updated first, packets arriving at $s_{3}$ will loop. Flows in a network may need to pass through a number of waypoints in a certain order at all times (waypoint invariance). In Figure 1(b), $s_{1}$, $s_{3}$ and $s_{5}$ are the waypoints to be traversed in that order and this cannot be achieved by sequencing the updates in any order[1]. While the above three properties can be preserved in many update scenarios by finding a suitable order of updates [1, 2, 3, 4], the example in Figure 1(c) cannot be solved by suitably sequencing updates, in any scenario. Initially the network has a policy $P_{1}$ which requires an ingress switch $s_{i}$ to drop packets whose ports are in the range $p_{1}-p_{2}$ and $s_{e}$, an egress switch, to drop packets whose destination addresses are in the range $IP_{1}$ to $IP_{2}$. Policies may be distributed over various switches in an SDN to reduce load on middleboxes or on links to them [5] or due to lack of rule space on a switch [6]. When the policy is changed to $P_{2}$, the network administrator desires either $P_{1}$ or $P_{2}$ to be applied to any packet $p$ and not, for example, $P_{1}$ at $s_{i}$ and $P_{2}$ at $s_{e}$. We call this property pure per-packet consistency (pure PPC).

In order to preserve all the properties for all scenarios, every Rules Update must be per-packet consistent [7] (PPC): every packet $P$ travelling through the network must use either rule $r_{0}\in R_{0}$ in every affected switch $s_{i}\in S$ or rule $r_{1}\in R_{1}$ in every affected switch $s_{i}$, and never a combination of both, during a Rules Update. Other rules in the switches are to be used if they match. The number of packets that get subjected to any of the inconsistencies can be quite large because the time to update a few rules at a switch, which is of the order of milliseconds [8], is far greater than the time to switch a packet out of a switch, which is of the order of nanoseconds, given that the line rate of switches is 10-100 Gbits/s on 10-100 ports [9] and assuming an average packet size of 850 bytes [10].

The basic algorithm that preserves PPC is a two-phase update (2PU) [7]: All the rules in all the switches check for version numbers. All the incoming packets are labelled with the appropriate version number by all the ingresses. To update a set of rules from $v_{0}$ to $v_{1}$, first the new $v_{1}$ rules and a copy of the unaffected rules are installed to check for $v_{1}$, in every internal ( non-ingress) switch. Now all the ingresses start labelling all the packets as $v_{1}$. All $v_{1}$ packets match only the new rules and the $v_{0}$ packets in the network match only the old rules, preserving PPC. After all the $v_{0}$ packets exit the network, the controller deletes the old rules on all the switches. 2PU affects all the switches in the network.

We propose an update algorithm called Proportional Per-packet Consistent Updates, PPCU, that is general, preserves PPC, confines changes to only the affected switches and rules and provides an all-or-nothing semantics for a Rules Update, regardless of the execution speeds of switches and links.

The summary of PPCU is as follows: Let the latest time at which all the switches in $S$ install the new rules be $T_{last}$. Each affected switch examines the time stamp, set to the current time by the ingresses, in each data packet. If its value is less than $T_{last}$, it is switched according to the old rules while if it is greater than or equal to $T_{last}$, it is switched according to the new rules.

Why preserving PPC is important: Proper installation of policies require preservation of PPC and no less. Middle box deployment in SDN [5] [11] rely on packets traversing middle boxes in a certain order [12] [13]. Languages [14] that enable SDN application developers to specify and use states, also require the order of traversal of switches that store those states to be preserved during an update. These examples emphasize the importance of preserving waypoint invariance for all scenarios and therefore of PPC. For many algorithms that preserve more relaxed properties, a PPC preserving update is a fall-back option, for example, if they cannot find an order of updates that preserve waypoint invariance [1] or if their Integer Linear Programming algorithms do not converge [15].

Why the number of switches involved in the update must be less: Algorithms that perform a Rules Update while preserving PPC either 1) need to update all the switches $B$ in the network [7] [16] or 2) identify paths affected by $S$ and update all the switches along those paths [1] [17] or 3) update only $S$ and all the ingresses [18] [19], regardless of the number of elements in $S$ and regardless of the number of rules to be updated.

If paths affected by a Rules Update are identified (category 2), every switch in the path, which may be across the network, needs to be changed, even to modify one rule on one switch. If one rule affects a large number of paths, such as “if TCP port=80, forward to port 1”, computing the paths affected is time consuming [1]. In update methods of categories 1 and 2, the number of rules used in every switch modified during the update doubles, as the new and old rules co-exist for some time. Hence the update time is disproportionately large, even if the number of rules updated is small.

If rules can be proactively installed on switches to prevent a control message being sent to a central controller for every missing rule [20], one controller can manage an entire data centre. In a data centre with a Fat Tree topology with a k-ary tree [21] where the number of ports $k=48$, the number of ingress switches is about $92\%$ of the total number of switches. Even to modify one rule in a switch, all of the ingresses will need to be modified, for algorithms of category 3.

Updating only switches in $S$ instead of $S$ and other switches in $B-S$ will reduce the number of controller-switch messages and the number of individual switch updates that need to be successful for the Rules Update to be successful. We wish to characterize this need as footprint proportionality (FP), which is the ratio of the number of affected switches for a Rules Update to the number of switches actually modified for the update. In the best case the FP is 1, and there is no general update algorithm that achieves this while maintaining PPC, as far as we know.

Why using rules with wildcards is important: Installing wildcarded rules for selected flows or rule compression in TCAM for flows or policies [22] reduces both space occupancy in TCAM [23] and prevents sending a message to the controller for every new flow in the network [20]. [24] provides solutions for the tradeoff between visibility of rules and space occupancy. Thus in a real network, using wildcarded rules is inevitable. Many of the existing solutions do not address rules that cater to more than one flow [8] [1]. A Rules Update may require only removing rules and not adding any rule or vice versa, on one or more switches in a network. Though [19] addresses this, it needs changes to all the ingresses for all the rules for any update and limits concurrency. We wish to have $FP=1$ for all the update scenarios possible.

Why concurrent updates must be supported: Support of multiple tenants on data center networks require updating switches for virtual to physical network mapping [25, 26]. Applications that control the network [27], [28], [29] and VM and virtual network migration[30] will require frequent updates to the network, making large concurrent updates common to SDNs. Existing algorithms that preserve PPC and allow concurrent updates limit the maximum number of concurrent updates [18] [19] [1], with the last one becoming impractical if the number of paths affected by the update is large. 2PU [7] does not support concurrency.

Why algorithms that use data plane time stamping are inadequate: Update algorithms that use data plane time stamping either cater to only certain update scenarios or require very accurate and synchronous time stamping [31] or do not guarantee an all-or-nothing semantics for Rules Updates as they depend on an estimate for the time at which the update must take place [32] [33], thus relying on the execution speeds on switches and links to be predictable.

Our contributions in this paper are:
1) We specify and prove an update algorithm PPCU that: a) requires modifications to only the affected switches $S$ and affected rules, while preserving PPC. b) allows practically unlimited concurrent non-conflicting updates. c) allows all update scenarios: this includes Rules Updates that involve only deletion of rules or only insertion of rules or a combination of both and Rules Updates that involve forwarding rules that match more than one flow. d) expects the switches to be synchronized to a global clock but tolerates inaccuracies. e) updates all of $S$ or none at all, irrespective of the execution speeds of the switches or links.
2) We prove the algorithm, analyze its significant parameters and find them to be better than comparable algorithms.
3) We illustrate that the algorithm can be implemented at line rate.

With the advent of line rate switches whose parsers, match fields and actions can be programmed in the field using languages such as P4 [34] and Domino [9], it has become possible to write algorithms for the data plane in software. Programmable switch architectures such as Banzai [9], RMT[35], FlexPipe[36] (used in Intel FM6000 chip set[37]) and XPliant[38] aid this and languages to develop SDN applications leveraging these abilities [39][14] and P4 applications [40] [41] indicate acceptability.

Data plane programmability is important to PPCU for several reasons: 1) It is easier to change protocols to add and remove headers to and from packets 2) It is possible to process packet headers in the data plane, still maintaining line rate 3) Once the packet processing algorithm is specified in a language, the language compiler finds the best arrangement of rules on the physical switch [42]. It is possible that rules that require a ternary match may be stored in TCAM and exact match rules in SRAM [35] or the former in a Frame Forwarding Unit and the latter in a hash table [36]. 4) The actual match-field values in the rules, the actions, the parameters associated with actions and the variables associated with rules can be populated and modified at run time. How to do this is a part of the control plane and is proposed to be “Openflow 2.0” [34]. Specifications such as SAI[43] or Thrift [44] or the run-time API generated by the P4 compiler [45] [41] may be used at present. We assume that switches support data plane programmability and can be programmed in P4.

Description of a programmable switch model: The abstract forwarding model advocated by protocol independent programmable switches [45] consists of a parser that parses the packet headers, sends them to a pipeline of ingress match-action tables (the ingress pipeline) that in turn consist of a set of match-fields and associated actions, then a queue or a buffer, followed by a pipeline of egress match-action tables (the egress pipeline). In this section, we describe only the features of P4 that are relevant to PPCU.

A P4 program consists of definitions of 1) packet header fields 2) parser functions for the packet headers 3) a series of match-action tables 4) compound actions, made of a series of primitive actions and 5) a control flow, which imperatively specifies the order in which tables must be applied to a packet. Each match-action table specifies the input fields to match against; the input fields may contain packet headers and metadata. The match-action table also contains the actions to apply, which may use metadata and registers. Metadata is memory that is specific to each packet, which may be set by the switch on its own (example: value of ingress port) or by the actions. It may be used in the match field to match packets or it may be read in the actions. Upon entering the switch for the first time, the metadata associated with a packet is initialised to $0$ by default. A register is a stateful resource and it may be associated with each entry in a table (not with a packet). A register may be written to and read in actions.

P4 provides a set of primitive actions such as $modify\_field$ and $add\_header$ and allows passing parameters to these actions, that may be metadata, packet headers, registers etc. When the primitive action $resubmit$ is applied to the ingress pipeline, a packet completes its ingress pipeline and then resubmits the original packet header and the possibly modified metadata associated with the packet, to the parser; the metadata is available for matching. If there are multiple $resubmit$ actions, the metadata associated with each of them must be made available to the parser when the packet is resubmitted. Similarly, when $recirculate$ is applied to a packet in the egress pipeline, the packet, with its header modifications and metadata modifications, if any, is posted to the parser. Conditional operators are available for use in compound actions to process expressions (we use if statements in the paper to improve readability).

While this specifies the definition of the programmable regions of the switch, actual rules (table entries) and the parameters to be passed to actions need to be specified and is described below. These need to be populated by an entity external to this model - the controller, through the switch CPU. This is facilitated by a run-time API.

Description of Rules and packets: A switch has an ordered set of rules $K$, consisting of $n$ rules $r_{1},r_{2},...r_{n}$. Each rule $r$ has three parts: a priority $P$, a match field $M$ and an action field $A$ and is represented as $r=[P,M,A]$. Let $r_{1}=[P_{1},M_{1},A_{1}]$ and $r_{2}=[P_{2},M_{2},A_{2}]$. $r_{1}\prec r_{2}$ if $P_{1}>P_{2}$. $M$ must be as per the match field defined in the table and $A$ must be one of the actions associated with the table. A packet $P$ only has match fields. An incoming packet is matched with the match fields of the rules in the first table specified in the control flow of the switch and the action associated with the highest priority rule that matches it is executed. The packet is then forwarded to the next match-action table, as specified in the control flow.

In $K$, some rules may be dependent on the other. For example, in Figure 2, the rules $a_{1}$, $a_{3}$, $a_{4}$ and $a_{14}$ depend on each other. The solid arrows show the existing dependencies among rules and the dotted arrows, the dependencies after a Rules Update.

$r$ with suitable subscripts or superscripts denotes an individual rule, while $R$ with suitable subscripts or superscripts denotes a set of rules, with $r_{0}\in R_{0}$, $r_{1}\in R_{1}$ and $r_{u}\in R_{u}$, throughout the paper. A packet that matches any $r\in R_{1}$ in any $s\in S$ is called a new packet. A packet that matches any $r\in R_{0}$ in any $s\in S$ is called an old packet and all old and new packets are called affected packets. The first affected switch containing at least one of $R_{1}$ or $R_{0}$ that a packet matches is denoted as $s_{f}$.

Disjoint Rules Updates: Let $P_{1}$ be the set of all packets that match at least one of $R_{0}$ or $R_{1}$ of a Rules Update $U_{1}$ and let $P_{2}$ be the set of all packets that match at least one of $R_{0}$ or $R_{1}$ of another Rules Update $U_{2}$. $U_{1}$ and $U_{2}$ are said to be disjoint or non-conflicting if $P_{1}$ and $P_{2}$ are disjoint. If $U_{1}$ and $U_{2}$ are disjoint, they can occur concurrently and every packet in the network will be affected by at most one of the updates $U_{1}$ or $U_{2}$. For example, let $R_{1}=\{[5,1000,forward\ 10]\}$. Let $s_{i}(null,R_{1})$ be the only update in $U_{1}$. Let $R_{0}=\{[5,1***,forward\ 5]\}$. Let $s_{j}(R_{0},null)$ be the only update of $U_{2}$. $U_{1}$ and $U_{2}$ conflict. Let $R_{1}^{\prime}=\{[5,1111,forward\ 10]\}$. Let $s_{i}(null,R_{1}^{\prime})$ be the only update in $U_{3}$. $U_{1}$ and $U_{3}$ are disjoint. One Rules Update can consist of one or more disjoint updates. In Figure 2, $a_{3}$ and $a_{7}$ need to be deleted and $a_{2}$, $a_{6}$ and $a_{10}$ need to be installed. They can all be part of the same Rules Update.

Relationship between the old and the new rules: Two ordered sets of rules $A$ and $B$ are said to be match-field equivalent if the total set of packets that they are capable of matching are identical. Thus $A=\{[P_{1},0000,A_{1}],[P_{2},01**,A_{2}]\}$ and $B=\{[P_{1},0000,B_{1}],[P_{2},0100,B_{2}],[P_{3},01**,B_{3}]\}$ are match-field equivalent.

The set of rules in $A$ for which no match-field equivalent rules of priority equal to or lower than $A$ exists in $B$ is called the special difference of $A$ and $B$, denoted $A-B$. If $R_{1}$ = $\{[P_{1},0000,A_{1}],[P_{2},00**,A_{2}]\}$ and $R_{0}$=$\{[P_{2},000*,B_{1}]\}$, where $P_{1}>P_{2}$, $R_{1}-R_{0}={[P_{2},00**,A_{2}]}$. The set of rules in $A$ for which no match-field equivalent rules of priority equal to or greater than $A$ exists in $B$ is called the inverse special difference of $A$ and $B$, denoted $A\sim B$. If $R_{0}$ = $\{[P_{2},00**,A_{2}]\}$ and $R_{1}=\{[P_{1},000*,B_{1}]\}$, where $P_{1}>P_{2}$, $R_{0}\sim R_{1}={[P_{2},00**,A_{2}]}$.

Either or both of $R_{0}\sim R_{1}$ and $R_{1}-R_{0}$ may not be $\phi$ for every affected switch. For example, in Figure 2, $\{a_{7}\}\sim\{a_{6}\}\neq\phi$ and $a_{10}$ has no corresponding old rule. Thus, both $R_{0}\sim R_{1}$ and $R_{1}-R_{0}$ are not $\phi$. This has implications for the update algorithm, as discussed in section V.

Providing an FP of 1 is difficult because multiple disjoint updates may be combined into one Rules Update and a switch rule may belong to more than one flow, due to usage of wild carded rules. Thus for packets belonging to different flows the first affected switch $s_{f}$ may be different - that is, there may be more than one $s_{f}$ belonging to a Rules Update. Neither the controller nor the switches know the paths that are affected by the update or the $s_{f}$ for each path. Due to these reasons, firstly, the controller cannot instruct a specific switch to relabel packets to switch to the new version of rules, as is done in some of the update algorithms [1]. Secondly, all switches must have the same algorithm, regardless of their position in a flow. Moreover, the rules to be deleted and installed and the unaffected rules depend on each other in complex ways, as discussed in 2, causing difficulty in maintaining PPC. The key insight is that all the affected switches must know if the rest of the affected switches and itself are ready to move to the next version, and after that, if a packet crosses its $s_{f}$, it must store that fact in its header ; the switch examines $TS$, the time stamped at the ingress on every packet, to understand the readiness and stores the result in two one-bit fields $f_{p1}$ and $f_{p2}$ in the packet.

Since the controller-switch network is asynchronous, when a controller sends messages to switches, all switches will not receive them at the same time, or complete processing them and respond to the controller at the same time or they may not respond at all. Solving race conditions arising out of these scenarios is possible if the values of $f_{p1}$ and $f_{p2}$ which are set by $s_{f}$ determine whether they match $R_{0}$ or $R_{1}$ or $R_{u}$, regardless of what the states of the switches in its path are. Also, the clocks at the switches may be out of sync to the extent the time synchronising protocol would permit. In the next section, we shall state the algorithm first and then describe the measures taken in them to address the above challenges.

Data plane changes at the ingress and egress switches: Each packet $p$ entering the network has a time stamp field and two one bit fields $f_{p1}$ and $f_{p2}$ added to it and removed from it programmatically, at the ingress and the egress, respectively. All ingresses set $TS$ to the current time at the switch and $f_{p1}$ and $f_{p2}$ to $0$ for all the packets entering it from outside the network. Altering the value of $TS$, wherever required in the update algorithm, occurs only after it is thus set.

Notations used: $flag$ indicates if the rule is new ($NEW$), old ($OLD$) or unaffected ($U$). $f_{p1}$, when set to $1$, indicates that the packet must be switched only according to $OLD$ rules, where they exist and $f_{p2}$, when set to $1$ indicates that the packet must be switched only according to $NEW$ rules, where they exist. The metadata bit $f_{1}$ when set to $1$ for a packet indicates that this packet must not be matched by a $NEW$ rule. The metadata bit $f_{2}$ when set to $1$ for a packet indicates that this packet must not be matched by an $OLD$ rule.

Algorithm at the control plane: This section first specifies the algorithm for the control plane, at the controller (Algorithm 1) and at each $s_{i}\in S$ (Algorithm 2) and later for the data plane, in Algorithm 3. The message exchanges below are the same as in [16] and [18]; the parameters in them and actions upon receiving them have been modified to suit PPCU.

1. 1.

   The Controller receives an Update Request from the application, with the list of affected switches $S$, $R_{0}$ and $R_{1}$, for every switch $s\in S$. The actions of all rules are as in Algorithm 3. An update identifier $v$ is associated with each update. The controller sends a “Commit” message to every switch $s\in S$ (line 5 of Algorithm 1) with $v$, $R_{0}$ and $R_{1}$ as its parameters.¹¹1It is assumed that the controller and the switch maintain all parameters related to an update in a store and check whether each message received is appropriate; we omit these details and the error handling required for an all-or-nothing semantics to improve clarity.

2. 2.

   A switch $s\in S$ receives the “Commit” message, extracts $v$, $R_{0}$ and $R_{1}$. The switch 1) changes the match part of the installed $R_{0}$ rules to check if $f_{2}=0$ and $f_{p2}=0$ 2) installs the $R_{1}$ rules and 3) sets $flag=OLD$ for the $R_{0}$ rules and $flag=NEW$ for the $R_{1}$ rules. The above actions must be atomic, as indicated in the algorithm. Now it sends “Ready to Commit” with the current time of the switch $T_{i}$ as a parameter (lines 3 to 16 in Algorithm 2).

3. 3.

   The controller, upon receiving “Ready To Commit”, stores the current time received. Let $T_{last}$ be the largest value of time received in “Ready To Commit” (Algorithm 1, line 10). Now it sends “Commit OK” to all switches $s\in S$ with $T_{last}$ (Algorithm 1, line 11).

4. 4.

   Upon receiving “Commit OK”, the switch sets $T=T_{last}$ in $R_{0}$ and $R_{1}$, which is considered a single atomic action. It now sends the current time $T_{i}$ in “Ack Commit OK”. See lines 17 to 28 in Algorithm 2.

5. 5.

   The controller receives “Ack Commit OK” from all the switches. Let the largest value of time received in “Ack Commit OK” be $T_{del}$. It sends “Discard Old” to all the switches $s$ with $T_{del}$.

6. 6.

   Let the time at which a switch $s$ receives “Discard Old” be $T_{i}$. The switch sends “Discard Old Ack” to the controller. The switch starts a timer whose value is $T_{del}+M-T_{i}$, where $M$ is the maximum lifetime of a packet within the network and $T_{del}$ the time received in “Discard Old”. (line 33 in Algorithm 2). When the timer expires, all the packets that were switched using the rules $R_{0}$ are no longer in the network. Therefore the switch deletes $R_{0}$. It sets $flag=U$ for every $R_{1}$ rule and modifies its $f_{1}$, $f_{2}$, $f_{p1}$ and $f_{p2}$ to check for $*$, as shown in lines 35 to 40, in Algorithm  2. With this, the update at the switch is complete.

7. 7.

   After the controller receives “Discard Old Ack” messages from all the switches, the update is complete (line 21 in Algorithm 1) at the controller.

Note: After $M$ units after timer expiry at the last affected switch, the last of packets tagged $f_{p2}=1$ will exit the network. Now the next update not disjoint with the current one may begin.

Algorithm at the data plane - actions at switches:

Algorithm 3 specifies the template for a compound action associated with every rule in a match-action table in a switch. Each rule has two metadata fields of $1$ bit each, $f_{1}$ and $f_{2}$, associated with it, initialised to $0$ by default, indicating that the packet is entering that table in that switch for the first time (If other tables in the same switch has rule updates, a separate set of two bits need to be used for each of those tables. For ease of exposition, we describe only one table being updated). Each rule has two registers, $T$, initialised to $T_{max}$, which is $1$ less than the maximum value that $TS$ can have, and $flag$, which decides if the rule is old ($OLD$), new ($NEW$) or unaffected ($U$), initialised to $U$. Unaffected rules have $f_{1}$, $f_{2}$, $f_{p1}$ and $f_{p2}$ set to $*$ in their match-fields and do not check for the value of $TS$. A new rule is always installed with a priority higher than that of an old rule. In P4, rules with ternary matches have priorities associated with them as such rules can have overlapping entries.

A rule may execute any of its own actions, which is what is referred to in 7, 18 and 31 in Algorithm 3. The parameters $action\_params$ that are passed to the compound action are in turn passed to the actions associated with the rule. For example, a new rule may require a packet to be forwarded to port $5$, instead of port $6$. Then, the table entry for the new rule would pass $5$ as a parameter from its rule, while the old rule would pass $6$.

Suppose $s_{f}$, the first affected switch of affected packet $p_{1}$ in Rules Update $U_{1}$, receives “Commit”. $TS<T_{max}$ of $p_{1}$ and let $R_{1}-R_{0}=\phi$ for $s_{f}$. $p_{1}$ will match $r_{1}$ in $s_{f}$, get its $f_{1}$ set to $1$ and get resubmitted (line 12). Now it will not match $r_{1}$ and instead match $r_{0}$, get its $f_{p1}$ set to $1$ (line 17) and get switched by $r_{0}$ in subsequent affected switches and $r_{u}$, if no $r_{0}$ exists. After $s_{f}$ receives “Commit OK” and sets its $T=T_{last}$, when a packet $p_{2}$ whose $T>T_{last}$ arrives at $s_{f}$, it gets switched by the new rule, gets its $f_{p2}$ set to $1$ (line 6) and gets switched by $r_{1}$ rules in all the subsequent affected switches and $r_{u}$ if no $r_{1}$ exists. Suppose in Rules Update $U_{2}$, $s_{f}$ has $R_{1}-R_{0}\neq\phi$ and receives “Commit”. Now a packet $p_{3}$ entering it with $TS<T_{max}$ gets its $f_{1}$ set to $1$, and then resubmitted(line 12). The resubmitted packet matches another rule $r_{u}$ and gets its $f_{p1}$ set to $1$ (line 27), making the packet use $r_{0}$ rules in subsequent switches and $r_{u}$, where no $r_{0}$ exists. After $s_{f}$ receives “Commit OK”, a packet $p_{4}$ whose $T>T_{last}$ matches $r_{1}$, gets its $f_{p2}$ set to $1$ and gets switched in subsequent switches by $r_{1}$ or $r_{u}$, if $r_{1}$ does not exist. The case for an update where $s_{f}$ has $R_{0}\sim R_{1}\neq\phi$ is similar.

Specific scenarios: We shall use a list (inexhaustive) of scenarios to illustrate that each step in Algorithm 3 is necessary: Consider two switches $s_{f}\in S$ and $s_{j}\in S$, in Figures 3 and 4, far apart from each other in the network. In all the cases, the lower priority rule forwards a packet to a port while a higher priority rule forwards a packet to the same port and increments a field $F$ in the packet. The value of $TS$ and the rest of the packet header are shown in boxes with dotted lines, before and after crossing $s_{f}$. The numbers $1$, $2$ and $3$ against the boxes indicate the sequence of occurrence of events.

Each disjoint Rules Update requires a unique update identifier $v$ for the duration of the update, to track the update states at the affected switches and the controller. The number of disjoint Rule Updates that can be simultaneously executed is limited only by the size of $v$. $v$ is exchanged only between the controller and the switches and hence is not dependent on the size of a field in any data packet. Therefore as many disjoint updates as the size of the update identifier or the processing power of switches would allow can be executed concurrently.

Feasibility of adding $TS$ at the ingress: We assume that all switches have their clocks synchronized and the maximum time drift $\gamma$ of switches from each other is known. If the network supports Precision Time Protocol, $\gamma=1\mu sec$ [31]. Intel FM6000, a programmable SDN capable switch, supports PTP, its $\gamma<1\mu sec$ and the time stamp is accessible in software [37]. The size of the register used to store a packet timestamp in FM6000 is $31$ bits. P4 supports a feature called “intrinsic metadata”, that has target specific semantics and that may be used to access the packet timestamp. Using the $add\_header$ and $remove\_header$ actions and intrinsic metadata, the $TS$ field may be added at the ingress and removed at the egress for every packet, for targets that support protocols such as PTP.

Asynchronous time at each switch: Since a single rule update in a TCAM is of the order of milliseconds, a time stamp granularity of milliseconds is sufficient for Rules Updates. Let us assume that the timestamp value is in milliseconds and that an ingress $s_{i}$ is faster than an affected internal switch (at the most by $1\mu sec$). Let $t_{1}$ be the time stamp of a packet $p$ in the network that is stamped by $s_{i}$, even before the Rules Update begins. After the Rules Update begins, let the (temporally) last affected switch send its current time stamp $T_{last}$ in “Ready To Commit”. Let $t_{1}>T_{last}$, since the ingress is faster. Now a switch $s_{j}\in S$, which is not an $s_{f}$, will switch $p$ with the new rules, violating PPC. To prevent this, each switch may set $T=\left\lceil{T_{last}+\gamma+1}\right\rceil$, instead of $T=T_{last}$. If an ingress clock is slow, there will be no PPC violations. Thus PPCU tolerates known inaccuracies in time synchronisation. To reduce packet transmission times, the size of $TS$ may be reduced, thus reducing the granularity. $s_{f}$ will need to wait longer to receive packets whose $TS\geq T_{last}$, thus lengthening the Rules Update time. Since the size of $TS$ is programmable in the field, the operator may be choose it according to the nature of the network.

We need to prove that the algorithms 1, 2 and 3 together provide PPC updates. $p(m,f_{p1},f_{p2})$ denotes an affected packet with match-field $m$ and fields $f_{p1}$ and $f_{p2}$. $r(m)$ denotes a rule whose match-field is $m$ (in both, $m$ excludes $TS$, $f_{p1}$ and $f_{p2}$). By definition of $S$, for a packet $p(m,f_{p1},f_{p2})$, any $s\in S$ has either a rule $r_{0}(m)$ or $r_{1}(m)$ or both.

Let us assume that the individual algorithms 1, 2 and 3 are correct. Let us assume that switches are synchronized, to make descriptions easier. In the description below, when we refer to a packet, we mean a packet affected by the update. The update duration is split into three intervals: before $s_{f}$ receiving “Commit”, from $s_{f}$ receiving “Commit” to the timer expiring, and after the timer expiry.

Property 1: After the Rules Update begins and before $s_{f}$ receives “Commit”, PPC is preserved: $s_{f}$ (and all $s_{j}$ that have not received “Commit”) switches all packets using a rule that will be marked $r_{0}(m)$ on receiving “Commit” or $r_{u}(m)$ (if no rule is to be deleted on that switch). The first $s_{j}$ that receives “Commit” switches using $r_{0}(m)$ or $r_{u}(m)$ (the latter if the switch does not have $r_{0}(m)$ but has only $r_{1}(m)$) and sets $f_{p1}=1$ for all received packets. Subsequent $s_{j}$s will not use $r_{1}(m)$ as $f_{p1}=1$ and use $r_{0}(m)$ if it exists (or $r_{u}(m)$ if no $r_{0}(m)$ exists) . Thus packets follow $r_{0}(m)$ if it exists or $r_{u}(m)$ if $r_{0}(m)$ does not exist, preserving PPC.

Property 2: Between $s_{f}$ receiving “Commit” at $s_{f}$ and its timer expires PPC is preserved: Property 2.1: As per Algorithm 3, if $s_{f}$ has received “Commit”, $p(m,f_{p1},f_{p2})$ will exit $s_{f}$ as either $p(m,0,1)$ or $p(m,1,0)$, until its timer expires.

Property 2.2: $s_{j}$ will receive any of the types of packets $1$) $p(m,0,0)$ with its $TS<T_{last}$ or $2$) $p(m,1,0)$ or $3$) $p(m,0,1)$ and no other type of packet, until its timer expires. Proof: Packets of type $1$ may reach $s_{j}$ as it may have crossed $s_{f}$ before $s_{f}$ received “Commit” and $TS$ will always be less than $T_{last}$ of such packets, as $T_{last}$ is the time at which the last switch has received “Commit”. If a packet of type other than these three reaches $s_{j}$, it means it has not crossed $s_{f}$, by Property 2.1. In that case, the only possibility is that $s_{j}$ is the first switch, which is not true by defintion.

By Algorithm 3, packets of types $1$ and $2$ referred to in Property 2.2 can use only $r_{0}$ rules (or $r_{u}$ if $r_{0}$ does not exist) on all switches $s_{j}$. By the same algorithm, packets of type $3$ match only $r_{1}$ rules (or $r_{u}$ if $r_{1}$ does not exist) on all switches $s_{j}$. In $s\notin S$, packets match $r_{u}$. $r_{u}$ can change the value of $f_{p1}$ or $f_{p2}$ of a packet only if $f_{1}$ or $f_{2}$ for that packet is set, which occurs only on an affected switch. Therefore, PPC is preserved.

Property 3: When the timer expires at any switch $s_{i}\in S$, all packets that were switched using $r_{0}$ at least at one $s\in S$ have left the network: Let us consider two cases: 1) All packets whose $TS<T_{last}$ get switched by $s_{f}$ using $r_{0}$ rules, by Algorithm 3. Such packets exit the network by time $T_{last}+M$. However, $T_{del}>T_{last}$. 2) All packets whose $TS\geq T_{last}$ that arrive at $s_{f}$ get switched using $r_{0}$, until that $s_{f}$ receives its “Commit OK”, by Algorithm 3. At time $T_{del}$, the last of $s_{f}$s have received “Commit OK” and have started switching packets using $r_{1}$. Hence at time $T_{del}+M$, all the packets (whether due to case 1 or 2) that were switched using $r_{0}$ have left the network. The current time at a switch is $T_{i}$ and the switch receives an instruction to start a timer that expires at $T_{del}+M$. The elapsed time, that is, $T_{i}-T_{del}$ must be subtracted from $M$ , giving the timer a value of $M-T_{i}+T_{del}$. Therefore when this timer expires, all the old packets have left the network.

Property 4: After timer expiry at $s_{f}$: All packets in the network have their $TS\geq T_{last}$ after timer expiry at $s_{f}$, as all packets whose $TS<T_{last}$ have left the network by the timer expiry at any switch, by Property 3. Therefore all packets leaving $s_{f}$ are of the form $p(m,0,1)$. Therefore all switches $s_{j}\neq s_{f}$ will use $r_{1}$ if it exists (or $r_{u}$ if $r_{1}$ does not exist), if the timer has not expired on $s_{j}$. If the timer has expired on $s_{j}$, it will only cause it to not check for $f_{p1}$, $f_{p2}$, $f_{1}$ or $f_{2}$ , which makes no difference to matching the rule that the packet matches. Since packets have only $r_{1}$ to match (or $r_{u}$ where it does not exist), PPC is preserved.

Thus due to Properties 1 to 4, PPC is preserved during and after the update.

The symbols used in the analysis are: $\delta$: the propagation time between the controller and a switch, $t_{i}$: the time taken to insert rules in a switch TCAM, $t_{d}$: the time taken to delete rules from the TCAM, $t_{m}$: the time taken to modify rules in the TCAM, $t_{v}$: the time taken to modify registers associated with a rule, $t_{s}$: the time for which a switch waits after it receives “Discard Old” and before it deletes rules, $n_{o}$: the number of old rules that need to be removed, $n_{n}$: the number of new rules that need to be added, $n$: the maximum number of rules in a switch, $k_{a}$: the number of affected switches, $k_{i}$: the number of ingresses, $k_{t}$: the total number of switches, $R_{1}$: The time between the switch receiving “Commit” and sending “Ready To Commit”, $R_{2}$: The time between the switch receiving “Commit OK” and sending “Ack Commit OK” and $R3$: The time between the switch receiving “Discard Old” and the switch performing its functions after timer expiry. It is assumed that the value of $\delta$ is uniform for all switches and all rounds, the values of time are the worst for that round, the number of rules, the highest for that round and that the processing time at the controller is negligible. Since unaffected rules have ternary matches for $f_{p1}$ and $f_{p2}$, we assume that all the match fields are stored in TCAM and the corresponding actions in SRAM [35] [42], for PPCU, and for other algorithms.