End-to-End Asynchronous Traffic Scheduling in Converged 5G and Time-Sensitive Networks

Fig. 1 shows an application scenario in which a TSN and a 5GS are connected via a gateway (GW). TSN is composed of end-stations (ESs) and TSN switches while 5GS is composed of user equipment (UEs) and a base station (BS). All the flows under consideration originate from UEs in 5GS to the ESs in TSN and belong to time-triggered (TT) flows. Specifically, the source of a TT flow periodically generates fixed-length packets. Each TT flow is subject to the e2e delay requirement for packet delivery.

Additionally, we assume that the clocks in TSN and 5GS are synchronous within their respective domains but can be asynchronous between different domains. Each TSN switch adopts TAS on each port. We also assume that 5GS uses the FDD duplex scheme where downlink and uplink transmissions use different frequencies and all UEs share a common 5G uplink.

Using an example, we show below that the simple integration of 5GS and TSN may enlarge e2e delay and jitter, where the simple integration means TAS’s reservation of a transmission opportunity for each frame of a TT flow on each link on its route.

In Figs. 2a–2d, the UE in 5GS sends a packet every $100ms$ to the ES in TSN via the GW and edge switch (Edge). We will use Figs. 2a and 2b to show the enlargement of delay and jitter caused by the clock skew between 5GS and TSN and the 5G transmission jitter, respectively.

As shown in Fig. 2a, in an ideal situation — i.e., there is no 5G transmission jitter or clock skew between 5GS and TSN — GW and Edge reserve a transmission opportunity every $100ms$ (= the flow period) on their output port, which is shown as black hollow circles, and the black dashed arrows depict the ideal transmission processes. Each packet experiences a constant delay of $45ms$ and the e2e jitter of 0. However, considering the clock skew between the two systems, the packets may arrive at TSN earlier (depicted by the blue line) or later (depicted by the red line) from the perspective of TSN. We assume the maximum clock skew is $10ms$, i.e., the clock of TSN may be at most $10ms$ earlier or later than that of 5GS. On the one hand, packet #1 arrives $10ms$ earlier than the prepared transmission opportunity and has to wait for it, leading to the actual e2e delay of $55ms$. On the other hand, packet #2 arrives $10ms$ later than the prepared transmission opportunity and must wait for the opportunity in the next period, leading to the actual e2e delay of $135ms$. Actually, the experienced e2e delay can be as low as $45ms$ and as large as $145ms$, creating a deviation of $100ms$ which equals the flow period. Thus, a small skew between the clocks of the two systems may lead to a significantly larger e2e delay and jitter than scheduled.

Similarly, as shown in Fig. 2b, we assume that the clocks are synchronized, the delay budget in 5GS is estimated to be the average delay over a period of time, and the transmission opportunities prepared by TSN are scheduled by reserving the delay budget of 5GS. The transmission jitter of 5GS is $10ms$, meaning packets may arrive at most $10ms$ quicker or slower than the ideal arrival time calculated by the delay budget. Packet #1 experiences an e2e delay of $55ms$ while packet #2 experiences $155ms$ and their deviation shown in the figure is thus $100ms$, which also equals the flow period. A small 5G transmission jitter may also lead to a significantly larger e2e delay and jitter than scheduled.

In fact, the e2e delay and jitter enlargement is caused by missing TSN transmission opportunities. To mitigate this enlargement, we propose the AAM to decouple the interaction between 5GS and TSN transmissions.

The basic ideas of the AAM are to (1) hold the packets at the edge switches to shape the resident time in TSN to be constant and (2) over-provision the transmission opportunities to reduce delay.

First, a holding process is introduced in the edge switch. Let $T_{i}$ be the period of the transmission opportunities (not necessarily equals the flow period as will be discussed later). As shown in Fig. 2c, $T_{i}=25ms$. Packets #1 and #2 experience similar transmission processes: (1) waiting at the GW, (2) transmitted from GW to Edge, (3) held at the Edge, and (4) transmitted from Edge to ES, and the corresponding delays are denoted as $f_{i}.wait$, $D^{TSN}_{i,(1)}$, $f_{i}.hold$ and $D^{TSN}_{i,(2)}$. The delay in TSN can be represented as:

where $D^{TSN}_{i,(1)}$ and $D^{TSN}_{i,(2)}$ are both constant assuming TSN is well organized. It is easy to get $f_{i}.wait\in[0,T_{i})$ so the holding time is for compensating the waiting time to be $T_{i}$, i.e., $f_{i}.wait+f_{i}.hold=T_{i}$. As can be seen from Fig. 2c, packet #1 waits for $10ms$ and is held for $15ms$, and packet #2 waits for $15ms$ and is held only for $10ms$. This way, the packets’ resident time in TSN is fixed and can be represented as:

So, all the packets of the flow experience the same delay in TSN no matter when the packets are delivered by 5GS, and the transmission coupling between the two network systems is eliminated, hence mitigating the delay and jitter enlargement. As shown in Fig. 2c, both packets experience the same e2e delay of $70ms$ regardless of the clock skew between the two network systems.

However, in many industrial use-cases [6] the delay requirement is no larger than the period. If $T_{i}=f_{i}.period$, the delay requirement cannot be satisfied according to Eq. (2). This way the transmission opportunities are over-provisioned to reduce delay, i.e., $T_{i}$ should be less than $f_{i}.period$.

Similarly, Fig. 2d illustrates how the AAM can help mitigate the issues presented in Fig. 2b. Moreover, the e2e transmission jitter is isolated only in 5GS.

Note that the parameter $T_{i}$ should be carefully determined to balance between the resource usage and the delay in TSN, and between the delay budgets in TSN and 5GS for the following reasons:

• •

  $T_{i}$ allows for a trade-off between usage of TSN resource and the delay in TSN. According to Eq. (2), the smaller $T_{i}$, the lower the delay in TSN. However, a smaller $T_{i}$ means more TSN resource usage for a single flow as shown in Figs. 2a and 2c.

• •

  $T_{i}$ balances the delay budgets in 5GS and TSN. The e2e delay is composed of 5G and TSN parts:

  	$\displaystyle D_{i}^{e2e}$	$\displaystyle=D_{i}^{5GS}+D_{i}^{TSN}$
  		$\displaystyle=D_{i}^{5GS}+T_{i}+D^{TSN}_{i,(1)}+D^{TSN}_{i,(2)}.$		(3)

  Given the e2e delay requirement of a flow, a larger $T_{i}$ will decrease the delay budget allocated to 5GS which will affect the performance of 5GS.

It is, therefore, necessary to establish a comprehensive model to jointly schedule the resources of 5GS and TSN when the AAM is applied; this will be discussed in Section IV.

The AAM is composed of the following two parts.

• •

  The gateway caches the received packet of $f_{i}$ until the next transmission opportunity, attaches the waiting time $f_{i}.wait$ and sends it to the destination.

• •

  The edge switch of the last hop holds the packet of $f_{i}$ for time duration $f_{i}.hold$ and then delivers it to the final destination, where $f_{i}.hold+f_{i}.wait=T_{i}$.

The implementation details in the gateway and the edge switch are described in Algorithms 1 and 2, respectively.

In Algorithm 1, there are two functions in the gateway. In the first function onReceivingPacketFrom5GS($f_{i}.packet$), once the packet of flow $f_{i}$ is received, the gateway caches the packet in the buffer $f_{i}.buffer$ prepared for this flow (Line 2). If there is already a packet in the buffer, the old packet will be replaced by the new packet. Then, the reception time of the packet is recorded (Line 3). In the second function onTransmissionOpportunity ($f_{i}.timer_{i}$), the function takes a timer as an input parameter. Since a timer is set for each flow $f_{i}$ to inform its transmission, the function first resets the timer for the next transmission opportunity for this flow (Line 5). It then checks whether the buffer of this flow is empty. On the one hand, the function does nothing if the corresponding buffer is empty. On the other hand, it attaches the waiting time to this packet and sends it before clearing the buffer if the buffer is not empty (Lines 6-13).

There is only one function in Algorithm 2. When the edge switch receives the packet $f_{i}.packet$, it delays the packet for the amount of time $f_{i}.T-f_{i}.packet.getWait()$ which is consistent with the process mentioned before (Line 2). The packet is then delivered to the destination after the delay (Line 3).

As shown in Fig. 3, AAM can be easily integrated into the 5GS-TSN architecture proposed in [15]. The base station complies with the 5G Network standard. The two black boxes implement Algorithm 1 and 2, respectively, and they are connected by a traditional TSN network. The frame’s waiting time at the left black box is incorporated into the frame payload and the right black box can extract the time from the received frames. The ATSM is implemented in the controller, whose detailed structure can be found from [15]. The controller collects the state of the converged network and flows’ requirements, and then calculates the settings of 5GS, TSN, and the two black boxes.

The wired TSN network is defined as an undirected graph $G(V,E)$, where $V$ is the set of nodes such as ESs, the gateway, and TSN switches, and $E$ is the set of physical full-duplex links in TSN. Specifically, each physical link contains two dataflow links [23]. Let $L^{TSN}$ denote the set of dataflow links in TSN, then

where ”$(v_{1},v_{2})$” denotes the physical link between nodes $v_{1}$ and $v_{2}$, ”$[v_{1},v_{2}]$” and ”[$v_{2},v_{1}]$” are the corresponding directed dataflow links.

The set of dataflow links in 5GS is denoted as $L^{5GS}$. The entire set of dataflow links in the converged network can then be represented as $L=\{L^{TSN},L^{5GS}\}$.

Let $F$ be the set flows under consideration. Each flow $f_{i}\in F$ is identified as:

where $f_{i}.period$ denotes the period of the flow, $f_{i}.length$ the packet length of the flow in one period, and $f_{i}.delay$ the e2e delay boundary required by the flow.

For each flow, we assume there is only one frame to be delivered within one period, and the route is determined in advance:

where $h_{i}$ is the number of hops for $f_{i}$. Table I summarizes the main notations/symbols used in this paper.

We formally define the transmission opportunities reserved in TSN and 5GS as potential transmission instances.¹¹1Since each transmission opportunity is allocated to an actually transmitted packet in the traditional traffic scheduling model of TSN, each transmission opportunity is also called a transmission instance [24]. However, in the AAM context, not every transmission opportunity corresponds to an actually transmitted packet. So, we extend the concept of transmission instances to potential transmission instances. Let $f^{l}_{i}$ denote the potential transmission instance of flow $f_{i}$ on the dataflow link $l$.

We now derive the constraints of the ATSM which accounts for both 5GS and TSN and schedules flows from an e2e perspective. The constraints introduced in IV-C6, IV-C7, IV-C8, and IV-C9 are borrowed from the mature scheduling model in TSN [6, 22], but are modified, if needed, for the AAM.

In addition to the TT flows, many other types of flows (e.g., best-effort (BE) flows) may exist in an industrial setting. We need to reduce the resources used by TT flows and leave as much resource as possible for the BE flows.

For the 5GS part, we try to minimize the number of RBs assigned to TT flows, which is inspired by [26] (we omit the superscript $l$ for readability as there is only one dataflow link in 5GS on the uplink):

For the TSN part, as we adopt the AAM, over-provisioned transmission opportunities will lead to unused resources. Thus, in TSN, we want to maximize the periods of the transmission opportunities reserved for the flows to reduce waste of resources:

To obtain the final objective function, we multiply Eq. (31) by -1 to change the optimization direction, and then use a weighting factor $\gamma$ to balance the two optimization objectives:

where $\gamma\in[0,1]$ indicates which network we want to optimize more.

The ATSM proposed above can be transformed into an Integer Linear Programming problem. The ”logical or” in Eqs. (18), (24) and (25) and the product of binary variable and bounded variable in Eqs. (16) and (17) can be linearized by the commonly used linearization methods [27]. But there still remains a difficulty: the number of constraints in Eqs. (24) and (25) cannot be determined as the number is related to the $T_{i}$ and $T_{j}$ which are decision variables.

To overcome this difficulty, we adopt the following strategy:

• •

  The minimum resource period can be supported by TSN is denoted as $minP$ whose unit is $\mu s$.

• •

  For any flow $f_{i}$ whose period is $f_{i}.period$, the available resource period for it can only be selected from the list $list_{i}=[minP,2\times minP,4\times minP,\ldots,T_{i,max}]$ whose length is $s_{i}$ and $T_{i,max}$ satisfies:

  	$$\displaystyle T_{i,max}=minP\cdot 2^{s_{i}-1}$$		(33)
  	$$\displaystyle minP\times 2^{s_{i}-1}\leq f_{i}.period$$		(34)
  	$$\displaystyle minP\times 2^{s_{i}}>f_{i}.period.$$		(35)

• •

  For any flow $f_{i}$ we introduce $s_{i}$ binary variables $\{b_{i,j}|j=0,1,\ldots,s_{i}-1\}$, and we let $T_{i}=\sum\limits_{j=0}^{j=s_{i}-1}list_{i}[j]\times b_{i,j}$. Furthermore, we introduce new constraints to make only one element of the list selected:

  	$$\displaystyle\forall f_{i}\in F:$$
  	$$\displaystyle\sum\limits_{j=0}^{j=s_{i}-1}b_{i,j}=1.$$		(36)

This way, the hyper period of the set $\mathbf{T}_{max}=\{T_{i,max}|f_{i}\in F\}$ is multiple of the hyper period of the set $\mathbf{T}=\{T_{i}|f_{i}\in F\}$. Then, we can expand the check range from $[0,\frac{T_{TSN}}{T_{i}}-1]$ to $[0,\frac{LCM(\mathbf{T}_{max})}{minP}-1]$ in Eqs. (16) and (17).

The optimization problem is transformed into an Integer Linear Programming problem. We use Gurobi [28] to solve the problem.

The topology of the simulation network is shown in Fig. 6, which has referenced the medium-sized topology in [6] and is extended with a 5GS, where blue squares represent ESs and red diamonds represent TSN switches. There are 36 ESs in TSN and 20 UEs uniformly distributed around the base station in 5GS. There are a total of 20 flows, consisting of three types, and the specific configuration is shown in Table II, which has referenced [32]. The remaining main simulation parameters are summarized in Table III.

In the first set of experiments, we evaluate the performance of AAM compared to TAM in the presence of the 5G transmission jitter and the clock skew between the two systems. To distinguish between different experiments, we name the experiment conducted by the ATSM introduced in Section IV as the AAM-Scheduling. The baseline is called the TAM-Scheduling, which is done with a scheduling model called the synchronous traffic scheduling model (STSM). The STSM is modified from the ATSM by not applying the AAM, i.e., the period of the potential transmission instances equals the flow period, the packets have to arrive at the GW before the start of the transmission opportunity prepared for it, and no holding process is adopted as in [22].

As shown in Eqs. (37) and (38), we use the coefficient of expansion (CE) and coefficient of variation (CV) of the e2e delay to evaluate the ratios of e2e delay enlargement and e2e jitter, respectively, which are defined respecively as the ratio of the mean actual e2e delay to the scheduled delay and that of the standard deviation of the e2e delay to the mean actual e2e delay:

where $D^{e2e}_{i,Scheduled}$ is the e2e delay calculated by the ATSM or the STSM, which both assume no transmission jitter in 5GS, $\bar{D}^{e2e}_{i,Actual}$ is the mean e2e delay the packets actually experience, and $\sigma(\cdot)$ is the standard deviation calculated by accounting for the e2e delay of each data packet.

Then, the mean coefficient of expansion (MCE) and mean coefficient of variation (MCV) are calculated to evaluate the ratios of e2e delay enlargement and jitter of the whole system:

In the second set of experiments, we evaluate the performance of the ATSM in the resource assignment under different traffic loads and optimization preferences. All flows have the same parameters and the length is $200B$ while the period as well as the delay requirement are $1ms$. We then change the number of flows to denote different traffic loads and change the $\gamma$ to denote different optimization preferences.

We then calculate TSN resource usage and 5GS resource usage, where 5GS resource usage is the ratio of the RBs allocated to the TT flows to the whole number of RBs in 5GS. In addition to the two types of resource usage of the ATSM, TSN resource usage of the STSM is also computed to represent the theoretical limit of TSN resource usage that can be achieved by the ATSM.