Single Threshold Packet Scheduling Policy for AoI Minimization in Resource-Constrained Network

At the transmitter side, the data freshness is only affected by the packet arrival rate $\lambda$. We assume that the transmitter data freshness at the $i$th time slot is denoted by $\Delta_{t}(i)$ and measured at the beginning of each time slot. If there is a packet arrival at the $i$th time slot, we have $\Delta_{t}(i)=0$. Otherwise, we have

$$\Delta_{t}(i)=i-G(i)$$

(1)

where $G(i)$ is the arrival time of the latest packet observed at the $i$th time slot.

At the receiver side, the data freshness is not only affected by the packet arrival process, but also by the transmission policy and channel quality. We assume that the data freshness at the receiver is measured by the observed AoI at the end of each time slot. Denoted by $\Delta_{r}(i)$ the AoI observed at the $i$th time slot. Following the conventional definition of AoI, we have:

$$\Delta_{r}(i)=i-U(i)+1$$

(2)

where $U(i)$ is the arrival slot of the latest packet successfully received by the end of $i$th slot. For consistence, we initialize $\Delta_{t}(0)=\Delta_{r}(0)=0$ and adopt the average AoI as the measurement of long-term data freshness, i.e.,

$\displaystyle\bar{\Delta}_{r}=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{i=1}^{T}\Delta_{r}(i),$

(3)

where $T$ is the observation interval.

The transmission cost is proportional to the number of transmissions made by the transmitter. We denote the transmitter action at the $i$th time slot by $a_{i}\in\{0,1\}$, where $a_{i}=1$ if the transmitter chooses to send the packet in the buffer and $a_{i}=0$ if the buffer is empty or the transmitter chooses to remain silent even though a packet is waiting in the buffer. The long-term average transmission cost is defined as

$\displaystyle\eta=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{i=1}^{T}a_{i}.$

(4)

This paper aims to investigate the tradeoff between the data freshness $\bar{\Delta}_{r}$ and the transmission cost $\eta$. The conventional pLGFS without resource constraint [5] achieves the minimum AoI with the maximum transmission cost. Hence, we have

$\displaystyle\bar{\Delta}_{r}\geq\frac{1}{\lambda}+\frac{\varepsilon}{1-\varepsilon},\textnormal{ and }\eta\leq\frac{\lambda}{1-(1-\lambda)\varepsilon},$

(5)

where the equality holds when the pLGFS transmission policy is adopted. On the other hand, when the packet is generated-at-will, which is equivalent to the proposed network model with $\lambda=1$, the optimal tradeoff between the data freshness and transmission cost has been presented in [12], which is reproduced as following

	$\displaystyle\bar{\Delta}_{r}(\delta)=\frac{(\delta(1-\varepsilon)+\varepsilon)^{2}+\varepsilon}{2(1-\varepsilon)(\delta(1-\varepsilon)+\varepsilon)}+\frac{1}{2},$		(6)
	$\displaystyle\eta(\delta)=\frac{1}{\delta(1-\varepsilon)+\varepsilon},$		(7)

where $\delta$ is the system design parameter and various tradeoff between $\bar{\Delta}_{r}$ and $\eta$ can be achieved by changing $\delta$. This paper aims to generalize the result of [12] to the network with stochastic arrival.

The problem $\mathbf{P0}$ can be modeled as a CMDP as in [12] and [13]. For the network model considered in this paper, the CMDP consists of the following key elements:

• •

  State: Under the assumption of random packet arrival and erasure channel, the optimal action of the transmitter at a typical time slot is affected by the data freshness observed by the transmitter at the beginning of this time slot and that observed by the receiver at the end of the previous time slot. Hence, the state of the environment can be characterized by $s_{i}=(\Delta_{t}(i),\Delta_{r}(i-1))$.

• •

  Action: At the $i$th time slot, the transmitter can choose to transmit the packet in the buffer or remain silent, corresponding to $a_{i}=1$ and $a_{i}=0$, respectively.

• •

  Reward: Since the objective is to minimize the average AoI observed by the receiver, we define the immediate reward at the $i$th time slot as the negative of the receiver AoI, i.e., $R_{i}=-\Delta_{r}(i)$.

• •

  Cost: Each transmission incurs a unit cost and hence we can define the cost at the $i$th time slot, denoted by $C_{i}$, as the action taken, i.e., $C_{i}=a_{i}$.

The state transition probability is related with the random packet arrival, channel status and action taken, which is illustrated in Fig. 2.

In summary, solving $\mathbf{P0}$ is equivalent to find the optimal policy for the above CMDP. The classical relative value iteration algorithm (RVIA)[15] can be used to find the optimal policy of CMDP by reducing the original infinite state space to finite size as approximation. However, as the optimal policy derived by RVIA has a multiple threshold structure [13], its implementation complexity is high and tractability is limited. In this paper, we derive a lower bound on the AoI under a given resource constraint and propose a low-complexity single threshold policy, which allows us to have a near optimal solution to problem $\mathbf{P0}$ in closed form, and hence investigate the tradeoff between AoI and transmission cost easily.

Suppose there are $N(T)$ packets successfully delivered to the receiver during the observation interval $T$. We denote by $Y_{j}$ the number of slots between the receiving time of the $j$th packet and the $(j+1)$th packet, and let $Y_{0}$ be the number of time slots before receiving the first packet. For notational convenience, we further denote by $Y_{N(T)}$ the remaining number of time slots after receiving the last packet, i.e., $Y_{N(T)}=T-\sum_{j=0}^{N(T)-1}Y_{j}$.

Let $S_{j}$ be the age of the $j$th packet when it is successfully delivered, where $S_{0}=0$. The receiver AoI during the time slots between the delivery of the $j$th and $(j+1)$th packet are $S_{j},S_{j}+1,...,S_{j}+Y_{j}$, respectively. Hence, the long-term average AoI is given by

	$\displaystyle\bar{\Delta}_{r}$	$\displaystyle=\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{j=0}^{N(T)}\frac{(S_{j}+S_{j}+Y_{j}+1)Y_{j}}{2}$
		$\displaystyle=\lim_{T\rightarrow\infty}\frac{1}{T}\left(\sum_{j=0}^{N(T)}S_{j}Y_{j}+\frac{1}{2}\sum_{j=0}^{N(T)}Y_{j}^{2}+\frac{1}{2}\sum_{j=0}^{N(T)}Y_{j}\right)$
		$\displaystyle\overset{(a)}{\geq}\lim_{T\rightarrow\infty}\frac{1}{T}\left(\frac{1}{2}\sum_{j=0}^{N(T)}Y_{j}^{2}+\frac{T}{2}\right)$
		$\displaystyle=\frac{1}{2}\lim_{T\rightarrow\infty}\frac{1}{T}\sum_{j=0}^{N(T)}Y_{j}^{2}+\frac{1}{2}$		(10)

where $(a)$ follows from the fact that $\sum_{j=0}^{N(T)}S_{j}Y_{j}\geq 0$ and $\sum_{j=0}^{N(T)}Y_{j}=T$.

Denote the expectation of $Y_{j}$ by $\mathbb{E}[Y_{j}]$, i.e., $\mathbb{E}[Y_{j}]\triangleq\frac{1}{N(T)+1}\sum_{j=0}^{N(T)}Y_{j}$. By Jensen’s inequality, we have

	$\displaystyle\mathbb{E}[Y_{j}^{2}]$	$\displaystyle=\frac{1}{N(T)+1}\sum_{j=0}^{N(T)}Y_{j}^{2}\geq(\mathbb{E}[Y_{j}])^{2}$
		$\displaystyle=\frac{(\sum_{j=0}^{N(T)}Y_{j})^{2}}{(N(T)+1)^{2}}=\frac{T^{2}}{(N(T)+1)^{2}}.$		(11)

Equation (11) gives us $\sum_{j=0}^{N(T)}Y_{j}^{2}\geq\frac{T^{2}}{N(T)+1}$, which can be substituted back to (10) as

$\displaystyle\bar{\Delta}_{r}$

$\displaystyle\geq\frac{1}{2}\lim_{T\rightarrow\infty}\frac{T}{N(T)+1}+\frac{1}{2}=\frac{1}{2}\lim_{T\rightarrow\infty}\frac{T}{N(T)}+\frac{1}{2}.$

(12)

Note that the term $\lim_{T\rightarrow\infty}\frac{N(T)}{T}$ is the long-term throughput of the network, which is bounded by the packet arrival rate and the capacity of the network. Specifically, we have

	$\displaystyle\lim_{T\rightarrow\infty}\frac{N(T)}{T}\leq\lambda$		(13)
	$\displaystyle\lim_{T\rightarrow\infty}\frac{N(T)}{T}\leq\eta_{\max}(1-\epsilon),$		(14)

where $\eta_{\max}(1-\epsilon)$ in (14) is the effective capacity of the network under the resource constraint. Substituting (13) and (14) into (12) renders the lower bound as

$\displaystyle\bar{\Delta}_{r}\geq\frac{1}{2}\left(\frac{1}{\min\{\lambda,\eta_{\max}(1-\epsilon)\}}+1\right).$

(15)

First, note that the transmitter will naturally choose to remain silent if the buffer at the transmitter is empty, i.e., the latest arrived packet has been successfully delivered, which happens with probability $P_{\mathrm{empty}}$. For all the best-effort transmission policies, we have

$\displaystyle P_{\mathrm{empty}}=\frac{(1-\varepsilon)(1-\lambda)}{1-\varepsilon+\varepsilon\lambda}.$

(16)

If the buffer is not empty, the optimal action at the $i$th time slot is determined based on the state $s_{i}=(\Delta_{t}(i),\Delta_{r}(i-1))$. If the transmitter decides to send a packet, it will receive a reward of $(-\Delta_{t}(i)-1)$ with probability $(1-\varepsilon)$, and a reward of $(-\Delta_{r}(i-1)-1)$ with probability $\varepsilon$. On the other hand, if the transmitter choose to keep silent, it will receive a reward of $(-\Delta_{r}(i-1)-1)$. The expected reward gain by sending the packet is hence given by

$\displaystyle\bar{R}_{i|a_{i}=1}=(1-\varepsilon)(\Delta_{r}(i-1)-\Delta_{t}(i)),$

(17)

which is obtained at the constant cost of $C_{i}=1$. From (17), the expected AoI improvement by sending a packet is proportional to $(\Delta_{r}(i-1)-\Delta_{t}(i))$. Therefore, we propose a low-complexity single threshold policy in Definition 1, which maximizes the expected reward gain under a given cost constraint.

Compared to the double threshold policy proposed in [13], the single threshold policy can be easily analyzed and it allows us to investigate the tradeoff between AoI and transmission cost more conveniently. Furthermore, the transmitter only needs to make the transmission decision at the time slot when new packet arrives. Specifically, assume a new packet arrives at the $i$th time slot, we have $\Delta_{t}(i)=0$. If the transmission is not activated, i.e., $\Delta_{r}(i-1)<\delta$, there will be no transmission activated until the next packet arrival, since both $\Delta_{t}$ and $\Delta_{r}$ grow linearly with time, resulting no change in their difference. On the other hand, if the transmission is activated at the $i$th time slot, i.e., $\Delta_{r}(i-1)\geq\delta$, this packet will be repeated until it is successfully delivered or preempted by the newly arrival packet, since the value $(\Delta_{r}(i-1)-\Delta_{t}(i))$ will not change until the packet is successfully delivered.

Denote by $P_{j}$ the probability that the AoI observed at the end of a typical time slot is $j$, i.e., $P_{j}\triangleq\Pr(\Delta_{r}=j),j=1,2,...$. Further denote by $P_{\Delta}$ the probability that the receiver AoI is greater or equal to $\delta$, i.e.,

$\displaystyle P_{\Delta}=\sum_{j=\delta}^{\infty}P_{j}.$

(19)

Based on the above analysis, the event $\Delta_{r}=j$ occurs in the following two cases.

• •

  Case 1: The previous packet arrival occurs at $j$ time slots before, the transmission is activated when that packet arrives, and that packet has been successfully delivered during the subsequent $j$ time slots. Case 1 occurs with probability

  $\displaystyle\bar{P}_{\mathrm{c1}}=P_{\Delta}(1-\lambda)^{j-1}\lambda(1-\varepsilon^{j})$

  (20)

• •

  Case 2: The previous packet arrival occurs at $m$ time slots before, where $1\leq m\leq j-1$, and the observed AoI is $(j-m)$ when that packet arrives. Furthermore, no transmission is activated when the previous packet arrives or the transmission is activated, but all the $m$ attempts are erased. Case 2 occurs with probability

  	$\displaystyle\bar{P}_{\mathrm{c2}}=$	$\displaystyle\sum_{m=j-\delta+1}^{j-1}P_{j-m}(1-\lambda)^{m-1}\lambda$
  		$\displaystyle+\sum_{m=1}^{j-\delta}P_{j-m}(1-\lambda)^{m-1}\lambda\varepsilon^{m}$		(21)

By combining the above two independent cases, we have

$\displaystyle P_{j}=\bar{P}_{\mathrm{c1}}+\bar{P}_{\mathrm{c2}}.$

(22)

From (19)-(22), we can solve for the distribution of $\Delta_{r}$, as given in Lemma 1. The detailed derivation are omitted for brevity.

Given the distribution of the receiver AoI, we are now ready to derive the long-term average AoI and transmission cost. Specifically, the average AoI is a function of threshold $\delta$, given by

$\displaystyle\bar{\Delta}_{r}(\delta)=\sum_{j=1}^{\infty}jP_{j}.$

(24)

Note that the transmission is activated when the buffer is not empty and the difference of data freshness between the receiver and the transmitter is greater than or equal to the threshold $\delta$. These two independent events occur with probability $(1-P_{\mathrm{empty}})$ and $P_{\Delta}$, respectively. Hence, the long-term average transmission cost is also a function of threshold $\delta$, given by

$\displaystyle\eta(\delta)$

$\displaystyle=(1-P_{\mathrm{empty}})P_{\Delta}=\frac{\lambda}{1-\varepsilon+\lambda\varepsilon}\sum_{j=\delta}^{\infty}P_{j}$

(25)

By substituting $P_{j}$ derived from Lemma 1 into (24) and (25), we can obtain the closed-form expressions of the average AoI and transmission cost, as given in Lemma 2.

It is observed that the average AoI $\bar{\Delta}_{r}(\delta)$ in (26) is a monotonically increasing function with the threshold $\delta$, while the transmission cost $\eta(\delta)$ in (27) is a monotonically decreasing function with $\delta$. To achieve the minimum AoI subject to cost constraint, we can find the maximum value of $\delta$ that meets the cost constraint. Mathematically, we have

$\displaystyle\bar{\Delta}_{r}^{*}=\min_{\delta}\bar{\Delta}_{r}(\delta).$

(28)

The optimal threshold $\delta^{*}$ is solved from

$\displaystyle\delta^{*}=\{\arg\max_{\delta\in\mathbb{N}}\eta(\delta)\leq\eta_{\max}\},$

(29)

where $\mathbb{N}$ is the natural number set. The proposed single threshold policy with the optimal threshold $\delta^{*}$ in (29) is referred as the “deterministic single threshold policy”.

Note that the integer constraint on the threshold may lead to under-use of the resource, i.e., $\eta(\delta^{*})<\eta_{\max}$. In such case, the AoI performance of the proposed transmission policy can be improved by introducing a “random policy” where the transmitter uses the threshold $\delta^{*}$ with probability $q$ and uses the threshold $(\delta^{*}+1)$ with probability $(1-q)$. The probability $q$ is determined by solving

$\displaystyle q\cdot\eta(\delta^{*})+(1-q)\cdot\eta(\delta^{*}+1)=\eta_{\max}$

(30)

The combination of the deterministic and the random single threshold policy mentioned above can achieve arbitrary point on the tradeoff curve between the AoI and transmission cost, and hence provide a complete solution to the problem $\mathbf{P0}$.

Note that if there is no resource constraint, we have $\delta^{*}=0$ and the proposed scheme reduces to the conventional pLGFS with performance given in (5). Furthermore, if we increase the packet arrival rate to $\lambda=1$, the network degenerates to the one with generate-at-will arrival model. The AoI and transmission cost equation in (26) and (27) reduces to (6) and (7), respectively, which are proven to be optimal in [12].