Cooperative Learning-Based Framework
for VNF Caching and Placement Optimization
over Low Earth Orbit Satellite Networks

LSNs create a breeding ground for diverse applications, including surveillance, tracking, mapping, weather forecasting, and disaster response [14], which can be decomposed into various supporting tasks such as remote sensing and Earth observation. An emerging paradigm within this context involves breaking down application tasks and abstracting them into VNFs [6]. This approach aligns with the SFC paradigm, facilitating the deployment and interconnection of VNFs in LSNs, the intricate challenges of which have concerned the research community during the last few years. Gao et al. [15] presented a hierarchical architecture for IoT users in satellite edge clouds, including IoT users, LEO satellites, and a cloud data center. They proposed a distributed VNF placement algorithm to minimize network bandwidth cost and service delay, considering satellite network characteristics like high latency and limited bandwidth. Also providing service to remote IoT users, the authors of [16] aimed at optimizing satellite deployment costs by modeling the problem as a network payoff maximization problem, solved with decentralized allocation through a non-cooperative potential game.

Qin et al. [17] examined a network model where service data is processed by satellites and sent to an Earth station. They formulated the problem as a congestion game to optimize delivery time and maintain continuous service, focusing on resource sharing and competition among SFCs. The authors then proposed two algorithms that achieve Nash Equilibrium to solve this problem. Jia et al. [18] tackled a VNF-based service delivery problem involving a network operation control center, Geostationary Equatorial Orbit (GEO) satellites as SDN controllers, and LEO satellites. They proposed an approximation algorithm focused on efficient resource allocation for practical uses. By employing both types of GEO and LEO satellites, the study in [19] divided the network into a control plane and a data plane. A joint optimization approach for elastic network resource provisioning was then introduced, enabling adaptable VNF deployment and traffic routing. In [20], the authors identified optimal SFC deployment paths that involve satellite deployment probabilities based on the satellite’s position and latitude. The authors of [21] proposed a heuristic and a mixed-integer linear programming algorithm to minimize end-to-end service latency while maximizing the service acceptance rate.

Evidently, placing SFCs on satellite networks is a challenging problem; thus following the recent interdisciplinary trends, Han et al. [2] devised an adaptive online SFC deployment algorithm for large-scale LSNs, based on deep reinforcement learning. Here, the authors proposed subnet division and a proximal policy optimization approach to reduce resource consumption in their solution. The cooperation/competition potential between individual satellites during the SFC placement problem was studied by Qin et al. in [3]; an approach based on potential game theory was adopted and three algorithms were proposed to minimize service delivery latency. On a different note, [4] suggested the use of a Tabu-search heuristic to solve the emerging integer non-linear programming problem of SFC deployment in satellite networks while minimizing VNF migrations.

Despite the recent research interest around SFC deployment on LSNs, the majority of the works do not address the dynamic nature of the network nor optimize the long-term system performance. Additionally, the aforementioned works assume a priori that satellites are capable of executing every type of VNF available in the system. However, in the LSN context, embedded satellite processor capability and storage are typically limited, hence, this is not a practical assumption. In the proposed framework, which extends our prior work in [8], we aim to specifically address these two limitations in the literature, by providing a holistic LSN service deployment framework (VNF placement and caching) that simultaneously optimizes the end-to-end service delay and the request serving rate.

We assume an LSN that consists of multiple satellites, with $\mathbb{V}=\{v_{1},\ldots,v_{V}\}$ being the set of their indices ($V=|\mathbb{V}|$). The satellites gather data (e.g., images of the Earth’s surface) and cooperatively execute computational tasks for the supported applications. The satellites’ movements are considered periodic, and also, we assume that an inter-satellite link (ISL) between a pair of satellites is active when the two satellites enter the communication range of each other. Additionally, each satellite $v\in\mathbb{V}$ is equipped with $R_{v}$ computational and $Z_{v}$ storage resources.

The introduced satellite network operates in a slotted, infinite time horizon, where $t\in\mathbb{N}=\left\{1,2,\ldots\right\}$ denotes a time slot. We use the binary parameter $e_{v,u}(t)=1$ (or $0$) to represent if the ISL between satellites $v$ and $u$, referred to as $(v,u)$-ISL, is active (or not). Due to the recurring nature of their orbits, satellites interact with each other at fixed intervals, and thus ISLs become active periodically as well. To model this periodicity, we introduce the following parameters; we assume that whenever the $(v,u)$-ISL is active, it remains active for $\tau_{v,u}$ time slots before becoming inactive. In addition, $T_{v,u}$ denotes the period of $(v,u)$-ISL (in time slots), implying that $T_{v,u}\geq\tau_{v,u}$ and $e_{v,u}\left(t\right)=e_{v,u}\left(t+T_{v,u}\right)$, $\forall t\in\mathbb{N},\;v,u\in\mathbb{V}$. Furthermore, we denote by $T$ the period of the entire network connectivity which can be computed as the least common multiple of $T_{u,v},\forall u,v\in\mathbb{V}$. For simplicity, we assume that every data transfer between a satellite pair is completed within a single time slot using an inter-satellite free-space optical link with high transmission speed, long communication range, and high reliability. In this model, we only focus on the the sets of parameters $T_{u,v}$ and $\tau_{u,v}$, $\forall v,u$ to describe the network topology and motions of satellites for simplifying the presentation and enhancing the clarify. An illustrative example is provided in Fig. 1 with the presence/absence of the two satellites in a time slot implying an active/inactive ISL.

As mentioned earlier, the satellites cooperatively execute computational tasks for certain types of applications. Each task requires data to be processed through the sequence of interconnected VNFs of an SFC. For instance, a wildfire detection involves several steps such as image processing, feature extraction, and fire classification [14], each one carried out by a separate function represented by a VNF in this work. An ordered sequence of $l_{h}$ VNFs comprises an SFC $h\in\mathbb{H}$, where $\mathbb{H}=\left\{h_{1},\ldots,h_{H}\right\},$ is the index set of all the available SFCs/services, $H=|\mathbb{H}|$. Also, we let $\mathbb{F}=\{\mathcal{F}_{1},\mathcal{F}_{2},\ldots\}$ be the set of all available VNFs. We denote by $F^{h}_{f}\in\mathbb{F}$ the $f^{\text{th}}$ VNF of SFC $h$ and by $\mathbb{S}_{h}=\left\{F^{h}_{f}\;|\;f\in\{1,...,l_{h}\}\right\}$ the set of VNFs comprising SFC $h\in\mathbb{H}$. For a satellite to execute a specific VNF, the corresponding source code needs to be installed in the satellite’s computing system. The storage used for caching is excluded from the $Z_{v}$ storage resources of a satellite. We denote by $\mathbb{F}_{v}$ the pre-installed set of VNF types on satellite $v\in\mathbb{V}$, with $|\mathbb{F}_{v}|$ being its caching capacity. A VNF $F^{h}_{f}$ can only be placed/executed on satellite $v$ if $F^{h}_{f}\in\mathbb{F}_{v}$. Due to their limited physical resources, it is not always feasible to install every available VNFs on the satellites. Therefore, we consider a scenario where a subset of VNF images has been pre-installed on each satellite. This means that $\mathbb{F}_{v}\subseteq\mathbb{F}$, covering the unique case $\mathbb{F}_{v}=\mathbb{F}$ where $v\in\mathbb{V}$.

At the end of time slot $t$, a service request is initiated with a probability $\mu$. A request initiation triggers satellite $v\in\mathbb{V}$ to request the placement of an SFC $h\in\mathbb{H}$. We call this satellite the requester; $\mu^{r}_{v}$ and $\mu^{s}_{h}$ are the probability that satellite $v$ will be the requester and the probability that SFC $h$ will be requested, respectively, conditioned on the occurrence of a service request. A service request for SFC $h$ will expire at the end of time slot $t+D_{h}$ if not successfully served, where $D_{h}$ is the SFC’s end-to-end delay tolerance. In our system model, we assume that at most one SFC request is initiated per time slot. Additionally, since the execution of an SFC may span multiple time slots, a satellite can manage multiple requests within a time slot, including its own and those forwarded from other satellites. In this work, each request is assigned a timestamp (described in Section IV-B) corresponding to its initiation time, and requests are processed in ascending order of these timestamps. In a practical context where a satellite initiates or receives multiple requests with the same initiation time slot, these requests can be buffered to eliminate their temporal overlap. Each buffered request is processed in a time slot, and the proposed framework in this paper remains applicable. Timestamps along with the use of buffers help determine the order in which requests are processed, thereby preventing resource conflicts.

We use the terms placement and execution interchangeably throughout this work. Executing SFC $h$ means that all the VNFs $F^{h}_{f}$ for $f=1,\ldots,l_{h}$ are executed in the given order. That means the output of the $f^{\text{th}}$ VNF becomes the input of the $(f+1)^{\text{th}}$ VNF in the sequence. The output of the last VNF in the sequence bears the results carried out by SFC $h$ and needs to be returned to the requester satellite to mark the request as successfully served. We also denote by $q^{h}_{f}$ and $g^{h}_{f}$ the computational and storage resources required to execute $F^{h}_{f}\in\mathbb{S}_{h}$ and store its output.

We assume that if satellite $v$ executes VNF $F^{h}_{f}$, and VNF $F^{h}_{f+1}$ is assigned to another satellite $u$ following a placement policy (e.g., because $F^{h}_{f+1}$ has not been pre-installed on $v$), $v$ needs to store the output of $F^{h}_{f}$ in its storage before forwarding¹¹1The forwarding path may consist of multiple satellites as relays. it to satellite $u$. In this case, when $F^{h}_{f}$ has been executed, $q^{h}_{f}$ computational resources on satellite $v$ are released, but $g^{h}_{f}$ storage resources are consumed to temporarily store its output. That happens because the communication can occur only when the $(v,u)$-ISL is active. After forwarding the data, $g^{h}_{f}$ storage resources are released on satellite $v$, while $g^{h}_{f}$ storage resources are consumed on satellite $u$ to store the received data. That means that data forwarding might fail if the receiving satellite does not have sufficient available storage space. On the contrary, if a satellite is executing a sequence of VNFs, the intermediate data is not stored in its storage.

In this work, we address the following two sub-problems:

• •

  The first sub-problem is to assign VNFs to satellites and define data routing paths to optimize the overall system performance. We refer to this as the VNF/service placement problem which is formally defined and solved in Section IV.

• •

  The second sub-problem is to define the optimal VNF subset installed on each satellite. We refer to this as the VNF caching problem which is formally defined and solved in Section V.

Fig. 2 illustrates a VNF placement constrained by installed VNF subsets. Table I summarizes the main notation. To reduce complexity, we will gradually augment the notation with additional variables as needed. In addition, we note that service requests and ISL transitions occur in each time slot. The former, along with the available resources of each satellite, constitutes the system state, which will be defined in the next section. The latter is due to satellite motions. Therefore, a time slot can be interpreted as a unit of time that quantifies both components.

We note that when a placement $\mathbf{p}$ is made at time slot $t$, the required resources of all involved satellites in future time slots are reserved accordingly. We define $K=\underset{h\in\mathbb{H}}{\max}~{}D_{h}$ as the largest end-to-end delay tolerance among all supported SFCs. We represent the system’s state by $\mathbf{x}$ which is updated at the beginning of every time slot; $\mathbf{x}$ encapsulates the satellites’ available resources for up to $K$ future time slots, the index of the requested SFC, as well as the index of the requester satellite.²²2It is sufficient for $\mathbf{x}$ to convey $K$ time slots from the current one because no placement can span more than $K$ time slots. The vector of available resources of satellite $v$ for the next $K$ time slots is given by $\mathbf{x}_{v}=(\mathbf{r}_{v},\mathbf{z}_{v})$, with $\mathbf{r}_{v}=\{r_{v,k}\;|\;r_{v,k}\leq R_{v},\forall k\in\{1,...,K\}\},~{}\mathbf{z}_{v}=\{z_{v,k}\;|\;z_{v,k}\leq Z_{v},\forall k\in\{1,...,K\}\}$, where $r_{v,k}$ and $z_{v,k}$ are the available computational and storage resources of satellite $v$, respectively, at time slot $k$. These allow us to formally define the system state as:

where $h$ and $n$ are the indices of the requested SFC and the requester at the end of the previous time slot, respectively. If there is no request in the previous time slot, $h=n=0$.

We define $\tilde{\mathbf{x}}=\{\tilde{\mathbf{x}}_{v},\tilde{h},\tilde{n}\;|\;\forall v\in\mathbb{V},\tilde{h}\in\mathbb{H},n\in\mathbb{V}\}$ as the transitioned system state resulted by placement $\mathbf{p}$ in the beginning of the following time slot; $\tilde{\mathbf{x}}_{v}=(\tilde{\mathbf{r}}_{v},\tilde{\mathbf{z}}_{v})$ represents the transitioned vector of available resources on satellite $v$. We compute the components of $\mathbf{x}_{v}$ as follows:

1. (i)

   if $v=\phi_{m_{f}}\in\boldsymbol{\phi}$ and $m_{f}\in\mathbf{m}$, meaning that satellite $v$ is executing VNF $F^{h}_{f}$, hence, $\phi_{m_{f}+m}=v,\;\forall m\in[0,d^{h}_{f,v}-1]$. Therefore, $\tilde{r}_{v,k}=r_{v,\left(k+1\right)}-q^{h}_{f}$, and $\tilde{z}_{v,k}=z_{v,\left(k+1\right)}$, $\forall k\in[m_{f},m_{f}+d^{h}_{f,v}-1]$.

2. (ii)

   else if $v=\phi_{m}\in\boldsymbol{\phi}$, $m\notin\mathbf{m}$, and $\exists m_{f}\in\mathbf{m}$ such that either $m_{f}<m<m_{f+1}$ or $m>m_{f=l_{h}}$, meaning $v$ stores the output of $F^{h}_{f}$. Therefore, $\tilde{r}_{v,(m-1)}=r_{v,m}$ and $\tilde{z}_{v,(m-1)}=z_{v,m}-g^{h}_{f}$. We note that if $m_{f}<m$, with $m_{f}\in\mathbf{m}$, then $m\geq 2$.

3. (iii)

   else either $v\notin\boldsymbol{\phi}$, i.e., satellite $v$ is not part of the placement $\mathbf{p}$, then, $\tilde{r}_{v,k}=r_{v,\left(k+1\right)},\tilde{z}_{v,k}=z_{v,\left(k+1\right)}$, $\forall k\in[1,K-1]$, or the placement $\mathbf{p}$ spans less than $K$ time slots ($|\boldsymbol{\phi}|<K$), then, $\tilde{r}_{v,k}=r_{v,\left(k+1\right)},\tilde{z}_{v,k}=z_{v,\left(k+1\right)}$, $\forall k\in[|\boldsymbol{\phi}|+1,K-1]$.

Since a valid placement $\mathbf{p}$ cannot span more than $K$ time slots, $\tilde{r}_{v,K}=R_{v}$, and $\tilde{z}_{v,K}=Z_{v}$ always. A representative example is provided below to aid the understanding of the definitions of system states and state transition:

Next, we define the transition probability from $\mathbf{x}$ to $\tilde{\mathbf{x}}$, given a placement $\mathbf{p}$, as:

Additionally, we define the system cost $\mathcal{C}\left(\mathbf{x},\mathbf{p}\right)$ for performing placement $\mathbf{p}$, while on state $\mathbf{x}$, as: $\mathcal{C}(\mathbf{x},\mathbf{p})=|\boldsymbol{\phi}|$ if the request is served, and $\mathcal{C}(\mathbf{x},\mathbf{p})=C_{p}\gg 0$ if rejected.

To accommodate the calculation of the long-term, time-dependent discounted cost, we augment the state and placement variables with time slot notation, $\mathbf{x}(t)$ and $\mathbf{p}(t)$, respectively. Then, this discounted cost can be given by:

where $\gamma\in\left[0,1\right]$ is the discount factor. The objective of the VNF placement task is to minimize the discounted cost (5). Let $J\left(\mathbf{x}\right)$ be the minimum discounted cost which can be calculated through the following DP Equation:

Solving (6) produces the optimal service placement $\mathbf{p}$ that minimizes the end-to-end delay for the requested service.

Finding a solution for the DP Eq. (6) subjected to constraints (7)-(11) can be computationally intractable due to its recursive nature. Furthermore, this equation assumes knowledge of the probabilities associated with service requests, i.e., $\mu$, $\mu_{v}^{r}$, and $\mu_{h}^{s}$, which are typically unobtainable in a realistic environment. To tackle this challenge, we reformulate the VNF placement problem as an MAQL model. In this context, each satellite operates as an independent agent with the goal of learning the optimal VNF placement policy for a requested service. The components of the MAQL model are defined as: