DeepCC: Bridging the Gap Between Congestion Control and Applications via Multi-Objective Optimization

We conduct experiments to evaluate the performance of existing state-of-the-art schemes¹¹1The source codes are provided by Pantheon [17]. in cellular and Wi-Fi links. We plot the distribution of throughput and 95th percentile queueing delay of them in Fig. 1. We then highlight the limitations with illustrative examples.

Limitation 1: Existing schemes can not generalize for diverse application requirements.

Manual policies, such as Cubic[5], allow the senders to deliver data based on heuristics rather than performance requirements. Application-specific optimization schemes, such as Remy and PCC, often do not have general applicability for diverse performance requirements. A fine-tuned scheme for one application works poorly for other applications if performance requirements change.

In principle, the learning-based schemes can support application requirements by designing their objective functions. However, the existing schemes, such as Remy or Vivace, at least in their current form, only provide three choices of objective functions. As a direct consequence, they work in some isolated operating points. As shown in Fig. 1, the family of Vivace can only operate on the high-throughput area (the left-top) and cannot achieve low latency, while that of Remy fails to escape from the scope of low-latency area (the right-bottom). None of them could cover the Pareto front[18]. Although Remy and Vivace can behave on different trade-offs with their three objective-specific functions, the performance range is quite limited so that they can not satisfy diverse requirements.

Even if there are infinite objective functions available that could achieve any trade-offs, it is still non-trivial for the applications to use such a learning-based scheme. The main reason is that none of these schemes provide a systematic approach that could guide the CC to select the suitable objective function to meet a specific quality of service (QoS) requirement (e.g., 8 Mbps throughput while keeping the delay within 70 ms and the loss rate within 1%, the shaded area as shown in Fig. 1(a)). Moreover, the average queueing delay (calculated as the difference between the observed round-trip time, i.e., RTT, and the minimum RTT) and throughput of a single scheme with one objective function can be highly variable under a single network scenario in Fig. 1(a), such as PCC or Indigo. Hence, in the view of the application providers, the existing learning-based schemes may be still far from satisfactory.

Limitation 2: Existing learning-based schemes cannot generalize well across a wide range of network conditions.

Except the application-specific demands, congestion control needs to adapt to a wide variety of heterogeneous network conditions. The existing online or offline schemes can benefit from their own properties but still face great limitations.

Schemes relying on offline learning inherently face the generalization problem. The model or agent (i.e., the learned control rules) can only be trained with limited data that cannot cover all the network conditions and thus may overfit the dataset, at least to some extent. For example, Indigo could attain low latency within its design scope in Fig.1(c). However, its performance can be degraded when the actual network conditions mismatch the training assumptions [19] as shown in Fig.1(b) .

Although the online schemes perform at least acceptably when facing new network condition, the convergence time is a fundamental problem for them. We compare the convergence behavior of the above-mentioned schemes as shown in Fig. 15. It is similar to the results that Vivace claims - PCC converges slowest [16]. In general, the offline schemes are more stable than online schemes[20]. This is because the offline schemes can leverage prior knowledge of networks obtained during training.

As many have observed, learning-based congestion control schemes have emerged in support to adapt to complicated network conditions. Although these schemes deliver a satisfactory performance of a single specific requirement, they still have limitations and fail to achieve different trade-offs of performance requirements. Along this direction, DeepCC is designed to tackle the following key challenges.

(1) Guaranteeing the application-specific demands. It is important to guarantee the specified demands for the applications. However, achieving the specified performance is non-trivial because CC schemes achieve the performance unpredictably under complicated network conditions. Further, learning-based schemes could provide a desired trade-off for one or a class of applications by defining a fine-tuned objective function, but it is difficult to achieve the deterministic performance demands defined by applications or users.

(2) Huge space of optimization objectives. Each metric of application requirements could span on a large scale. The diverse requirements with multiple dimensions that fall into different trade-offs of the performance metrics further make it harder to deal with. The huge space of optimization objectives poses a great challenge, especially for reinforcement learning, which must ”explore” the action space in training to learn a good policy for each optimization objective.

(3) Poor generalization for learning-based schemes. Online learning has the ability to adapt to the environment quickly, whereas the convergence time is too long. By contrast, the learned agent or model through offline learning, which is provided a good starting point by offline learning, can perform well in the scenarios that are similar to the training environments. But it could fail to adapt to unseen network conditions.

Note that it is non-trivial to directly learn to achieve the various performance requirements offline, since the application requirements may fall in a large performance space especially when considering different network conditions. Hence, we use the relative instead of the absolute value to express our objective function in offline learning. Further, the goal as a relative weight of the performance metrics can become a direct “knob” to be tuned by users or applications online to achieve the application desired performance.

The multi-objective expression includes not only multi-dimensional performance metrics, i.e., throughput, delay, and loss rate, but multiple trade-offs between them. The multi-objective function is composed of measurement and goal. Among them, the measurement indicates the current transmission performance that includes the throughput, delay, and loss rate. Table  II summarizes these symbols.

In our approach, the multi-objective function, i.e., reward, is set to reflect the desired performance of throughput, delay, and loss rate that we wish to optimize under different relative weights. So we set the reward $r_{t}$ as the compound of measurement and goal at each time step $t$. For congestion control, we set the performance of throughput as an award, while the performance of delay and loss rate as a penalty. Therefore, when computing a reward, the 2nd-dimension and 3rd-dimension of measurement use the negative value of them.

We consider a DeepCC agent generated by the multi-objective DRL through offline learning, which can potentially deal with the multiple objectives, the continuous decision space, and the adaptation problem by leveraging the great power of deep neural networks (NNs). Unfortunately, the basic Deterministic Policy Gradient (DDPG) algorithm  [21], an advanced DRL algorithm that deals with the continuous action space, could not directly support for DeepCC with multi-objective optimization due to its limited expressiveness of the fixed scalar reward. Recently, some novel multi-objective reinforcement learning algorithms have been proposed, such as DFP [22] for the game Doom and UVFAs [23] for Atari games. However, both DFP and UVFAs cannot directly be applied to continuous control problems.

Hence, we design a multi-objective DDPG algorithm based on basic DDPG architecture. Specifically, we use a multi-objective function as the reward function of DeepCC (provided by §IV-A), expressed as the combination of the goal and performance metric measurement. In the offline training process, we set a random goal for each training episode to achieve a wide range of trade-offs between the performance metrics.

Unlike traditional CC schemes that use hardwired rules to regulate cwnds or sending rate, DeepCC agent learns the flexible policy of sending rate directly from the agent-environment interactions. As shown in Fig. 3, the input of agent includes not only network state but also measurement and goal. Then the agent derives the proper action, i.e., sending rate, when receiving an ACK. Moreover, the multi-objective function of the agent uses the compounded information of measurement and goal which is used to train and improve the neural network model. The ultimate aim of the learning algorithm is to maximize the expected cumulative discounted reward. Leveraging the power of DNNs, the agent can learn near-optimal control policies that mapping state, measurement, and goal to the action for each optimization objective.

Formulating congestion control problem as a DRL task in DeepCC requires specifying the state and action.

The objective of the training is to maximize $J_{mul}$ and minimize the loss $L$ of critic network, which is defined as:

where $y_{t}$ is estimated by:

To achieve the multi-objective training, we randomly set the goal vector $g_{t}$ for each training episode. Therefore, the DeepCC could successfully generalize across a wide range of trade-offs between different performance metrics. When $L$ tends to zero and the average $J_{mul}$ of a batch size data no longer increases, the agent is considered to converge.

As a summary, the actor and critic function are trained by the policy gradient and value-based method respectively. After training, the actor function parameterized by $\theta^{\mu}$ will converge to different near-optimal policies corresponding to different objectives with the help of the critic function. Once being trained, the agent learns near Pareto front with different goals under different network conditions (See §VI-A).

How to adaptively choose the proper policy form learned Pareto optimal policies remains a challenge. To address this issue, we design a policy controller, i.e., an online tuning module to achieve the requirement about throughput, delay, and loss rate, i.e., the target. Our key insight is that we can regard the action regulation as a black box where the input includes state, measurement, and goal driven by the ACK signal. Then the goal performs as a knob that controls the agent to pursue different targets. Specifically, DeepCC continuously detects the changes in the difference between current measured performance and application requirements, and automatically tunes the goal with multi-dimensional gradient descent.

During a TCP session, we adopt the multi-dimensional gradient descent algorithm to tune the agent online. According to the changes in network conditions and the prior measurements, the multi-dimensional gradients about the distance between measurement and target value are computed and applied to generate a new goal for the agent. Thereby the current measurement can be fed to enable a negative-feedback loop to influence the action choice until the performance converges to the application requirements under unseen network conditions.

To reduce the variance of the estimated value $m_{t}$, we set a k-step average of $J$ as the loss function in practice. Taking the relationship between $m_{t}$ and $g_{t}$ into account, the actual loss function is described as:

In this paper, the target value $T=(T^{(1)},T^{(2)},T^{(3)})$ is defined as the desired max-min bound for each dimensional performance metric. The detailed descriptions of $T^{(i)}$ are shown in Table II. In practice, if $m^{(i)}$ is satisfied the bound of $T^{(i)}$, we set $f(g_{t}^{(i)})=T^{(i)}$. Namely, we only update gradients when the target requirements are not satisfied. If the application can not provide the concrete performance target but have a performance preference, e.g., the high throughput or low delay, the target $T$ could be presented as an ($n$-1)-dimensional vector with the default value. In this case, the target $T$ is not related to the uninterested metric. Under the low-delay mode, the target $T$ is set as the minimum RTT and zero packet loss. While in the high-throughput mode, the target $T$ can be defined as the observed maximum bandwidth and zero packet loss.

We use a gradient descent method to update the $g_{t}$. The gradient of $J$ with respect to $g_{t}$ can be derived as:

Since it is non-trivial to model the relationship between the goal $g_{t}$ and the measurement $m_{t}$, we can not easily obtain the analytic expression of the function $f$. Here we use a numerical method to approximate $\bigtriangledown_{g_{t}}f(g_{t})$:

Finally, the goal can be updated by the following formula.

where $\alpha$ is the learning rate.

Since the specific function form of $f(g_{t})$ is unknown and the function is not necessarily a convex function, the traditional gradient descent method may fall into local optimum, resulting in performance degradation. To solve this problem, we design a simple rule-based approach to help “jump out” of the local optimum. In practical terms, if the distance between the $i$th-dimensional measurement and target is larger than a threshold, the $i$th-dimensional goal is increased by a pre-setting value.

Moreover, non-congestion packet loss is a common phenomenon in the Internet [16]. The goal of loss rate (i.e., $g^{(3)}$) should not change due to random packet loss. For DeepCC, we set the constraint for the gradient of the loss rate. That is to say, if the target is not satisfied with the measurement of throughput or delay, the gradient of loss rate will be set to equal that of delay. The above method ensures that the gradient of loss is not affected by non-congestion packet loss since it always follows the gradient of delay.

Since the sum of each dimension of goal vector is 1 in the training process, each dimension of updated goal $g^{(i)}$ will be normalized before fed into the agent as follow:

In fact, it could cause a large variance of goal and huge fluctuations in the convergence process when updating $g_{t}$ directly following e.q. (11). Additionally, it is difficult to choose the proper learning rate $\alpha$. An adaptive learning rate algorithm Adam [24] is a first-order optimization algorithm that is less sensitive to the choice of learning rate than the basic gradient descent algorithm. It also has advantages in non-convex optimization problems. Therefore, we combine Adam algorithm to update the gradient $\bigtriangledown_{g_{t}}f(g_{t})$ in practice.

To validate the effectiveness of multi-objective optimization, we repeatedly run DeepCC’s agent under three network scenarios with different goals in a broad range using Mahimahi[15]. Among them, we select 5 representative results, which are in the mode of high throughput, low delay, or the trade-offs. We also compared DeepCC with the off-the-shelf schemes. As shown in Fig. 5(a) and (b), we intuitively visualize the performance of different schemes in a 2D throughput-delay space. The corresponding loss rate is shown via the color filled.

Specifically, DeepCC with $g_{1}$ in high-throughput mode achieves comparable throughput with the state-of-the-art CCs, while reduces the queueing delay by up to 75% with almost zero packet loss. When running in the low-latency mode, DeepCC with $g_{5}$ performs almost the same as Remy. Other trade-off points stand in the middle and attain different trade-offs. These results show the efficiency of our multi-objective optimization. So DeepCC can achieve a large range of accessible trade-offs only by tuning the goal without any retraining or redesigning efforts and it achieves almost zero packet loss without sacrificing other performance metrics due to the high penalty of loss rate in the reward function.

As shown in the above experiments, DeepCC can reach a wide range of performance by flexibly adjusting the goal. DeepCC can improve the throughput from a low-throughput point to a high-throughput one up to 9X. Likewise, it can reduce the queueing delay from a high-latency point to a low-latency one up to 10X.

Here, we denote the target completion ratio (TCR) as the ratio of target achieved to all targets as our quantitative evaluation metric. Intuitively, for a given congestion control scheme, the higher completion ratio is obtained, the more application requirements can be achieved. Since all the existing learning-based schemes lack tunable objectives, we test each scheme with its default objective under the given network link and calculate its completion ratio. For DeepCC, we test it with different targets and leave the agent unchanged. The results are shown in Fig. 7(a) and Fig. 7(b). As expected, DeepCC achieves the highest target completion ratio among all the schemes. Even in an untrained network scenario, DeepCC still achieves 67.4% completion ratio. The results indicate that DeepCC can tune the objective function to adapt to both network conditions and application requirements through adjusting the goal .

DeepCC agent is trained on a PC with an Intel Core (TM) i7-6850k 3.6 GHz with 8 GB memory. It requires approximately two million training iterations to obtain the near Pareto optimal policies that perform well enough. In total, the training process took about 15 hours. At runtime, it takes about 0.5$\sim$0.9 ms on average to get the action (i.e., the model inference time) or update the goal. We carefully set the decision interval as 1/2 minimum RTT and the online tuning interval as four RTT.

Here, we investigate the impact of different decision intervals on the performance. Intuitively, the decision interval determines the frequency of model inference. To evaluate the impact on average throughput, we set different intervals across different RTTs environments. Fig.9 depicts the results over different decision intervals. There are two takeaways here. First, fixed decision intervals are not advisable. The larger or smaller value of the decision interval can not achieve better performance. Second, the decision interval should change dynamically in different RTT environments. According to the results, we empirically set 1/2 minimum RTT as DeepCC’s decision interval.

Online tuning interval determines the execution frequency of the gradient descent. If the interval is too large, DeepCC cannot adjust the policy in time according to the network condition to satisfy application requirements. On the other hand, if the interval is too small, DeepCC will adjust its policy too frequently thus inccuring unstablizaiton. To evaluate the impact of tuning intervals, we set experiments running the online tuning module with different intervals from half to 8 RTTs. The results in Fig.9 show that the average normalized throughput under 4 RTT is the best than that of other intervals. Therefore, we use 4 RTT as online tuning interval in this paper.

To evaluate DeepCC with competing flows, we set up our testbed to experiment with the coexistence of DeepCC or non-DeepCC flows. We use two hosts as the sender and receiver respectively. They are connected through a router running OpenWrt[28]. We use tc[29] in OpenWrt to regulate the bottleneck link.

As shown in Fig. 11, we report the average throughput achieved by each scheme and Cubic throughout time. The results indicate that BBR, Vivace and Copa are aggressive and get nearly all the bandwidth from the Cubic flow. However, when completing with Remy, Indigo and PCC, Cubic is aggressive, while PCC’s share of bottleneck link’s bandwidth changes from the high to low and does not grow in the presence of Cubic.

In the first few seconds, Cubic quickly grabs the bandwidth. When the queue is full and packet loss occurs due to congestion, Cubic reduces its cwnds. However, at this time, BBR is not sensitive to packet loss and still takes two times of BDP as its cwnds. Therefore, BBR flow can fully leverage the queue while the Cubic flow can not share fairly the bandwidth due to its sensitivity to packets loss. PCC, Vivace and Copa set their sending rate based on their predefined utility function. They require good start point and need time to find the good sending rates when competing with the Cubic flow. In other cases, Remy and Indigo are delay-sensitive congestion control schemes. They focus on the delay and have lower rate while Cubic can achieve high bandwidth. In above cases, the friendliness between these schemes and Cubic is not desirable.

In Fig.11, Cubic flow can share fairly the bandwidth with Cubic flow. However, DeepCC (the target is set with 5 Mbps, 180 ms, 1%) mainly uses the available bandwidth outside of the cubic flow, and achieves a proportional fairness with the cubic flow. When the sending rate of Cubic decreases, the available bandwidth grows and DeepCC increases the sending rate accordingly to obtain higher bandwidth. When the sending rate of Cubic increases, the available bandwidth reduces and DeepCC also reduces the sending rate. Although the obtained bandwidth of DeepCC fluctuates, in a long run it achieves similar average throughput with that of Cubic.

Furthermore, we use DeepCC with different targets to investigate its completing behaviors. Fig. 12 shows that the strong preemption of DeepCC in high-throughput mode benefits from the ability of utilizing the capacity of the bottleneck. It has a higher throughput than that of Cubic. When the target is in the low-latency mode, DeepCC has a weaker preemption of available bandwidth than that of Cubic. When the target is set in the middle, DeepCC achieves a better friendliness on average throughput with competing Cubic flows. As expected, DeepCC with different targets achieves varying proportional share of the bandwidth.

In the following, we evaluate the robustness and convergence of DeepCC. The emulated network link is set as a steady link with 12 Mbps, 60 ms RTT, and 90 KB of buffer.

To evaluate the robustness to packet loss of DeepCC, we set up a steady link with a stochastic loss rate ranging from 0% to 6%. The target for DeepCC is set as 11 Mbps throughput, 100 ms delay, and 5% loss rate. The target loss rate represents the maximum tolerable link loss rate. As shown in Fig. 14, Remy is insensitive to packets loss, whereas it only achieves low throughput. PCC maintains throughput until the packet loss rate reaches 5%, i.e., the predefined loss tolerance included in its objective function. The throughput of Vivace gradually decreases as the stochastic loss rate increases.

DeepCC remains insensitive to random packet loss and obtains consistent high throughput. This is because the goal (more precisely $g^{(3)}$) is not updated by the online tuning algorithm when the observed loss rate is less than that of target. Thus DeepCC ignores the random loss and runs to pursue the other two dimensions of the target (i.e., throughput and delay). If the observed loss rate is higher than that of target and the other two dimensions of measurement are satisfied with the target, DeepCC does not update the gradient according to the constraint of loss rate gradient in the online tuning algorithm.