TCP D*: A Low Latency First Congestion Control Algorithm

TCP congestion control has been a long studied problem in networking [1, 2, 3]. The first step in understanding this complex issue is to define and quantify what congestion is. Congestion can generally be understood as the degradation in network performance as packets are transmitted faster than the rate at which parts of the network (the bottlenecks) can process them. We quantify the congestion by considering the outstanding queues at these bottlenecks and we can equate the level of congestion to the number of packets in the queue. As congestion increases each packet must wait longer in the queue, increasing latency. Additionally, if the number of packets exceeds the maximum queue length, then they are dropped.

Obviously, dropped packets are bad because they need to be re-transmitted for TCP to achieve reliable delivery. Congestion also degrades the goodput of the network and increase latency, which poses other major issues. The first major issue is user experience. Many modern network applications such as AR/VR, online gaming, video conferencing, and remote systems administration are sensitive to latency change on the order of a few 10s of milliseconds [4, 5]. If latency degrades far enough, even web browsing is impacted. Empirical study shows that the QoE of web browsing can quickly deteriorate with delay greater than 2 seconds [6]. The second issue occurs when multiple flows from heterogeneous sources are present. Specifically if some sources are application limited, while others are not, then increased latency from congestion can cause the smaller flows to be starved of throughput. For example, if there is one flow that is downloading a large file (such as an hour long movie), then without limiting throughput to prevent congestion, smaller flows (like web page requests) can take much longer to download [7].

Now we understand why we want to reduce congestion, and preferably we will have no packets being queued. Additionally, it is important to ensure that the network links are fully utilized. Thus we can state the goal of congestion control as such, maximize throughput while minimizing latency. Ideally we would send data as fast as the bottleneck can process it, while at the same time maintaining an empty queue. This is known as Kleinrock’s optimal operating point [8]. This optimal operating point is achieved by maximizing power, which is the ratio of the throughput to the delay, thus achieving maximum throughput with the minimum delay.

The oldest and yet the most prevalent methods of congestion control use losses as the primary feedback of congestion [1, 2, 9]. This is far from ideal, as losses only occur when bottleneck queues are full and congestion is maximized. Things get even worse with deep buffers at the bottleneck links. In the worst case this can lead to large latencies, on the order of seconds in poorly configured networks. This effect has been termed bufferbloat [7]. One way to mitigate this is through active queue management (AQM) [10, 11, 12]. However, adoption of AQM on commodity devices has been slow.

A better alternative to loss based congestion control is delay based congestion control [13, 14, 15]. Delay based algorithms use the packet’s round-trip times (RTTs), to estimate latency. Since there is a direct relationship between congestion and RTT, delay based algorithms can operate much closer to Kleinrock’s optimal operating point. There are also congestion control algorithms that are hybrids in that they use both delay and loss as feedbacks. TCP Illinios [16] and TCP Compound [17] are two examples.

The next consideration in congestion control is how to control the rate at which packets are sent and how many packets are in queues. The most straightforward way to do this it to set a pacing rate, which is the rate at which a client transmits data. A pacing rate determined by the congestion control algorithm can be maintained using a qDisc such as FQ/pacing and HTB [18] or a scalable traffic shaping function implemented on a programmable NIC such as Carousel [19].

The other approach to control the rate at which packets are sent is to use a congestion window, which sets the maximum number of unacknowledged packets that a client may send. This is how most of the congestion algorithms such as TCP Reno [2], CUBIC [9], and HTCP [20] operate. As the tradeoffs between the methods are very important, we discuss them below.

When dealing with clients that send packets in bursts, a congestion window may let packets through all at once, overwhelming bottlenecks and leading to temporary congestion. By pacing the packets instead we can smooth out the sending of packets to a rate that bottlenecks can tolerate.

One advantage of the congestion window is that it has a linear relationship to the length of any outstanding queues, and thus to the degree of congestion. Briefly, the physical network links will be transmitting a number of packets equal to the network’s bandwidth-delay product (BDP). If the congestion window is greater than the BDP, than the excess packets must be stored in queues. The fact that the number of packets in queues is the difference of the BDP and congestion window is a core principal of many congestion control algorithms [1, 2, 9]. Conversely, the pacing rate does not have a linear relationship with how many packets are in queues. In fact, when operating close to the bottleneck rate, small changes can lead to large fluctuations in the number of queued packets [21, 1]. This means pacing rate based algorithms must react quickly and precisely to changes in the network to maintain optimal operation.

In this paper we discuss a newly developed TCP congestion control algorithm called TCP D*. It adopts BBR’s approach of maximizing delivery rate while minimizing latency [14]. However, TCP D* uses the congestion window as its primary control. We posit that this can result in a lower RTT, although potentially at the expense of throughput. With this paper, we hope to open up a discussion about the trade-offs of using a congestion window versus pacing packets. We found that generally the congestion window approach reduces latency potentially at the expense of some loss in throughput. On the other hand, a pacing rate approach maximizes throughput, potentially at the expense of increased latency.

TCP D* is built upon earlier work in RTT based congestion control algorithms. One of the first, and simplest, such algorithms was TCP Vegas [13]. TCP Vegas extended TCP Reno [2] in order to react to RTT congestion signals. In essence, TCP Vegas used the difference between the base RTT and current RTT to estimate the number of packets at the bottleneck, and attempts to keep it in the range of 2-4 packets. This resulted in a lower RTT and better throughput than TCP Reno under good conditions. However, TCP Vegas performed poorly under changing network conditions, due to maintaining the same base RTT estimate for the entire lifetime of the flow. Similar algorithms that use variations of this technique include FAST TCP [22] and Compound TCP [17].

TIMELY [15] is a delay based algorithm that diverges significantly from TCP Vegas. TIMELY uses the delay gradient, or the derivative of the RTT, to control the pacing rate. In particular, it increases the rate on negative gradients, and decreases the rate on positive gradients. Delay gradient computation requires very accurate RTT values, which can only be realistically obtained using hardware support in the NIC. This makes TIMELY more suitable to data center use.

TCP LoLa [23] is another delay based algorithm that uses a congestion window. TCP LoLa’s goal is to set the RTT to a fixed amount over the base RTT, which hopefully allows it to fully utilize throughput while keeping latency at tolerable levels. To do this it cycles between a cubic gain cycle, a fair flow balancing cycle, and a holding cycle.

TCP D* is implemented as a Linux kernel module. It consists of both a congestion avoidance and a slow start algorithm. Before discussing these algorithms in detail, it is important to understand the basic principles behind TCP D*.

The TCP D* Linux module has been tested under a wide variety of network conditions, both physical and simulated. These conditions include dedicated wide area networks (WANs), poorly tuned networks that are easily congested, and networks with different RTT flows. Except for cases where simulation was used, we also tested BBR [14], CUBIC [9], TCP Reno [1], and TCP Vegas [13]. The tests where conducted using Flent [25, 26], which is a wrapper around the Netperf and ping testing utilities.

In this paper we have presented a new congestion control algorithm, called TCP D*, which is based off of concepts from BBR, but is driven by a congestion window first design. Our hope is to drive discussion about the advantages and disadvantages of using a congestion window versus a pacing rate for congestion control. Several questions remain to be investigated about the tradeoffs between these approaches. How do the two approaches perform in networks that have flows sending/receiving data at different rates? This is important as movie streaming and web page requests often occur on the same network and have very different speed requirements. How do these approaches affect performance in terms of shallow versus deep buffers? This would be relevant for data center networks.

There are also improvements for TCP D* and inquiries into its operation we wish to pursue in the future. The first is to improve D*’s performance with low variance dedicated WANs. We have already begun working on this, and have some promising solutions. We will also look into how TCP D* performs under different network conditions, specifically low-latency (sub-millisecond RTT) data center networks and networks with packet policing or shaping. Another improvement to consider is fairness when flows have different base RTTs. A possible course of investigation is whether this can be done with a knowledge of how many flows converge on a bottleneck, perhaps enabled by in-network telemetry.

Our goal with this paper is to open up new avenues of research on the problem of congestion control in new networking domains and for next generation distributed applications. We hope that others working on congestion control will consider the lessons learned and apply them to their own algorithms.