DQN Control Solution for KDD Cup 2021 City Brain Challenge

Figure 2 shows the distribution of road lengths in this competition, which ranges from 37 meters to more than 4,000 meters. Meanwhile, an action of traffic signal phase to be selected will last for 10 seconds in the default setting. As a result, it may be ineffective to take all vehicles in the corresponding lanes of the intersection into consideration.

Our first work is to define the zone of influence. Specifically, for each intersection $i$, we define the segment of each lane where the distance to its center is less than $k$ meters as the zone of influence and denote it as $A_{i}^{k}$. Meanwhile, for each lane, the lane length is set as the distance capping of $k$, which can be expressed as:

The design and calculation of the states and rewards in our solution framework hereinafter are based on this definition.

In Table 1, we summarize some important notations used in our framework. For a vehicle $v$ extracted from the objective “info” in simulation environment , we define the delay of the vehicle as 1 minus the speed $v$ dividing the speed limit of the current lane $speedLimit(v)$

Meanwhile, we define the queue status $q(v)$ to indicate whether a vehicle has a speed smaller than $0.3$ m/s. $J$ is the index set of the lanes of an intersection, ranging from 0 to 23. The index of a nonexistent lane is marked as $-1$. We further define $x^{t}_{j}(A_{i}^{k})$,$d^{t}_{j}(A_{i}^{k})$ ,$q^{t}_{j}(A_{i}^{k})$ as the number of vehicles, the delay index, and the queue length within the zone of influence $A_{i}^{k}$ on lane $j$ at step $t$, respectively. More specifically, the delay index of a lane is the average delay of all vehicles being considered. For a given intersection $i$ and distance $k$, one can easily obtain the statistics for each lane based on the information of vehicles within the zone of influence at time step $t$.

In addition, the definition of the actions are clearly described in the official document: at step $t$, for each intersection $i$, the agent chooses a phase $p_{it}$, indicating that the traffic signal should be set to phase $p_{it}$, which is an element in $\mathcal{S}=\{1,2,3,4,5,6,7,8\}$.

For each intersection, the state includes the current phase $p$, the current time step, the time duration of current phase and the statistic features of vehicle number, delay index, queue length and pressure for each phase based on the definition of zone of the influence.

Table 2 shows the statistic features we use in the state design. To calculate the aggregate feature of each traffic signal phase, we extract the feature for the corresponding lanes given a specific value of $k$ and add them together.

For instance, for the pair of feature “vehicle number” and $k=60$ of signal phase 1, we summarize the total vehicle number in the segments of of lane $0$ and $6$ within zone of influence with $k=60$, and it can be expressed as:

The same state for other phases can be obtained likewise. As the number of total signal phases is 8, correspondingly, the number of feature dimension is 8.

Further, for the pair of feature “pressure of vehicle number” and $k=60$ of signal phase 1, in addition to upstream vehicle counts, we summarize the total downstream vehicle numbers on lane list $[15,16,17,21,22,23]$ within zone of influence with $k=60$ and divide it by 3. To calculate the pressure, we use the upstream vehicle number minus the downstream vehicle number divided by 3, which can be express as:

Various reward designs have been proposed in the literature. Our first attempt is to choose the rewards of queue length and delay (Wei et al., 2019b). Specifically, given an intersection $i\in I$ at time $t$ and its’ zone of influence $A_{i}^{k}$, the rewards of queue length or delay index at time $t$ is the sum of all queue length or delay index in the next timestap $t+1$.

A combination of delay index and queue length (we denote it as “DQ”) can empirically achieve better results:

Another reward recommended by the official competition organizers is max pressure (MP) (Wei et al., 2019a). For an intersection $i$ of time $t$, the MP reward is defined as the sum of the absolute pressures over all traffic movements:

However, the performance of the original reward function is relatively poor. In our implementation, combining the MP reward with the DQ reward can achieve better results:

Further, the reward function that achieves the best single-model result is a new one we proposed based on diagnosis analysis of our experiment results and we name it as “Twin-DQ”:

The aim of the first term of the equation is to minimize the delay and queue length of the upstream lanes, while the purpose of the second term is to minimize the difference of DQ between timesteps $t$ and $t+1$. The basic idea of “Twin-DQ” is to consider the rewards of upstream lanes and downstream lanes separately. An action of the traffic signal can affect the queue length and delay in the upstream lanes directly. However, the influence for the downstream lanes is more complicated. Thus, the reward “Twin-DQ” tries to maintain the congestion level of the downstream lanes, which may facilitate the coordination between upstream and downstream intersections. In our experiments, choose $k$ equal to$100$, achieve best results in the local evaluation dataset.

By default, the decision making process by the agent for a given intersection is triggered if the green time exceeds 30 (or 20) seconds. A major reason for this setting is that switching to another phase would result in a 5-second “all red” period, during which all vehicles may not pass this intersection. Thus, a default setting of a relatively long green time (e.g., 20 or 30 seconds) can avoid such a situation and guarantee higher returns in the long run. However, it is observed that there are situations in which the current phase goes on with no vehicles in upstream lanes. A more sophisticated approach is to consider not only the green time of the current phase, but also alternative conditions that may trigger the agent to make better decisions. As a result, the suite of control schemes we use to trigger the DQN-agent to do the phase selection action is as follows:

Condition 1:

: the green time of the current signal phase exceeds 30 seconds.

Condition 2:

: the queue length of the current signal phase in the upstream zone of influence with $k=60$ equals zero.

Condition 3:

: the queue length of the current signal phase in the downstream zone of influence with $k=60$ equals 8.

Condition 4:

: the queue pressure of the current signal phase in the zone of influence with $k=60$ is less than -5.

During the training process of the DQNs, if any of conditions above is met, the DQN-agent is triggered to begin the phase selection process.

Our training framework follows the classical paradigm of Double Q-Network (double DQN) (Van Hasselt et al., 2016; Mnih et al., 2015) which includes an online network and a target network. The output dimensions of both networks are set to 8, which correspond to 8 candidate signal phases to be selected. Meanwhile, We set the the parameter $\gamma$ to 0.8, which aims at maximizing the total rewards of the next 5 rounds. The $\epsilon$-greedy policy is adopted to train the model. A detailed description of the training parameters are presented in Table 3.

To stabilize the training process of the DQN framework, we design a two-objective neural network. Specifically, one objective is used for predicting the Q-values, and the other one is used for predicting the rewards. The loss function consists of two terms, namely, the smooth-L1 loss of the predicted Q-values and the smooth-L1 loss of the predicted rewards.

As the reward signals are straightforward, it is observed that this approach can greatly accelerate and stabilize the convergence of the training process in our experiments.

In this subsection, we describe a rule-based agent design in addition to the DQN agent. The core of the rule-based agent is the calculation of vehicle density near the center of the intersections. In order to alleviate traffic congestion in multiple directions, phases with high vehicle density on both its upstream lanes are given higher priority. For each upstream lane, the density is calculated as

where $k_{\rm{up}}$ is a global value for the zone of influence in the upstream lanes. The density of downstream lanes is calculated as

where $l_{j}$ is the length of the downstream lane with index $j$, such that the range of $l_{j}-k_{\rm{up}}$ is complement to the upstream zone of influence. The order of priority for each upstream lane, $o_{j_{1}}$, is decided by the relative density $\alpha_{j_{1}}=\alpha^{\rm{up}}_{j_{1}}-\mu\alpha^{\rm{down}}_{j_{2}}$, where $j_{2}$ is the index of the corresponding downstream lane for the upstream lane $j_{1}$ and $\mu$ is a coefficient to be tuned. Specifically, the values of order ranges from 1 to 12 (1 for the highest priority). The difficulty of the designing of the rules mainly comes from the fact that each phase generally correspond to two upstream-downstream lane pairs. An empirical observation is that a phase with two high-priority lane pairs is more favorable than a phase with one top-priority lane and a low-priority lane. Although designing a perfect chain of rules is unlikely, carefully tuning the requirement for both lanes of a phase to be balanced is crucial.

More specifically, the rule has 4 layers of logic. If none of the phases satisfy the condition of a layer, the logic flow moves on to the next layer. For a given phase being considered, we link the dominant upstream-downstream lane pair with index $j_{\mathbf{I}}$ (same as its upstream lane index), while the other lane pair is linked with index $j_{\mathbf{II}}$. The layers of logic are given as follows:

• •

  Layer 1: the first layer aims at picking a phase that corresponds to a lane blocked for too long. The number of blocked round for each lane is calculated and ordered. At the first round, a lane pair $j_{\mathbf{I}}$ is selected if its upstream lane is blocked for more than $C_{\rm{block}}$ seconds and its blocked time is the largest. A phase will be selected if $o_{j_{\mathbf{I}}}<=5$ and $o_{j_{\mathbf{II}}}<=4$. If more than one phase satisfies the conditions, the phase serving two roads is picked. At the second round, a lane pair $j_{\mathbf{I}}$ is chosen if its upstream lane is blocked for more than $C_{\rm{block}}+50$ seconds and its blocked time is the second largest. A phase will be selected if $o_{j_{\mathbf{I}}}<=2$ and $o_{j_{\mathbf{II}}}=1$. The value of $C_{\rm{block}}$ starts with 200 and increases over time until it reaches 300.

• •

  Layer 2: the second layer aims at picking a phase that corresponds to two high-density lanes. At this layer, the vehicle densities for both lane pairs of each phase are checked and a phase will be selected if it has two balanced, high-relative-density lane pairs. Given the balance coefficient $C_{\rm{balance}}$, a lane pair is balanced if $\alpha_{j_{\mathbf{I}}}<\alpha_{j_{\mathbf{II}}}/C_{\rm{balance}}$ (suppose $\alpha_{j_{\mathbf{I}}}>=\alpha_{j_{\mathbf{II}}}$). The selection of the second layer has a total of 5 rounds. For the first 4 rounds, the lane pair with $o_{j_{\mathbf{I}}}=1$ is picked in advance, and the lane pair $j_{\mathbf{II}}$ can be selected if the two lane pairs are balanced and $o_{j_{\mathbf{II}}}$ is less than or equal to 2, 3, 4, or 5, respectively. For the 5th round, a phase is selected if its two lane pairs satisfy $o_{j_{\mathbf{I}}}<=2$ and $o_{j_{\mathbf{II}}}<=3$ and the two pairs are balanced. The values of $C_{\rm{balance}}$ for the 5 rounds start with 0.15, 0.2, 0.2, 0.2, and 0.25 and they decrease over time until they reach 0.13, 0.18, 0.18, 0.18, and 0.23, respectively.

• •

  Layer 3: the third layer aims at picking a phase that corresponds to two low-speed lanes. Here, the threshold for low speed, $C_{\rm{speed}}$, is set to 1 m/s. The average speeds of the upstream lanes are calculated as $s_{j}$, and the speeds are sorted in ascending order (denoted as $o^{\rm{s}}_{j}$). The prerequisite for a phase to be selected is that the average speeds at the two upstream lanes corresponding to the phase are lower than $C_{\rm{speed}}$. In addition, at the first round, a phase is selected if $o^{\rm{s}}_{j_{\mathbf{I}}}<=2$, $o^{\rm{s}}_{j_{\mathbf{II}}}<=2$, $o_{j_{\mathbf{I}}}<=4$, and $o_{j_{\mathbf{II}}}<=4$. Further, at the second round, a phase will be selected if $o^{\rm{s}}_{j_{\mathbf{I}}}<=3$, $o^{\rm{s}}_{j_{\mathbf{II}}}<=3$, $o_{j_{\mathbf{I}}}<=5$, and $o_{j_{\mathbf{II}}}<=5$.

• •

  Layer 4: the final layer picks the phase corresponding to the lane with the highest density if none of the previous conditions can be satisfied. At the same time, a phase that serves two roads is given higher priority (unless one of the roads does not exist). A second round of phase selection is added if the lane with the highest density is a right-turning lane.