SafeLight: A Reinforcement Learning Method toward Collision-free Traffic Signal Control

The controller receives a representation of the current intersection state. In literature, different factors are considered in the state space definition, e.g., the number of stopped vehicles, the number of approaching vehicles (i.e., queue length), the average speed of approaching vehicles, and the maximum of the vehicle waiting time. Some papers also use the vehicle positions with the help of advanced sensing capabilities. The state space can be any subset of the above sensing data, among which queue length is the one commonly used in most existing works.

Based on the state, the controller chooses an action (i.e., signaling decision) to take. There are two major types of action space in TSC: cyclic and acyclic action spaces. In the cyclic setting, the control action is defined as a cycle — a complete sequence of signal phases with computed duration. As shown in Figure 1, the cycle has four phases in the order of $(\phi 2,\phi 6),(\phi 1,\phi 5),(\phi 4,\phi 8),(\phi 3,\phi 6)$. The RL agent interacts with the environment cycle by cycle, adaptively adjusting a phase duration in the cycle by either increasing or decreasing its current duration. In the acyclic setting, the control action is defined as a phase $p$. Each time the agent chooses a phase $p$ from all possible phases as its action $a_{i}^{t}$ to take at the current state. In this setting, the execution time of each action, $\Delta t$, is fixed.

After taking an action, the traffic controller receives a reward to indicate how good the signaling decision was. The reward function is one important signal to guide the RL agent toward the well-defined objective, e.g., improving mobility. Vehicle delay, waiting time, intersection throughput, and queue length can also be used in the reward function.

As is shown in Figure 2, left-turn phasing in practice often operates in the following modes: protected-only, where left-turn drivers have the exclusive right of the way, permitted-only, where left-turn drivers may yield to the opposing vehicles, and protective-permitted which is the combination of the aforementioned two. Many studies have shown that permitted-only and protective-permitted operations can reduce delay while having a negative impact on safety. To ensure safety, it is possible to adjust the left-turn phasing mode by integrating domain rules or guidelines. Several left-turn phasing guidelines in the Federal Signal Timing Manual (Koonce and Rodegerdts 2008) could be used to build our safety model, which determines if a left-turn phasing mode is unsafe under certain conditions. For example, based on the guidelines, if the $85\%$ percentile speed (Koonce and Rodegerdts 2008) of opposing traffic is greater than $45$ mph, then the permitted left-turn phasing mode is unsafe. More details of the logical rules used in our design can be found in the Appendix.

Note that there are different safety rules or guidelines on left-turn phasing. In this paper, we refer to the guidelines suggested by the Federal Signal Timing Manual (Koonce and Rodegerdts 2008), and other guidelines could also be similarly integrated into our proposed framework.

A safety module can be integrated into RL by directly influencing the actions. Inspired by the residual RL method (Johannink et al. 2019), where the final action comes from the action learned from an RL agent and the action from a human-designed controller, in our setting, the action from the RL agent is filtered by the safety model as shown in Figure  3(b) with a mark of $1$. Instead of directly learning a safe action, the learned action is corrected whenever a potential safety issue is detected by the safety guidelines. Such supervision is partial (a minimum interference only if the action is unsafe). For example, when the action given by RL includes a permitted left turn and it is determined unsafe, the permitted left turn signal will turn red, prohibiting all left-turn movements.

The safety module can also be integrated into the loss function, which becomes multi-task learning. In multi-task learning, where the performance relies on multiple optimization objectives, the standard approach is to train a model that can minimize a loss function including multiple terms $\mathcal{L}(.,.,\lambda)$ parameterized by a vector $\lambda$, corresponding to different optimizing tasks (Kokkinos 2017; Zamir et al. 2018). Simple average or weighted sum are two common methods to balance different terms: $\mathcal{L}(.,.,\lambda)=\sum_{i}\lambda_{i}\mathcal{L}^{i}(.,.)$. We leverage such knowledge to build upon the existing RL model layering up one more optimizing task, i.e., minimizing collisions, into the loss function as shown in the equation below.

where $Q_{target}=r+\gamma Q(s^{\prime},arg\max_{a^{\prime}}(Q(s^{\prime},a^{\prime};\theta)),\theta^{-})$ is generated by double Q-learning algorithm and $A(s,a;\theta)=Q(s,a)-V(s)$ is the advantage function. The first term of the loss function is from the backbone RL algorithm (3DQN), by which the parameters can be updated by the Mean Square Error (MSE) of the target network and online network. $\Delta$ denotes the second term as illustrated in Figure 3(b) with a mark of $2$, advocating safer actions through minimizing the gap between the actual action and the related safe action. Specifically, the Kullback-Leibler (KL) divergence (Hershey and Olsen 2007) of the normalized advantages over the action space and the desired safe advantage distribution inferred by the safety model $\pi_{H}(s)$ is measured. The normalized advantages reflect the agent’s current preference over the actions and can be used as a measure of how far it is to the desired safe action. Note that when the action from the RL agent is safe (i.e., no collisions occur), the value of the second term in the loss function will be zero.

A safety module can be integrated into the reward function. Similar to the idea of reward shaping (Ng, Harada, and Russell 1999), the domain knowledge is transformed as additional rewards to guide the RL agent to incorporate human expertise. We consider a general form of reward shaping, i.e., the additive form formally defined as $r^{\prime}=r+F$, by adding a safety-related numeric reward value. In our approach, we take the same strategy as how we integrate it into the loss function, shown in Figure 3(b) with a mark of $4$. Once an unsafe action is confirmed by the safety model, a desired safe advantage distribution is formulated. Then the difference between the true advantage distribution over the action space and the desired safe advantage distribution can be measured using the KL divergence and used as an extra numeric reward value denoting the penalty for deviating from desired safe standard, i.e.,

A safety module can be integrated into the state. The safety model can be embedded into an m-dimensional latent space via a layer of Multi-Layer Perceptron, as shown in the middle of Figure 3(b) with a mark of $3$. We define it as $Embed(\pi_{H})=\sigma(\pi_{H}W_{e}+b_{e})$ where $W_{e}$ and $b_{e}$ are weight matrix and bias vector to learn, and $\sigma$ is ReLU function. This embedding is then incorporated into the Q network via concatenation to serve as an additional feature. Such feature-rich input can be combined with SafeLight-Loss or SafeLight-R to further boost the performance.

We perform experiments on a synthetic intersection (Liang et al. 2019) and a real-world intersection (Mei et al. 2022; Ault and Sharon 2021) with various settings. The synthetic intersection in Figure 4(a) has four approaches and twelve one-way vehicular movements. The real-world dataset is built based on the intersection at Cologne, Germany, as shown in Figure 4(b). Real traffic flow is adopted into the simulated environment. The intersection has 8 approaching lanes. In both intersections, the permitted left turn and U-turn are allowed. The setup of SUMO simulation environment assumes: (1) accurate vehicle detection, which is often available in practice via cameras or other sensors; (2) same deceleration rate, which can be modified based on needs and (3) permitted left turn in the timing plan.

The collisions are generated by deliberately tuning up the probability that causes vehicles to ignore conflicting vehicles already in the intersection to mimic the aggressive driving behaviors, which are the root cause of any collisions (Hamdar, Mahmassani, and Chen 2008). Using the generated collision data instead of the real-world collision data is more beneficial since (1) we can obtain a much larger collision dataset so that the sample is not biased; (2) unlike uncomfortable braking and predicted safety risk score, the number of collisions under this stress testing is the most straightforward indicator of safety hazards.

We evaluate the performance on both mobility and safety and answer the above research questions. The average waiting time, throughput, and the number of stop vehicles are chosen as the performance measure for mobility. The number of collisions that happened within one simulation time frame is selected as the safety measure.

Figure 5 summarizes the key performance under three different settings (synthetic dataset and cyclic action space, real-world dataset and cyclic action space, real-world dataset and acyclic action space). We have the following observations:

The convergence curves of RL methods are shown in Figure 6. Compared with Backbone RL(3DQN), SafeLight-Act can quickly converge to lower average waiting times and maintain strong safety performance throughout the training. This is reasonable since this method directly corrects the unsafe learned action. Although the reward is delayed under SafeLight-Act, from the results we can still observe that occasional intervention does not hurt the learning much.

As discussed in earlier sections, there are two types of action spaces in TSC, i.e., cyclic and acyclic actions. Here, we investigate whether our design could generalize to different action settings.

In general, the acyclic action space increases collisions, which is one of the reasons why most existing real-world traffic signals operate in a cyclic manner. Among all the methods, SafeLight-Loss and SafeLight-Act perform relatively well under both the cyclic and acyclic action settings. In comparison, the Syn-R can perform well in the cyclic setting but fail to minimize collisions in the acyclic setting. The performance gap in different settings attributes to the ad-hoc tuning of two reward factors, which verifies that the performance is sensitive to individual settings.

To verify that the proposed SafeLight can maintain superior performance under different backbone RL models. We adopt another RL model, IPPO, from the benchmark (Ault and Sharon 2021) to evaluate the performance in the same synthetic environment as the previous experiment. The state space is defined as the queue length plus the max waiting time per lane. The action is defined as acyclic. The reward function is defined as the same as the state space. Table 1 shows the evaluation results. We can see that SafeLight can still outperform other baseline methods in terms of minimizing collisions.