Learning to Optimize Industry-Scale
Dynamic Pickup and Delivery Problems

Due to multiple factors such as the market cycle, weather, rhythm of production, etc., there is a strong pattern in the spatial-temporal distribution of delivery demand. We first attempt to reveal the pattern. The delivery order data of four days are selected from the same month. Each day is split into $144$ equal-duration intervals (i.e., each interval is with a duration of $10$ minutes). For each factory and each time interval, we sum up the amount of cargoes in delivery orders associated with the factory and time interval, to get the accumulated delivery demand. In this way, we could construct a spatial-temporal distribution of delivery demand, which is visualized in Fig. 2. In this figure, the vertical axis represents $27$ factories in our manufacturing campus and horizontal axis denotes the $144$ time intervals. Firstly we find that the patterns of four day are similar to some extent; the closer the day, the more similar their patterns. Secondly, from the spatial perspective, it is obvious that some factories has much higher delivery demand than other factories on all the four days. Moreover, from the temporal perspective, for a specific factory, the delivery demand concentrates on several time intervals, such as 10 a.m. to 12 a.m. and 2 p.m. to 5 p.m. Based on above observations, it is believed that we can predict the Spatial-Temporal Distribution (STD) of delivery demand using historical data, to help solve DPDP. Formally, we give the definition of STD matrix of delivery demand and then describe how to predict it.

The STD matrix for the future day $d$ could be predicted if we have historical delivery order data of past consecutive $k$ days. First we construct a series of STD matrices $\bf{E}^{d-1},\bf{E}^{d-2},...,\bf{E}^{d-k}$ for past consecutive $k$ days according to Definition 1. Then the STD matrix $\bf{E}^{d}$ of day $d$ could be predicted in an element-wise way. Specifically, each element $e^{d}_{i,j}$ could be predicted by:

where $\{e^{d^{\prime}}_{i,j}|d^{\prime}=d-1,d-2,...,d-k\}$ is a time series drawn from historical STD matrices and $G$ is an aggregate function applied on the time series to get prediction. Here for efficiency of inference, we just take the average function as $G$. However, to get more accurate prediction, advanced spatial-temporal prediction methods [23, 24] could be directly applied on these STD matrices.

Practices in Didi Chuxing [25, 26] suggest that ‘hitchhiking’ could effectively optimize the logistics cost due to the reduction of travel length. Thus in DPDP, the delivery capacities are supposed to be consistent with delivery demands from both the spatial and temporal perspectives. In this way, vehicles could cover more delivery demands along planned route with less marginal logistics cost. To this end, Spatial-Temporal Score (ST Score) is proposed. For vehicle $k$ and its route $rp_{k}$, ST Score is utilized to measure the difference between delivery capacities and predicted delivery demands along the route $rp_{k}$. The following definitions are given to formulate the calculating procedure.

Various divergence metrics (such as KL and JS divergence) could be applied to measure the difference of two vectors in Eq.(5). The reasons why the JS divergence is adopted here are 1) that the JS divergence is naturally symmetric in calculation and 2) that the experimental results show that JS divergence performs slightly better than symmetric KL divergence in our solution, given in Section VI. Intuitively, the smaller the ST Score, the higher the probability of ‘hitchhiking’ occurring. The whole calculating procedure is depicted in Fig. 3.

As described in Section IV-A, we utilize the ST Score to guide the policy learning, making it possible to dispatch the fleet to the ‘hot spot’ of demand. Besides, graph convolution is often adopted to learn the relation between entities in the environment [14, 20]. Motivated by the two perspectives, we propose Spatial-Temporal Aided Double Deep Graph Network (ST-DDGN). The dynamic dispatching environment is regarded as a graph and each vehicle in the environment is abstracted as a node in the graph. As shown in Fig. 4, ST-DDGN consists of three main components: route planner, multi-layer perceptron (MLP), and neighborhood attention. Roughly speaking, for a given delivery order $o_{t}^{i}$, ST-DDGN first calculates a Q-value for each vehicle $k\in K$ taking the order, which evaluates how good assigning order $o_{t}^{i}$ to vehicle $k$ is. And then the order is finally assigned to the vehicle with the highest Q-value. More specifically, the route planner is responsible for constructing the state $s_{t,k}^{i}$ for vehicle $k$ and order $o_{t}^{i}$. Then the state $s_{t,k}^{i}$ is sent to the initial MLP to get an initial representation. In order to obtain the high-level representation of interaction between vehicle $k$ and other vehicles, a followed neighborhood attention module integrates the initial representations within a predefined range, namely neighborhood. In other words, the neighborhood attention module functions as a communicator among the fleet of vehicles. Note that the neighborhood attention modules could be stacked. At last the initial and high-level representations are concatenated and passed on to the final MLP which gives out the Q-value. Note that the policy network in the proposed framework is not limited on DDQN. Other advanced policy gradient methods, such as A3C [27] and DDPG [28], could also be incorporated into our framework. In the following, the characteristics of DDGN are introduced at length.

Constraint embedding. To accommodate prior constraints, contextual DQN [29] shrinks the output action space with contextual masks based on prior rules. It wastes computational resources if ST-DDGN adopts such a way to filter feasible vehicles. For a given order, not every vehicle is feasible to take the order since some of them violate one or more constraints. Hence it is not necessary to make inference for each vehicle. Besides, due to the architecture of ST-DDGN which could be seen as a factorization of centralized decision maker, the inference time increases as the total number of vehicles involved becomes large. Thus we could exclude infeasible vehicles at early stage of inference. The route planner module takes this job. For a given delivery order $o_{t}^{i}$ and a vehicle $k$, the route planner is responsible for 1) checking whether the vehicle $k$ is feasible to take order $o_{t}^{i}$; 2) planning a shortest route for vehicle $k$ taking order $o_{t}^{i}$ if it is feasible; 3) calculating entries of state $s_{t,k}^{i}$ defined in Section IV-B if it is feasible. For more details about the route planner, please see the pseudo code in Algorithm 2.

The feasible and infeasible vehicles can be distinguished using the route planner. For those infeasible vehicles, they are not involved in the following inference, where the weight connections related to the state are cancelled. Besides, the cancellation would lead to no error generated in back propagation, which means that the infeasible vehicle makes no contribution to the final learned policy. Note that we directly set the Q-value of infeasible vehicles to extremely small negative, ensuring that these vehicles have no chance to be chosen.

Neighborhood attention. As shown in Fig. 5, each attention block corresponds to one vehicle (e.g., vehicle $k$). The block at level $l+1$ ³³3Here the level is specially defined for neighborhood attention and starts from $0$. But level $0$ refers to initial representation obtained from the first MLP. receives lower level representations (i.e., $h_{1}^{l},...,h_{k}^{l},...,h_{K}^{l}$) from both vehicle $k$ and other vehicles. These lower level representations together comprise a feature matrix. In additions, an adjacent matrix with respect to vehicle $k$ is also taken as input into the block. More specifically, each row of the adjacent matrix is an one-hot vector. The length of the vector is $K$ and the position of $1$ in the vector refers to a neighbored vehicle of vehicle $k$. Here we adopt the Euclidean distance to measure the spatial proximity between vehicles and select $N\!E$ nearest vehicles with respect to vehicle $k$ as neighbors. Thus the adjacent matrix is of $\{0,1\}^{N\!E\times K}$. A relational feature is the product of the feature matrix and adjacent matrix. Then the relational feature is sent to multi-head scaled dot-product attention [30], followed by a Dense [31] layer. In this way, a higher level representation (i.e., $h_{k}^{l+1}$) could be obtained for vehicle $k$ and its neighbors. The corresponding hyperparameters of the neighborhood attention block are given in Section VI.

The neighborhood block could be stacked to multiple levels. The connection weights are learned in an end-to-end way and shared by all attention blocks at the same level, equal to a spatial convolution over the graph. With the stacking of multiple attention blocks, one attention block at higher level can integrate more information through the graph convolution with larger receptive field. We stack two levels of blocks in this work owing to the trade-off between accuracy and efficiency.

Simulator is of crucial importance in our dispatching system, responsible for the simulation of interaction between agent and environment. The historical data is fed into the simulator to simulate how our dispatching system runs in real world. Algorithm 1 presents the details how the simulator runs in one episode (i.e., one 24-h day).

At the initial phase, the delivery orders of one episode are sampled uniformly from a historical data pool. Then these orders are sorted in the ascending order of creation time $t_{c}^{i}$ and partitioned into $T$ time intervals of the same duration. It should be noted that the time interval is just used to assign the time feature to each delivery order instead of buffering orders. Vehicles starting at various depots are initialized following the actual spatial distribution. Within each time interval $t$, the information, such as the route, current location and residual capacity, of all vehicles is updated first. For each unassigned order within the time interval $t$, the most ‘appropriate’ vehicle is selected by the learned policy to serve the order. After that, the information of the selected vehicle is further updated due to serving the order, including the route and route length.

The MIP is used to get an optimal solution of DPDP by assuming an ideal case where all orders information within a day can be known beforehand. Due to extremely high computation complexity of MIP, only four tiny-scale testing instances are studied, where $5$ vehicles serves $6,7,8$ and $10$ orders respectively. The comparison between DRL methods and optimal solution is presented in Table I.

As seen in Table I, the ST-DDGN and DGN achieve the best solution quality among four DRL methods in all testing instances. When serving $6$ orders, the two graph-based DRL methods converge to a solution close to the optimum. In terms of the wall time, even in the smallest scale of instance ($6$ orders), MIP spends $300.36$s on finding the optimal solution. When serving more than $8$ orders, the MIP method is intractable (the result cannot be obtained within two hours even using the world fastest commercial solver Gurobi while the inference time of all DRL methods is less than $1$s.

This experiment is conducted on multiple large-scale instances where $50$ vehicles are dispatched to serve $150$ delivery orders. Fig. 6 records the results on NUV and TC. In terms of NUV, baseline 3 performs best on all instances. ST-DDGN stands behind baseline 3, using $26.4$ vehicles on average. Besides, other DRL-based methods also perform much better than baseline 1 and baseline 2. With regards to TC, baseline 2 costs much less than baseline 3 even though it runs out of all vehicles. The result indicates that pursuing the least used vehicles blindly (i.e., baseline 3) might bring higher operation cost. Among the three baselines, baseline 1 achieves the least TC ($11167.5$). Graph-based DRL methods (DGN and ST-DDGN) outperform all three baselines. Especially, our ST-DDGN shows a certain advantage ($11080.5\pm 112.5$) over the best baseline.

To verify the effectiveness of our ST-DDGN in real scenario, we further compare the performance between ST-DDGN, vanilla DRL methods and baselines on a group of industry-scale instances. Each instance involves $600+$ orders and $150+$ vehicles. Here, the ‘industry-scale’ refers to that these instances are directly from the real daily transportation data (including delivery orders and cost setting) instead of the sampled instances used in previous experiments. Fig. 7 presents the results, where the horizontal axis gives the index of testing instance (i.e., Day 1, Day 2, … ) and vertical axis records NUV and TC respectively. It can be observed from Fig. 7(a) that baseline 2 uses all the vehicles and baseline 3 needs the least vehicles, which corresponds with the result of previous experiment. Correspondingly, baseline 1 performs the best among the three baselines. In comparison, DRL methods use less vehicles than baseline 1. Specifically, for serving all delivery orders, baseline 1 uses $91.8$ vehicles on average while DRL methods only use $84.125$ vehicles on average, which justifies the effectiveness of our MDP formulation in reducing the number of used vehicles. While on TC, ST-DDGN surpasses all other algorithms on most of instances, where ST-DDGN costs $33162.7$ and baseline 1 costs $36812.8$ on average. Even in the instances where other algorithms perform better (such as Day 10 and Day 16), ST-DDGN could still achieve similar performance. Thus, it is reasonable to consider that ST-DDGN can learn a policy with a trade-off between reducing the fixed cost and lowering the operation cost, which reduces the total cost effectively.

In this subsection, we would like to further justify the effectiveness of ST-DDGN via ablation studies. Compared to vanilla DRL methods, we introduce 1) the ST Score to balance ‘supply and demand’ and 2) the graph convolution (i.e., neighborhood attention module) to grasp the interactions among vehicles. Thus, an ablation study concerning the two components above is performed. Specifically, four various versions of the proposed model are involved, presented in Table II.

The convergence results of above four DRL methods are presented in Fig. 8. With respect to NUV (see Fig. 8(a)), all DRL methods learn a policy that can use less vehicles to serve all the orders than the baseline. Furthermore, it can be seen that models with graph convolution (DDGN and ST-DDGN) can learn better policies than DDQN and ST-DDQN. First, the training of DDGN and ST-DDGN converges much faster than that of the other two DRL methods. With regard to TC (see Fig. 8(b)), DDGN and ST-DDGN also perform much better than their counterparts. Although DDQN and ST-DDQN use the same number of vehicles with DDGN and ST-DDGN when converged, the total cost of DDQN and ST-DDQN is $5\%$ higher than that of DDGN and ST-DDGN. Besides, the learning curves in Fig. 8(b) show that ST-DDGN starts to converge at the $150$th episode while DDGN starts to converge at $200$th episode. Similar trend is also revealed on the learning curves of ST-DDQN and DDQN. Thus, it could be inferred that the introduction of ST Score indeed accelerates the policy learning. Because with the guide of ST Score, the neural network is supposed to take into consideration spatial-temporal balance between ‘supply and demand’ when dispatching vehicles.

In Section V-C4, we have inferred that the introduction of ST Score accelerates the policy learning. To support the inference, some evidences are given here.

The spatial-temporal distribution of delivery demand on a large-scale instance is presented in Fig. 10, which reveals the ‘hot spot’ of delivery demand. We then train four DRL models (ST-DDGN, DGN, DQN and AC) and adopt the policies being optimized on the instance respectively. For each model, we record the spatial-temporal distribution of vehicle capacity at each episode. The results are presented in Fig. 9. Furthermore, we calculate the Frobenius norm between demand distribution and capacity distribution at each episode, which is the meaning of ‘Diff’ in each subfigure. We hope that the Diff decreases gradually when policy iterating. The results show that all the four DRL models indeed learned the spatial-temporal demand distribution when policy converged. Because in each subfigure of Fig. 9, the Diff decreases when the policy iterates. Furthermore, it is observable that 1) the Diff of ST-DDGN is the smallest $(236.8)$ among that of four DRL methods when policy converged, and 2) the Diff of ST-DDGN drops fastest among four DRL methods. To some extent, it can explain why ST-DDGN can learn faster and better than other DRL methods on large-scale instances.

To summarize, we think our agent learned with ST-DDGN has the following preferences in dispatching. First, the agent dispatches as many vehicles to the high demand grid as possible. Because, to some extent, the spatial-temporal distribution of delivery capacity is consistent with that of delivery demand. In additions, it seems that the agent would consider the future delivery demand when dispatching vehicles for current order since some of delivery orders are postponed to be served together with future orders (i.e., hitchhiking). Thus, we believe that the introduction of spatial-temporal metric is helpful for the proposed relational learning framework to solve DPDP.