Reinforcement Learning for Trade Execution with Market Impact^††thanks: We thank Robert Almgren and Isaac Scheinfeld for helpful comments. Support from Swiss NSF Grant 10003723 is gratefully acknowledged.

We divide states into market states, which are visible to all market participants, and private states, only visible to the algorithm.

The algorithm observes states at the discrete time steps $t\in\{t_{0},t_{1},\dots,t_{N-1}\}$ right before taking an action $a(t).$ For $K\in\operatorname{\mathbb{N}},$ the action space is the simplex

An action $a=(a^{0},a^{1},\dots,a^{K}),$ represents the allocation of orders to limit and market orders. More precisely, $a^{0}$ is the fraction of the remaining inventory $M(t)$ to be sent as a market order, $a^{k}$ for $k\in\{1,2,\dots K-1\}$ is the fraction of the remaining inventory $M(t)$ to be placed $k$ ticks above the best bid price, and $a^{K}$ is the fraction of the remaining inventory $M(t)$ to be held outside of the order book. We always require that $K\leq D,$ such that the algorithm always posts orders within the visible order book levels defined in (1). For illustration, let us assume $K=3,$ that the current inventory is $M(t)=10,$ and that $a(t)=[0.1,0.5,0.3,0.1].$ With this allocation, the algorithm sends one lot as a market order, five lots as a limit order one tick above the best bid price, three lots as a limit order two ticks above the best bid price, and keeps one lot on the side without placing it in the order book. In the event that the vector $a(t)M(t)$ includes non-integer values, we round the entries $a^{0}(t)M(t),a^{1}(t)M(t),\dots,a^{K-1}(t)M(t)$ to the closest integer sequentially, always assuring that total allocated orders to not exceed $M(t).$ To preserve queue positions, the algorithm only cancels orders if necessary to reach the new allocation when reallocating orders, while prioritizing canceling orders with higher queue positions first.

In each time step $t_{n},$ the algorithm places market and limit orders according to an action $a_{n}.$ The market orders are filled immediately after $t_{n},$ leading to an instant cash flow, while the limit orders remain in the book waiting to be filled. The total cash flow from market and limit order fills in the interval $(t_{n},t_{n+1}]$ is denoted by $\bar{r}(s_{n},a_{n}).$ If any inventory is left at time $t_{N},$ the algorithm is forced to send a market order of size $M(t_{N}).$ In that case, the cash flow $\bar{r}(s_{N-1},a_{N-1})$ also contains the proceeds generated by the market order sent at time $t_{N}.$ Suppose that the algorithm sells $\gamma_{n}$ lots in the interval $(t_{n},t_{n+1}].$ Then, we define a normalized reward function, which compares the reward $\bar{r}(s_{n},a_{n})$ to the cash flow that would have been generated by trading at the initial bid price. More precisely, we define

Then objective (2) becomes

which describes the average revenue per lot relative to the initial bid price. The metric in (3) is a normalized version of the industry-standard measure used to evaluate execution algorithms, commonly known as implementation shortfall. We observed in our numerical experiments that normalizing the cash flows increases the training speed of the reinforcement learning algorithm.

We compare our trained reinforcement learning algorithm with two heuristic benchmarks and an alternative reinforcement learning algorithm using Dirichlet distributions. The Dirichlet-RL algorithm is explained in Appendix C. The heuristic strategies trade according to the following rules:

• •

  Submit and leave algorithm (SL algorithm): Submits the entire position $M$ at $t=0$ as a limit order at the best ask price.

• •

  Time-weighed average price⁴⁴4This is abbreviated with TWAP and represents a standard execution algorithm banks and hedge funds use. algorithm (TWAP algorithm): Divides the position into blocks of size $M/N.$ For $n\in\{0,1,\dots,N-1\},$ it posts a limit sell order of size $M/N$ at the best ask price every time step $t_{n}.$

Both algorithms sell any leftover inventory $M(t_{N})$ with a market order.

The trade execution problem described in Section 3 is a stochastic control problem. The transition probabilities from state $s_{n}$ to $s_{n+1}$ and the reward function $r(\cdot)$ are not available in closed form, but we can simulate them. Therefore, the problem can be solved with reinforcement learning. Actor-critic policy gradient algorithms are a class of reinforcement learning algorithms that aim to find an optimal stochastic policy $\pi_{\theta}(\cdot\mid s_{n}),$ referred to as actor, by estimating the gradient of the objective function $J(\pi_{\theta}).$ A separate parameterized value or advantage function, referred to as critic, is used to reduce the variance of the gradient estimate. We use a neural network with weights $\theta$ to map states $s_{n}$ to the parameters of the policy’s density. The gradient of the objective function can be computed as

by the policy gradient theorem; see e.g. Schulman et al. (2015). We note that the actions $a_{0},\dots a_{N-1}$ are sampled from a conditional distribution with density $\pi_{\theta}(\cdot\mid s_{n})$ and the states $(s_{0},\dots,s_{N-1})$ are sampled from the market environment. Here, $A^{\pi_{\theta}}$ is the advantage function, which is defined as

where $s$ is a state and $a$ an action at time step $n.$

In most applications of policy gradients with continuous action spaces, the multivariate normal distribution is used as an action distribution. However, in our application, the actions must be contained in the simplex. Therefore, we require a distribution with support on the simplex. The works Tian et al. (2022) and Winkel et al. (2023) have used the Dirichlet distribution to achieve this. But we found that the Dirichlet distribution performs sub-optimally in our experiments, even with extensive hyperparameter tuning. An alternative to the Dirichlet distribution is the logistic-normal distribution, which is also defined on the simplex.

The logistic transformation $h$ is a function that maps vectors $x=(x^{0},x^{1},\dots,x^{K-1})$ in $\operatorname{\mathbb{R}}^{K}$ to actions $a=(a^{0},a^{1},\dots,a^{K})$ in $\operatorname{\mathbb{S}}^{K},$ where

The inverse mapping $h^{-1}$ is given by

Let $X$ be a random variable with multivariate normal distribution on $\operatorname{\mathbb{R}}^{K}$ with mean $\mu=(\mu^{0},\dots,\mu^{K-1})\in\operatorname{\mathbb{R}}^{K}$ and covariance matrix $\Sigma=(\sigma^{ij})_{i,j=0,\dots,K-1}\in\operatorname{\mathbb{R}}^{K\times K}.$ Then, the transformed random variable $h(X)$ has a logistic-normal distribution. The distribution has support $\operatorname{\mathbb{S}}^{K},$ and its density function is given by

see, e.g. Atchison and Shen (1980) for a detailed overview of the logistic-normal distribution.

We choose a policy that has logistic-normal distribution with density $\pi_{\theta}(\cdot\mid s)$ for each state $s.$ The parameters $\theta$ are the weights of a neural network (or any other parametric family of functions) that maps states $s$ to the mean of the underlying normal distribution $\mu.$ We could also make the algorithm learn the covariance matrix $\Sigma.$ But here, we only learn $\mu$ and model $\Sigma$ as $\sigma_{i}I_{K},$ where $I_{K}$ is the $K$-dimensional identity matrix and $\sigma_{i}$ a constant that is reduced after each training epoch $i.$ This allows for exploration at the beginning of training but forces the algorithm to gradually reduce the variance of the stochastic policy until it becomes almost deterministic. Let $\varphi_{\theta}$ be the density of the underlying normal distribution on $\operatorname{\mathbb{R}}^{K}$ with mean $\mu_{\theta}$ and covariance matrix $\Sigma.$ Since the normalizing constant in (6) does not depend on $\theta,$ we obtain for a state $s$ and action $a\in\operatorname{\mathbb{S}}^{K}$ that

Combining equations (4) and (7) yields

Here $x\sim\varphi_{\theta}$ means that each element $x_{n}$ in the sequence $x=(x_{0},x_{1},\dots,x_{N-1})$ is sampled from a normal distribution with conditional density $\varphi_{\theta}(\cdot\mid s_{n})$ for each state $s_{n}$ in the sequence $(s_{0},s_{1},\dots,s_{N-1}).$ We emphasize that the gradient in (9) only affects the density of the underlying normal distribution. The elements $x_{n}\in\operatorname{\mathbb{R}}^{K}$ are then transformed with $h$ to obtain actions $a_{n}\in\operatorname{\mathbb{S}}^{K}$ without affecting the gradient.

The initialization of the mean and the scaling of the variances of the stochastic policy $\pi_{\theta}$ is an important aspect of training the policy. The mean and covariance of the logistic-normal distribution are not known in closed form. However, for the log ratios, we have

for $j,k,l,m\in\{0,1,2,\dots K\},$ with the convention $\mu^{K}=0$ and $\sigma^{j,K}=0.$ We set the parameters of the last layer of the neural network corresponding to the policy to zero at initialization with a bias term $b\in\operatorname{\mathbb{R}}^{K}.$ Then, the bias $b$ controls the initial allocations of the policy. Furthermore, we choose a diagonal covariance matrix to ensure that unrelated log ratios are uncorrelated. Indeed, for different $j,k,l$ and $m,$ equation (11) implies

For indices $j$ and $k$ with $j\neq k,$ we obtain for the variances of the log ratios that

The initial parameters of the covariance matrix $\Sigma$ control the initial exploration rate of the policy. As training progresses, we gradually lower the entries in the covariance matrix to discourage exploration and encourage exploitation. For $H\in\operatorname{\mathbb{N}}$ gradient steps and for an initial variance $\sigma_{\text{init}}>0$ and final variance $\sigma_{\text{final}}>0,$ we set the variances on the diagonal at each gradient step $i\in\{1,2,\dots,H\}$ to

The choice of the initial and final variances depends on the application.

Our first category of traders are noise traders who submit market, limit, and cancellation orders according to independent Poisson processes (Cont et al., 2010; Abergel and Jedidi, 2013). The intensities of market orders, limit orders, and cancellations arrivals are modeled in the following way.

• •

  Market buy and sell orders arrive with intensity $\lambda^{M}.$

• •

  For $k\in\{1,\dots,D\},$ limit buy and sell orders arrive with intensity $\lambda^{L,k}$ at $k$ ticks below the best ask price and $k$ ticks above the best bid price, respectively.

• •

  If the spread is $j$ ticks, then for $k\in\{j,\dots,D\},$ cancellations of limit buy orders arrive with intensity $\lambda^{C,k}v^{b,{k-j+1}}$ at $k$ ticks below the best ask price, and cancellations of limit sell orders arrive with intensity $\lambda^{C,k}v^{a,{k-j+1}}$ at $k$ ticks above the best bid price.

In contrast to the noise traders, the intensities of the tactical traders depend on volume imbalance. We define weighted volumes on the bid and ask sides by

where $c\in\operatorname{\mathbb{R}}^{+}$ is a damping factor. The weighted volume imbalance is then defined by

The damping factor $c$ effectively controls how sensitive the imbalance is to volumes posted at deeper levels in the book. Let $I^{+}$ and $I^{-}$ be the positive and negative parts of the imbalance. The intensities for different order types are defined as follows.

• •

  For $d^{M}\in\operatorname{\mathbb{R}}^{+},$ market buy orders arrive with intensity $d^{M}I^{+}$ and market sell orders arrive with intensity $d^{M}I^{-}.$

• •

  For $k\in\{1,2,\dots,D\},$ limit buy orders arrive with intensity $d^{k,L}I^{+}$ at $k$ ticks below the best ask price, and limit sell orders with intensity $d^{k,L}I^{-}$ at $k$ ticks above the best bid price.

• •

  If the spread is $j$ ticks, then for $k\in\{j,\dots,D\},$ cancellation of limit buy orders arrive with intensity $d^{k,C}I^{-}v^{b,{k-j+1}},$ at $k$ ticks below the best ask price, and and cancellations of limit sell orders arrive with intensity $d^{C,k}I^{+}v^{a,{k-j+1}},$ at $k$ ticks above the best bid price.

For $k\in\{1,\dots,D\},$ market orders, limit orders and cancellations arrive with sizes $1+\sigma^{M}|Z|,1+\sigma^{L,k}L|Z|$ and $1+\sigma^{C,k}|Z|$ respectively, where $Z$ is a random variable with a standard normal distribution for both noise and tactical traders. Order sizes are always rounded to the closest integer. Furthermore, both types of traders always cancel orders with the highest queue positions first, and no more orders can be canceled than exist at a given level.

The strategic trader buys or sells a position in the market using limit and market orders.

• •

  The direction, either buy or sell, is drawn randomly at the start time $-\Delta t$ with probability $1/2$ each.

• •

  The trader sends a market order of size $\nu^{M}\in\operatorname{\mathbb{N}}$ at time steps $-\Delta t,-\Delta t+\Delta t^{M},-\Delta t+2\Delta t^{M},...$ and stops trading at terminal time $T.$

• •

  The trader sends a limit order of size $\nu^{L}\in\operatorname{\mathbb{N}}$ at time steps $-\Delta t,-\Delta t+\Delta t^{L},-\Delta t+2\Delta t^{L},...$ and stops trading at terminal time $T.$

We simulate three increasingly complex market environments. The first market contains only noise traders and serves as a baseline. Price movements are entirely random, with no response from market participants. The second market includes both noise traders and tactical traders. In this setting, order flows react to order book imbalances, a behavior commonly observed in empirical studies of limit order book dynamics; see e.g. Stoikov (2018). The third market adds a strategic trader to the previous participants. It simulates a market that exhibits upward or downward price drift due to the presence of a strategic trader. This drift requires the execution algorithm to adapt its execution strategy by front-loading or back-loading orders. While such pronounced trends are rare in real markets, this environment tests whether the algorithm can detect directional signals from order book features and adjust its behavior accordingly. Although these simulations are stylized, they capture key characteristics of real-world market behavior. Importantly, our simulations differ significantly from historical market data replay because in our simulations, trades generate market impact. First, large market orders have direct impact since they consume the best levels on the opposite side of the order book. Second, limit orders have indirect impact as the tactical traders react to order book imbalance. Those effects cannot be incorporated in a simulation relying on historical data replay, as the data is fixed and does not react to the algorithm’s trading decisions. We compare both algorithms with the heuristic algorithms⁵⁵5We refer to the LN and DR algorithms and the benchmark algorithms as execution algorithms. defined in Section 3.4, in each of the three market environments, and for small and large positions to be liquidated.

As mentioned in Section 3, the execution algorithms trade at discrete time steps $t_{n}=n\Delta t$ for $n\in\{0,1,\dots,N-1\}.$ For our simulations, we choose $\Delta t=15s$ for a total of $N=10$ steps, such that the total execution duration is $150s.$ We use a short simulation horizon to reduce computational overhead during training. This choice does not compromise the validity of our results, as the underlying logic and outcomes would also hold for longer execution horizons. In each market, the execution algorithms trade a small (20) and a large (60) number of lots. What can be considered a small or large number of lots depends on the average traded volume in the interval $[0,T].$ In our examples, the average traded volume over the time interval is 100 lots in each market. When adding a new trader to the market, we adjust the trading intensities of the other traders so that the overall average trading volume remains the same at 100 lots. Hence, 20 lots correspond to 20% and 60 lots correspond to 60% of the total traded volume. In all three markets, we start the simulation at time $-\Delta t,$ where at initialization, we set the order book to its long-term average shape. Therefore, when the execution algorithms start trading at $t=0,$ the order book has moved from its initial state to a random state. The average shape of the order book can be estimated with a Monte Carlo simulation, which is described in more detail in Appendix A.2. The specific parameter choices for each trading agent are discussed in Appendix A.1.

We use the two heuristic strategies of Section 3.4 as benchmarks and introduce an additional benchmark based on the Dirichlet distribution, which is explained in more Detail in Section C. It would also be feasible to compare against other trade execution benchmark algorithms, such as the volume-weighted-average price algorithm. However, our algorithm is not directly comparable with other reinforcement learning algorithms that have been mentioned in the introduction. This has two reasons. First, algorithms developed on simulations relying on historical data are not comparable because our market simulation is inherently different from historical market data replay. Second, none of the related research works use the same state and action spaces formulation. In particular, no other work views the order placement problem as a dynamic allocation problem.

Figure 2 shows an evolution of the order book in the interval $[0,T]$ for the market with only noise traders. The bid and ask prices move from $p^{b}(0)=1000$ and $p^{a}(0)=1001$ three ticks down to $p^{b}(T)=997$ and $p^{a}(T)=998.$ The composition of the order book changes due to the arrival of market orders, limit orders, and cancellations.

The test results are summarized in Table 1, which contains the expected value and the standard deviation of the sum of normalized rewards defined in (3) for all market environments, position sizes, and execution algorithms. For example, the reward of $-1.10$ for the SL algorithm at 60 lots in the market with noise traders means that on average, the algorithm sold each of the 60 lots at $-1.10$ ticks less than the initial bid price. The plots in Figure 3 show histograms of normalized rewards based on 10,000 simulated executions. The random seeds used for the processes of different trading agents are selected independently from those employed during training, thereby avoiding over-fitting to the training simulations.

We train the LN algorithm with Algorithm 1. As explained in Sections 3.1–3.2, the parameter $K$ controls how many order book levels the algorithm observes and how deep it may post orders into the order book. In our simulations, the price rarely moves more than five ticks throughout the execution horizon. Therefore, we choose $K=6$ for the dimension of the simplex $\operatorname{\mathbb{S}}_{K}.$ This means that the RL algorithm may place limit orders up to five ticks above the best bid price and observes the volumes on the first five levels on both sides of the book. To encode the policy $\pi_{\theta}$ and the value function $V_{\vartheta},$ we choose two separate feed-forward neural networks with two hidden layers, $\tanh$-activation functions in the hidden layers and no activation functions in the last layers. The output layer of the policy network has $K$ nodes, and the last layer of the value function network has one node. We start by setting the covariance matrix $\Sigma$ to the identity matrix and then progressively reduce its values according to the linear schedule (12). We choose an initial bias $b=(-1,-1,\dots,-1)\in\operatorname{\mathbb{R}}^{K}$ for the policy network. We obtain from (10) that

showing that action $a_{K}$ is more likely than actions $a_{0},\dots,a_{K-1}.$ This means the algorithm does not place any orders at the start of the training cycle, which ensures that the algorithm observes complete trajectories up to the terminal time $T$ early in the training process, preventing it from getting stuck in local minima. Further implementation details concerning the training of the LN algorithm are discussed in Appendix B.

First of all, we observe that in all three market simulations, the LN algorithm outperforms the DR algorithm, which is visible in Table 1 and Figure 3. Figure 4 shows the development of the episode returns over the full training cycle. The upper panel shows the results for 20 lots, and the lower panel shows the results for 60 lots. We observe that in all experiments, except when the agent executes 20 lots in the market only containing noise traders, the rewards for the LN algorithm converge faster than for the DR algorithm. This suggests that the LN algorithm is a superior alternative to the DR algorithm in this allocation problem. As the performance of the LN and DR algorithms is qualitatively similar, we focus our attention on the performance of the LN algorithm relative the the heuristic benchmarks in the following sections.

The first two rows of Table 1 demonstrate that the LN algorithm outperforms both heuristic benchmarks and the DR algorithm for 20 and 60 lots. This is also visible in the histograms in Figure 3(a). We remark that all algorithms in all markets have a high standard deviation. This is unavoidable as the algorithms operate in a highly stochastic environment. Sending a market order at the beginning of the execution period would result in a strategy with almost zero standard deviation; however, the execution price would be worse than the initial best bid and, therefore, worse than most of our execution strategies. For 20 lots, the LN algorithm’s histogram plot exhibits a bump at one and two ticks above the best bid price, indicating that it places limit orders deeper in the book and cuts losses when the price moves against it. For 60 lots, while the histograms of the LN and SL algorithms appear similar, the LN algorithm’s histogram exhibits less skew to the left, further suggesting that it effectively limits losses when the market moves away. The TWAP algorithm has the worst performance for both position sizes. Since the market does not react to volume imbalance, placing the entire position passively at the start has no downside.

The third and fourth rows of Table 1 and Figure 3(b) display the results for the market consisting of noise and tactical traders. Both demonstrate the LN algorithm’s superior performance relative to the benchmark algorithms for small and large position sizes. The performance of the LN algorithm is slightly better than the DR algorithm. As before, for 20 lots, we observe bumps in the histogram at one and two ticks above the best bid price for the LN algorithm. For $60$ lots, the LN algorithm’s histogram has less left skew than the benchmark densities, indicating that it cuts losses when prices move against it. The presence of tactical traders reacting to order book imbalances introduces indirect market impact. Consequently, posting the entire position passively at the beginning, as done by the SL algorithm, triggers the tactical traders to trade in the other direction, leading to the worst performance. Spreading the order out over the whole time horizon, as done by the TWAP algorithm, is better. Still, the LN algorithm performs best, suggesting intelligent placement strategies.

The fifth and sixth rows of Table 1 and Figure 3(c) show the results for the market consisting of noise, tactical, and strategic traders and demonstrate the strong performance of the LN algorithm. The presence of a strategic trader introduces an upward or downward drift in the market. The LN algorithm detects this drift using the historical market and limit order flow features and the mid-price drift feature defined in Section 3.1. The performance of the LN algorithm is slightly better than the DR algorithm. The relative shift of the LN algorithm’s histogram compared to those of the benchmark algorithms suggests that it trades more aggressively when the strategic trader is selling and more patiently when the strategic trader is buying.