Stock price formation: useful insights from a multi-agent reinforcement learning model

Background: Multi-agent systems (MAS) or agent-based models (ABM) have had a long history of statistical inference in quantitative finance research. In particular, they have been used to study phenomena leading to price formation and hence general market microstructure, such as: the law of supply and demand Benzaquen and Bouchaud (2018), game theory Erev and E.Roth (2014), order books Huang et al. (2015), high-frequency trading Wah and Wellman (2013); Aloud (2014), cross-market structure Xu et al. (2014), quantitative easing Westerhoff (2008), market regulatory impact Boero et al. (2015), or other exogenous effects Gualdi et al. (2015). Remarkably, financial MAS have highlighted over the years certain specific patterns that are proper to virtually almost all asset classes and time scales, called stylised facts. These have since been an active topic of quantitative research Lipski and Kutner (2013); Barde (2015), and can be grouped in three main neighbouring categories: i- distributions of price returns are non-gaussian Cristelli (2014); Cont (2001); Potters and Bouchaud (2001) (they are asymmetric, negatively skewed, and platykurtic), ii- volatilities and volumes are clustered Lipski and Kutner (2013) (large jumps in prices and volumes are more likely to be followed by the same), iii- price auto-correlations decay Cont (2001, 2005) (there is a loss of arbitrage opportunity). In particular, the second stylised fact has long-range implications on the dynamics of meta-orders, and comprises the square-root impact law Bouchaud (2018a) (growth in square-root of orders impact with traded volumes). Implicit consequences of these stylised facts have fed numerous discussions pertaining to the validity of market memory Cont (2005); Cristelli (2014) and the extension of the efficient market hypothesis Fama (1970); Bera et al. (2015).

Epistemology: In the past, the epistemological pertinence of such models was sometimes put into question, because of the general conceptual challenge of framing realistic agents. Unlike other disciplines from hard science, financial MAS indeed had to justify their proper bottom-up approach to complex system inference Gao et al. (2011) in the light of a specific challenge of their field, namely the difficulty to realistically model and emulate human agents. Such critics especially carried greater weight by the fact that previous generations of financial MAS relied on so-called zero-intelligence agents (Gode and Sunder, 1993): these would trade according to specific and imbedded rules of trading, thereby overshadowing certain dynamics proper to game and decision theory that are crucial to real market activity. Yet, if compared with other famous types of models used in quantitative finance, like econometrics Greene (2017), MAS have two major advantages: i- they naturally display specific emergent phenomena proper to complex systems Bouchaud (2018b), and ii- they require fewer model assumptions (no gaussian distributions, no efficient market hypothesis Fama (1970); Bera et al. (2015), etc.). As for their shortcomings, we can mention specific conceptual challenges pertaining to: i- modelling complex system heterogeneity Chen et al. (2017), ii- discretionary framing of certain model parameters (e.g. number of agents) apart from possible empiricism Platt and Gebbie (2018).

Prospects: These general conceptual and epistemological considerations have been upset in the past few years, by the notable progress of two fields. The first, namely machine learning and especially multi-agent reinforcement learning Silver et al. (2018) (RL), has produced spectacular results that have far-reaching applications to other domains of interest to quantitative finance, like decision theory and game theory. The second, neuroscience and neurofinance especially, has benefitted from the wide use of brain-imaging devices Eickhoff et al. (2018) and peer data curation Smith and Nichols (2018). On both ends, some sort of technological emergence within these two fields is of special interest and relevance to financial MAS research, in that machine learning can imbed results from the latter Lefebvre et al. (2017); Palminteri et al. (2015), and vice-versa Duncan et al. (2018); Momennejad et al. (2017). Among other machine learning applications to finance (Hu and Lin, 2019; Neuneier, 1997; Deng et al., 2017), and in a way similar to what has been done for recent order book models (Spooner et al., 2018; Biondo, 2019; Sirignano and Cont, 2019), next-generation MAS stock market simulators can now be designed, and outstrip the former epistemological considerations pertaining to agent design, with a whole new degree of realism in market microstructure emulation. More importantly, these can address issues never computationally studied before, such as the crucial issues of agent information and learning, central to price formation Dodonova and Khoroshilov (2018); Naik et al. (2018) and hence to all market activity.

Our study: We have designed such a MAS stock market simulator, in which agents autonomously learn to do price forecasting and stock trading by reinforcement learning, via a centralised double-auction limit order book, and shown in a previous work Lussange et al. (2019) its calibration performances with respect to real market data from the London Stock Exchange daily quotes between $2008$ and $2018$. We shall not review here its design in full details, but will recall its general architecture in Section II. Then our study will focus in Section III on agent learning: how can we gauge how heterogenous are the agent policies as compared to one another, and what are the trading characteristics of successful agents. Finally in Section IV, we will study the impact of such agent learning on the whole market at the mesoscale, and in particular with respect to herding or reflexivity effects when agents emulate the investments of the best (or worst) agent, and when increasing proportions of agents trade randomly as “noise traders.” As said, the greatest interest of financial MAS is to study and quantitatively gauge the impact of agent learning, which is at the heart of the price formation processes and hence of all market activity, and the motivation for multi-agent reinforcement learning for such a framework is motivated by the important parallels between reinforcement learning and decision processes in the brain Dayan and Daw (2008).

Architecture: We here briefly sketch the general architecture of our MAS simulator. Our model relies on a number $I$ of economic agents autonomously trading a number $J$ of stocks, over a number $T$ of simulation time steps, where we set $T_{y}=286,T_{m}=21,T_{w}=5$ as the number of annual trading days. All $I$ agents and their individual parameters are initialised at $t=0$, with a portfolio made of specific stock holdings of value $A_{equity}^{i}(t)=\sum_{j=0}^{J}Q^{i,j}(t)P^{j}(t)$, where $Q^{i,j}(t)$ is the number of stocks $j$ of agent $i$ and $P^{j}(t)$ the market price of stock $j$, together with risk-free assets (e.g. a bank account) of value $A_{bonds}^{i}(t)$. Then all market prices are initialised at $P^{j}(t=0)=\pounds 100$, and as in other models Franke and Westerhoff (2011); Chiarella et al. (2007), the simulation generates $J$ time series $\mathcal{T}^{j}(t)$, which correspond to the fundamental values of the stocks. These are not fully known by the $I$ agents. Instead, each agent $i$ approximates the values $T^{j}(t)$ of stock $j$ according to a proprietary rule (Murray, 1994) of cointegration $\kappa^{i,j}[\mathcal{T}^{j}(t)]=\mathcal{B}^{i,j}(t)$. The time series $\mathcal{B}^{i,j}(t)$ are hence the approximation of the fundamental values of stock $j$ over time $t$ according to agent $i$. Each agent thus relies on these two sources of information for its stock pricing strategy: one that is chartist, and one that is fundamental. At $t=0$, the agent are agnostic with respect to trading. They are then allowed to learn over the course of $1000$ time steps, after which their stock holdings and risk-free assets are reset back to their first value. The simulation and following results are hence applied to agents after this learning phase of $1000$ time steps.

Initialisation: Let $\mathcal{U}()$ and $\mathcal{U}\{\}$ denote the continuous and discrete uniform distributions, respectively. Each agent is then initialised with the following parameters: a drawdown limit $l^{i}\sim\mathcal{U}(50\%,60\%)$ (which is the maximum year-to-date loss in net asset value below which the agent is set as bankrupt), a reflexivity parameter $\rho^{i}\sim\mathcal{U}(0,100\%)$ (which gauges how fundamental or chartist the agent is via a weighted average of its price forecast), an investment horizon $\tau^{i}\sim\mathcal{U}\{T_{w},6T_{m}\}$ (which is the number of time steps after which the agent liquidates its position), a trading window $w^{i}\sim\mathcal{U}\{T_{w},\tau^{i}\}$ (which assesses the optimal trading time for sending an order), a memory interval $h^{i}\sim\mathcal{U}\{T_{w},T-\tau^{i}-2T_{w}\}$ (which is the size of the past lag interval used by the agent for its learning process), a transaction gesture $g^{i}\sim\mathcal{U}(0.2,0.8)$ (which scales with bid-ask spread to set how far above or below the value of its own stock pricing the agent is willing to deal the transaction), and a reinforcement learning rate $\alpha\sim\mathcal{U}(0.05,0.20)$ proper to both reinforcement learning algorithms $\mathcal{F}^{i}$ and $\mathcal{T}^{i}$ (see below).

Order book: At each time step, the agents may send transaction orders to the order book of each stock, whose function is to match these orders and process associated business transactions. More specifically, a number $J$ of order books are filled with all the agents’ trading limit orders for each stock $j$ at time step $t$. All buy orders are there sorted by descending bid prices, all sell orders are sorted by ascending ask prices, each with their own associated number of stocks to trade. Then the order book clears these matching orders at same time step $t$, with each transaction set at mid-price between buy and sell-side, starting from the top of the order book to the lowest level where the bid price still exceeds the ask price. Importantly, we then define the market price $P^{j}(t+1)$ of stock $j$ at the next time step $t$ as that last and lowest level mid-price cleared by the order book. We also define the trading volume $V^{j}(t+1)$ as the number of stocks $j$ traded during that same time $t$. We also model the friction costs via broker fees (broker fees, ) applied to each transaction set at $0.1\%$, an annual risk-free rate of $1\%$ applied to $A_{bonds}^{i}(t)$, and an annual stock dividend yield of $2\%$ according to (Div, ) applied to $A_{equity}^{i}(t)$.

Agents: Each agent autonomously uses two distinct reinforcement learning algorithms to interact with the market. For a brief introductory sum up of reinforcement learning, we refer the reader to Lussange et al. (2019), and to Sutton and Barto (1998); Wiering and van Otterlo (2012); Szepesvari (2010) for a thorough study of the subject. A first algorithm $\mathcal{F}^{i}$ learns the optimal econometric prediction function for the agent’s investment horizon, depending on specific local characteristics of the market microstructure and the agent’s fundamental valuation $\mathcal{B}^{i,j}(t)$. It thus outputs this price forecast, which will in turn enter as input the second reinforcement learning algorithm $\mathcal{T}^{i}$. This second algorithm is in charge of sending an optimal limit order to a double auction order book (N and Larralde, 2016) at this same time step, based on this prediction and a few other market microstructure and agent portfolio indicators. Each reinforcement learning algorithm is individually ran by each agent $i$ following a direct policy search, for each stock $j$, and at each time step $t$. Each algorithm has $27\times 27=729$ and $108\times 9=972$ potential action-state pairs, respectively. We define the sets of states $\mathcal{S}$, actions $\mathcal{A}$, and returns $\mathcal{R}$ of these two algorithms according to the following:

i- Forecasting: In the first algorithm $\mathcal{F}^{i}$, which is used for price forecasting, the agent continuously monitors the longer-term volatility of the stock prices $s_{0}^{\mathcal{F}}=\{0,1,2\}$, their shorter-term volatility $s_{1}^{\mathcal{F}}=\{0,1,2\}$, and the gap between its own present fundamental valuation and the present market price $s_{2}^{\mathcal{F}}=\{0,1,2\}$. This allows the agent to retrieve useful information on the microstructure and topology of the volatility, while avoiding dimensionality issues. Out of this state, it learns to optimise its price prediction at its investment horizon $\tau^{i}$ by selecting from a direct policy search three possible actions: choosing a simple forecasting econometric tool based on mean-reverting, averaging, or trend-following market prices $a_{0}^{\mathcal{F}}=\{0,1,2\}$, choosing the size of the historical lag interval for this forecast $a_{1}^{\mathcal{F}}=\{0,1,2\}$, and choosing the weight of its own fundamental stock pricing in an overall future price estimation, that is both fundamentalist and chartist $a_{2}^{\mathcal{F}}=\{0,1,2\}$. This is done in proportion to the agent reflexivity parameter $\rho^{i}$. Via this action $a_{2}^{\mathcal{F}}$, the agent thus learns to gauge how fundamental or chartist it should be is in its price valuation, via a weighted average involving $\rho^{i}$ in the agent’s technical forecast of the market price $\hat{P}^{i,j}(t)$ and its fundamental pricing $\mathcal{B}^{i,j}(t)$. At each time step, the rewards $r^{\mathcal{F}}=\{-4,-2,-1,1,2,4\}$ are defined according to percentiles in the distribution of the agent’s mismatches between past forecasts at time $t-\tau^{i}$ and their eventual price realisation at time $t$. In parallel, an off-policy method computes the optimal action that was to be performed $t-\tau^{i}$ time steps ago, now that the market price $P^{j}(t)$ is realised, and updates the agent policy accordingly.

ii- Trading: In the second algorithm $\mathcal{T}^{i}$, which is used for stock trading, the agent continuously monitors whether the stock prices are increasing or decreasing according to the output of the former algorithm $s_{0}^{\mathcal{T}}=\{0,1,2\}$, their volatility $s_{1}^{\mathcal{T}}=\{0,1,2\}$, its risk-free assets $s_{2}^{\mathcal{T}}=\{0,1\}$, its quantity of stock holdings $s_{3}^{\mathcal{T}}=\{0,1\}$, and the traded volumes of stock $j$ at former time step $s_{4}^{\mathcal{T}}=\{0,1,2\}$. Out of this state, it learns to optimise its investments by selecting two possible actions via a direct policy search: sending a transaction order to the order book as holding, buying, or selling a position in a given amount proportional to its risk-free assets and stock holdings $a_{0}^{\mathcal{T}}=\{0,1,2\}$, and at what price wrt. the law of supply and demand $a_{1}^{\mathcal{T}}=\{0,1,2\}$. The cashflow difference between the profit or loss consequent to the agent’s action, and that without action having been taken, are then computed $\tau^{i}$ time steps after each transaction. The rewards $r^{\mathcal{T}}=\{-4,-2,-1,1,2,4\}$ are defined according to percentiles in the distribution of these agent’s past cashflow differences. In parallel, an off-policy method computes the optimal action that was to be performed $t-\tau^{i}$ time steps ago, now that the market price $P^{j}(t)$ is realised, and updates the agent policy according to it.

Following calibration performances shown in a previous work Lussange et al. (2019), we have used a multi-agent reinforcement learning system to model stock market price microstructure. The advantage of such a framework is that it allows to gauge and quantify agent learning, which is at the source of the price formation process, itself at the foundation of all market activity. We first studied agent learning, and then its mesoscale impact on market stability and agent performance.

According to our results on policy learning, we posit that there are more trading strategies that yield successful portfolio performance, than unsuccessful.

We also found that best performing agents have a propensity for being fundamentalists rather than chartists in their approach to asset price valuation, and to be less stringent in their choices of transaction orders (i. e. willing to transact orders with larger bids or lower asks).

Next, we studied the impact on the market of agent learning rates, and found that market volatilities at all time-scales did not vary much (with more agents with larger learning rates, or when all agent collectively have larger learning rates), except for tail events, with average numbers of crashes greatly increasing. We also found that agent bankruptcy rates were not much impacted by such variations in reinforcement learning rates.

Then we studied the effect of herding or reflexivity, when increasing percentages of agents follow and emulate the investments of the best (and worst) performing agent at each simulation time step. As expected, we found that both such behaviours greatly increase market instability. Yet remarkably, bankruptcy rates of simulations with a best agent herding set up remain quite stable, regardless of the percentages of such agents, and regardless of the yet strongly increasing market volatilities and numbers of crashes.

Finally, we sought to explore the impact of agent trading information on the price formation process, with larger proportions of “noise traders” (i. e. agents trading randomly), and found a much greater market stability with increasing percentages of such agents, with a number of crashes virtually vanishing. We also found such markets to be more prone to display bull regimes, and that agents bankruptcy rates slightly diminished.

We trust such predictions under our model assumptions would be of interest not only to academia, but to industry practitioners and market regulators alike. A natural extension of our model would be to endow agents with a short selling ability (in order to account for specific microstructure effects in times of bubbles for instance). One could also add to each agent to capacity to perform proper portfolio diversification in the model’s multivariate framework. Finally, we graciously acknowledge this work was supported by the RFFI grant nr. 16-51-150007 and CNRS PRC nr. 151199.