Learning Optimal Behavior Through Reasoning and Experiences^††thanks: We thank Sam Gershman for extensive and helpful discussions. Email addresses: Ilut cosmin.ilut@duke.edu, Valchev valchev@bc.edu.

By abstract reasoning we mean the internal deliberation process through which the agent thinks abstractly about his decision problem, and learns about $Q_{\pi^{\ast}}(a,s)$ generally.¹¹¹¹11For ease of exposition, we assume that the action space is discrete with cardinality $N_{a}=|a|$, but this can be relaxed and the framework can be defined on a continuous space of actions as well. We will remain agnostic about the specific mode of deliberation the agent engages in, but for example they can be trying to do value function iteration or just thinking forward through the decision tree of their problem. This kind of reasoning is deliberate and abstract, as it tries to deduce new information about $Q_{\pi^{*}}(a,s)$ over and above the simple experience of the flow utility given past choices. It is a potentially powerful source of information for the agent, but is mentally costly.

Formally, each period the agent can generate a vector of reasoning signals $\eta_{t}^{R}$ as unrestricted linear functions of $Q_{\pi^{\ast}}(a,s_{t})$

where $\varepsilon_{\eta,t}\sim N(0,\Sigma_{\eta,t})$ and $\Omega_{t}$ is a $N_{a}\times N_{a}$ matrix. The agent optimally chooses the structure of the matrix $\Omega_{t}$ and the noise variance matrix $\Sigma_{\eta,t}$, subject to a cost-benefit tradeoff introduced below in subsection 2.4.2.

The vector of signals $\eta_{t}^{R}$ then updates beliefs over $Q_{\pi^{\ast}}(a,s)$ at all pairs $(a,s)$:

where the signal-to-noise ratio $\alpha_{t}^{R}(a,s)$ is given by

which can be tractably computed from $\widehat{\Sigma}_{t-1}(a,a^{\prime},s,s^{\prime})$ and the update based on $\eta_{t}^{E}$ described above.

Importantly,the reasoning signals further reduce uncertainty, so that

denote the posterior variance function over the vector

Interpretation. Our terminology of reasoning is similar to the notion of planning or model-based learning in the computational and RL literature (Sutton and Barto (2018)). Reasoning can in practice work through various specific processes. Consider for example a specific tool used in the model-based RL – the simulation of a limited number of future paths of actions and resulting states and utility flows in the future (see for example Moerland et al. (2023)). In our framework, this simulation is captured by the noisy signals $\eta_{t}^{R}$ that update the prior beliefs of the agent. In a related way, by producing new information internally to the agent, without being observed by an outsider, the notion of reasoning here also resembles in economics that of ‘fact-free learning’ in Aragones et al. (2005) and Alaoui and Penta (2022).

However, our proposed framework purposefully abstracts from the specifics of the mental method behind $\eta_{t}^{R}$. Instead it aims to capture a key tradeoff that many different solution methods share, namely that taking more computational steps in any given method (e.g. simulate more and further paths forward), leads to (i) higher accuracy of the resulting solution, but (ii) is cognitively costlier, as we describe next.

Overall, by incorporating both signals $\eta_{t}^{E}$ and $\eta_{t}^{R}$ in the conditional beliefs of our agent we are integrating, in an endogenous way, both “model-free” and “model-based” learning, which are the two main types of learning discussed in RL.

This joint integration of both types of learning is present in both some recent treatments of RL (see Sutton and Barto (2018)) and also in classical RL algorithms such as Dyna-Q (Sutton (1990, 1991)).¹²¹²12These algorithms include many extensions, such as Dyna-Q+ (Sutton (1990)), which like in the current framework further stimulates exploration. Nevertheless, the RL literature usually resorts to ad-hoc assumptions on how the two types of learning are mixed together, while our framework proposes an endogenous arbitration between the two types of signals based on formal Bayesian updating notions.

Furthermore, the specific formulation of belief updating for both signals in equations (9) and (6) resembles temporal difference solution techniques, used extensively in RL (Sutton (1988)), and in particular their Bayesian, GP-based version, connecting to the approach in Engel et al. (2003, 2005).

Equations (9) and (11) describe the updated beliefs, taking as given the structure for the signal in equation (8). Critically, given the current state $s_{t}$ and prior beliefs in equation (4), we let the agent jointly choose her optimal reasoning structure (i.e. $\Omega_{t}$ and $\Sigma_{\eta,t}$) and the action policy $\pi_{t}(a|s_{t})$ that he will follow this period.

The objective function in (12) is subject to two types of constraints on the action distribution. First, is that of feasibility: $\pi(a|s_{t})=0$ for actions that do not satisfy the budget constraint $a\notin B(s_{t})$, and that the action distribution probabilities sum to one:

Second, the action distribution $\pi_{t}(a|s_{t})$ is subject to an entropy constraint:

Interpretation. The objective function in (12) and the entropy constraint in (13) allow our framework to capture jointly a variety of forces of interest emphasized in the RL and cognitive science literature, as follows.

The first term in the objective of equation (12) reflects the agent’s benefit of exploitation, or “greediness” in the RL language. This force incentivizes the agent to choose the action $a_{t}$ with the current highest estimated value $\widehat{Q}_{t}(a_{t},s_{t})$, where the latter is given in equation (9). The second term captures a possible cognitive dissonance cost (Aronson (1969), Akerlof and Dickens (1982)). Essentially, when the primitive disutility parameter $w>0$, there is a disutility cost of uncertainty over the values associated to each action. This cost generates a benefit of reasoning that goes purely through the reduction of that dissonance and the disutility of facing uncertainty about the Q-function. This mechanism is qualitatively similar to the one used in Ilut and Valchev (2023), and is distinct from the novel exploitation-experimentation tradeoff described below.

The third term in the objective function measures the cognitive cost of reasoning as proportional to the information content of signals $\eta_{t}^{R}$, with information flow quantified as the reduction in entropy achieved by acquiring these signals, following a standard information theory approach, e.g. Sims (2003a). The reasoning cost captures the fact that increasing the reasoning intensity (e.g. mentally simulating more paths forward) requires higher cognitive effort. Like in the cognitive control literature (e.g. Kool et al. (2017), Botvinick et al. (2004), Botvinick and Cohen (2014)), the constant marginal cost $\kappa>0$ can be interpreted as the opportunity cost of cognitive capacity, which can be otherwise employed on other, outside-the-model tasks.¹³¹³13Thus, $\kappa$ will be higher if individuals have a higher opportunity cost of cognitive capacity or if their particular deliberation process behind $\eta^{R}_{t}$ takes longer to produce a given amount of information. In addition, $\kappa$ will also be higher if the environment is more complex and it is thus objectively harder to come up with insights on the unknown optimal $Q$-function.

In turn, the interpretation and aim of the constraint in (13) is to capture a desire for experimentation. As in typical bandit problems, as long as the value function $Q_{\pi^{*}}(a,s)$ is uncertain, there is a desire to explore the function in other parts of the state space. Following the RL literature, we parsimoniously model this desire to experiment using the idea of “entropy regularization”, or maximum entropy RL, (eg. Mnih et al. (2016), Haarnoja et al. (2017), Eysenbach and Levine (2019)). This regularization requires the entropy of the action distribution (the LHS of equation (13)) to be bigger than some minimum threshold, thus ensuring randomization. With this constraint, the chosen action distribution $\pi_{t}(a|s_{t})$ is not degenerate, but always puts some probability of exploring any feasible action, thus capturing the exploration incentive inherent to dynamic learning problems. Importantly, in our framework this entropy constraint builds in that experimentation is valuable only to the extent to which the $Q$-function is uncertain – indeed, if $Q_{\pi^{*}}(a,s_{t})$ was known, there would be no point to experiment as there is nothing further to learn. Hence, when the experimentation parameter $h>0$, in the RHS of equation (13) the entropy lower bound is set proportional to the remaining subjective uncertainty over $Q_{\pi^{*}}(a,s_{t})$, as measured by $\sum_{a}\sigma_{t}^{2}(a,s_{t})$.

Let $\delta_{t}$ denote the Lagrange Multiplier on the constraint in equation (13). Given the set of feasible actions in $B(s_{t})$, the optimal policy action is

where $\widehat{\pi}_{t}^{greedy}(a|s_{t})$ is the degenerate greedy policy

Softmax. The optimal policy in (14) takes the form of the softmax function widely used in statistics and machine learning. This function takes the expected rewards of following any given action and transforms them into action probabilities (eg. Sutton and Barto (2018)). However, in that literature $\delta_{t}$, known as the ‘temperature’ parameter, is typically exogenous. The lower is $\delta_{t}$, the more the expected rewards affect the probability of taking actions. In our model, $\delta_{t}$ is endogenous and state and time-dependent. Specifically, it will be high for states $s_{t}$ where the agent faces a higher uncertainty (and thus $\sum_{a}\sigma_{t}^{2}(a,s_{t})$ is high).

In economics, a similar softmax function describes the multinomial logic model of choices (eg. Luce (1959)), typically micro-founded from a random utility model where valuations of actions are subject to additively separable and independently distributed shocks drawn from the extreme-value distribution (eg. McFadden (1974)). Instead, the optimal policy $\widehat{\pi}_{t}(a|s_{t})$ proposed here does not rely on such random utility shocks.

A complementary micro-foundation for logit choice in the literature is based on rational inattention (Sims (2003b)). In that work (eg. Matějka and McKay (2014) in economics, or the policy compression work of Lai and Gershman (2021) in cognitive sciences), the entropy of the actions distribution, i.e. LHS of equation (13), speaks to an information flow constraint across actions, given signals about the rewards associated to those actions. Here instead the entropy on actions distribution speaks to the entropy regularization logic in RL, per above.

To see that comparison, consider setting $h=0$ in constraint (13). In that case, the constraint would not be binding and the optimal action becomes $\widehat{\pi}_{t}^{greedy}(a|s_{t})$. Critically, even without a role for the entropy on actions distributions, when $\kappa>0$ there is still an entropy informational cost on reasoning signals in objective (12). So, when $h=0$ reasoning is here still imperfect, but given imperfect posterior Bayesian estimates $\widehat{Q}_{t}(a,s_{t})$, the agent would take the greedy perceived action with probability one. The extreme case of $h=0$ would thus have no role for exploration, a fundamental feature to dynamic learning problems.¹⁴¹⁴14This discussion also implies that the framework could extend the RHS of constraint (13) to be $c+h\sum_{a}\widetilde{\sigma}_{t}^{2}(a,s_{t})$. Even if uncertainty over $\widehat{Q}_{t}(a,s_{t})$ would disappear, when the new parameter $c>0$, it would still difficult for the agent to compare all the $\widehat{Q}_{t}(a,s_{t})$ at current state $s_{t}$ for all possible actions, generating stochastic choice like in the policy compression logic. That integrated softmax could then nest both policy compression and experimentation sources.

At another extreme, consider the limit of costless reasoning, i.e. $\kappa=0.$ In that case, $\sigma_{t}^{2}(a,s_{t})=0$ as cognitive effort is chosen to be infinitely high, and agents learn $Q_{\pi^{*}}(a,s)$ perfectly. As there is no remaining uncertainty, the entropy regularization constraint (13) would not bind, $\delta_{t}=0,$ and the optimal action follows the greedy policy. The objective in equation (12) would then recover exactly the problem solved under full rationality (see equations (1) and (2)) since $\widehat{Q}_{t}(a,s)=Q_{\pi^{*}}(a,s).$

The optimal reasoning choice consists of the agent deciding on the matrix $\Omega_{t}$ and signal noise variance matrix $\Sigma_{\eta,t}$. To solve this problem, we use insights from rate-distortion theory.

Let $\lambda_{t}^{(i)}$ denote the eigenvalues of posterior variance ${\widehat{\Sigma}}_{t}(a,a^{\prime};s_{t})$. Using the standard property that a matrix determinant equals the product of its eigenvalues, $\ln(|\widehat{\Sigma}_{t}(a,a^{\prime};s_{t})|)=\sum_{i}\ln(\lambda_{t}^{(i)}),$ it follows that optimal reasoning can be expressed as choosing the resulting eigenvalues $\lambda_{t}^{(i)}$ optimally.

For notational convenience, it is useful to sort the eigenvalues in descending order so

By standard properties of variance matrices $\Sigma_{\eta,t}$ is positive definite, and thus the (sorted) eigenvalues of the posterior variance $\widehat{\Sigma}_{t}(a,a^{\prime};s_{t})$ must be weakly smaller than those of the “prior” variance $\widehat{\Sigma}_{t}^{E}(a,a^{\prime};s_{t})$, hence $\lambda_{t}^{(i)}\leq\lambda_{E,t}^{(i)}$ (where $\lambda_{Et}^{(i)}$ denote the sorted eigenvalues of $\widehat{\Sigma}_{t}^{E}(a,a^{\prime};s_{t})$). Taking this constraint into account and optimizing the objective function over $\lambda_{t}^{(i)}$, the optimal reasoning choice implies

Hence, the agent has an optimal target level for uncertainty and wants to only reduce eigenvalues that currently bigger than that threshold $\frac{\kappa}{w+\delta_{t}h}$ – a property known as reverse ‘water filling’ in rate-distortion theory (see Cover and Thomas (1999), chapter 13). Thus, we can show that the optimal signal structure is for $\Omega_{t}$ to be the matrix of eigenvectors of $\widehat{\Sigma}_{t}^{E}(a,a^{\prime};s_{t})$, and $\Sigma_{\eta,t}$ be a diagonal matrix with entries¹⁵¹⁵15In practice, the eigenvalues are likely to decline in value quickly, so only a few of these signals will typically have non-infinite noise variance making the solution very tractable and easy to implement.

Put together, equations (14) and (16) uncover a tight intuitive interaction between actions and reasoning intensity. For example, take states $s_{t}$ where uncertainty is high and thus the endogenous $\delta_{t}$ is larger. By (14), a tighter constraint induces the agent to explore more and thus deviate from the action with the currently highest perceived $\widehat{Q}_{t}(a,s_{t})$. On the other hand, reasoning lowers uncertainty and thus relaxes this constraint, lowering $\delta_{t}$ and leads to a policy function $\pi_{t}(a|s_{t})$ that selects actions $a$ with a higher expected payoff $\widehat{Q}_{t}(a,s_{t})$.

Looping back to the choice of how much cognitive effort to invest in reasoning, from (16) we see that when $\delta_{t}$ is larger, the agent employs more cognition to achieve a greater reduction in uncertainty (the optimal target for $\lambda_{t}^{(i)}$ is lower). Intuitively, she values that reduction in uncertainty precisely because it allows her to select an action closer to the currently perceived greedy policy, not having to worry about experimentation.

Overall, the framework is well suited to apply to a range of concrete economic environments, some of which we actively investigate in our own work in progress. For example, a standard consumption-savings problem in a simple Aiyagari (1994) framework is a transparent and widely-studied setting in economics. Indeed, the consumption-saving decision is a fundamental mechanism in a number of different economic settings, and has been used as a laboratory for other recent bounded rationality and behavioral papers, such as for example Ilut and Valchev (2023) and Lian (2023). Due to its core elements, the bounded rationality framework proposed here appears promising in delivering endogenously empirically documented properties for consumption that are puzzling for standard fully-rational models: (a) experience effects of past income shocks on future consumption choices, (b) high sensitivity of consumption to income shocks; (c) large heterogeneity in consumption responses, through stochastic signals and choice.