A Formal Solution to the Grain of Truth Problem

Let $\mathcal{X}$ denote a finite set called alphabet. The set $\mathcal{X}^{*}:=\bigcup_{n=0}^{\infty}\mathcal{X}^{n}$ is the set of all finite strings over the alphabet $\mathcal{X}$, the set $\mathcal{X}^{\infty}$ is the set of all infinite strings over the alphabet $\mathcal{X}$, and the set $\mathcal{X}^{\sharp}:=\mathcal{X}^{*}\cup\mathcal{X}^{\infty}$ is their union. The empty string is denoted by $\epsilon$, not to be confused with the small positive real number $\varepsilon$. Given a string $x\in\mathcal{X}^{\sharp}$, we denote its length by $|x|$. For a (finite or infinite) string $x$ of length $\geq k$, we denote with $x_{1:k}$ the first $k$ characters of $x$, and with $x_{<k}$ the first $k-1$ characters of $x$. The notation $x_{1:\infty}$ stresses that $x$ is an infinite string.

A function $f:\mathcal{X}^{*}\to\mathbb{R}$ is lower semicomputable iff the set $\{(x,p)\in\mathcal{X}^{*}\times\mathbb{Q}\mid f(x)>p\}$ is recursively enumerable. The function $f$ is computable iff both $f$ and $-f$ are lower semicomputable. Finally, the function $f$ is limit computable iff there is a computable function $\phi$ such that

$$\lim_{k\to\infty}\phi(x,k)=f(x).$$

The program $\phi$ that limit computes $f$ can be thought of as an anytime algorithm for $f$: we can stop $\phi$ at any time $k$ and get a preliminary answer. If the program $\phi$ ran long enough (which we do not know), this preliminary answer will be close to the correct one.

We use $\Delta\mathcal{Y}$ to denote the set of probability distributions over $\mathcal{Y}$. A list of notation can be found in Appendix A.

A semimeasure over the alphabet $\mathcal{X}$ is a function $\nu:\mathcal{X}^{*}\to[0,1]$ such that (i) $\nu(\epsilon)\leq 1$, and (ii) $\nu(x)\geq\sum_{a\in\mathcal{X}}\nu(xa)$for all $x\in\mathcal{X}^{*}$. In the terminology of measure theory, semimeasures are probability measures on the probability space $\mathcal{X}^{\sharp}=\mathcal{X}^{*}\cup X^{\infty}$ whose $\sigma$-algebra is generated by the cylinder sets $\Gamma_{x}:=\{xz\mid z\in\mathcal{X}^{\sharp}\}$ [LV08, Ch. 4.2]. We call a semimeasure (probability) a measure iff equalities hold in (i) and (ii) for all $x\in\mathcal{X}^{*}$.

Next, we connect semimeasures to Turing machines. The literature uses monotone Turing machines, which naturally correspond to lower semicomputable semimeasures [LV08, Sec. 4.5.2] that describe the distribution that arises when piping fair coin flips into the monotone machine. Here we take a different route.

A probabilistic Turing machine is a Turing machine that has access to an unlimited number of uniformly random coin flips. Let $\mathcal{T}$ denote the set of all probabilistic Turing machines that take some input in $\mathcal{X}^{*}$ and may query an oracle (formally defined below). We take a Turing machine $T\in\mathcal{T}$ to correspond to a semimeasure $\lambda_{T}$ where $\lambda_{T}(a\mid x)$ is the probability that $T$ outputs $a\in\mathcal{X}$ when given $x\in\mathcal{X}^{*}$ as input. The value of $\lambda_{T}(x)$ is then given by the chain rule

$$\lambda_{T}(x):=\prod_{k=1}^{|x|}\lambda_{T}(x_{k}\mid x_{<k}).$$

(1)

Thus $\mathcal{T}$ gives rise to the set of semimeasures $\mathcal{M}$ where the conditionals $\lambda(a\mid x)$ are lower semicomputable. In contrast, the literature typically considers semimeasures whose joint probability (1) is lower semicomputable. This set $\mathcal{M}$ contains all computable measures. However, $\mathcal{M}$ is a proper subset of the set of all lower semicomputable semimeasures because the product (1) is lower semicomputale, but there are some lower semicomputable semimeasures whose conditional is not lower semicomputable [LH15c, Thm. 6].

In the following we assume that our alphabet is binary, i.e., $\mathcal{X}:=\{0,1\}$.

Oracles are understood to be probabilistic: they randomly return $0$ or $1$. Let $T^{O}$ denote the machine $T\in\mathcal{T}$ when run with the oracle $O$, and let $\lambda_{T}^{O}$ denote the semimeasure induced by $T^{O}$. This means that drawing from $\lambda_{T}^{O}$ involves two sources of randomness: one from the distribution induced by the probabilistic Turing machine $T$ and one from the oracle’s answers.

The intended semantics of an oracle are that it takes a query $(T,x,p)$ and returns $1$ if the machine $T^{O}$ outputs $1$ on input $x$ with probability greater than $p$ when run with the oracle $O$, i.e., when $\lambda^{O}_{T}(1\mid x)>p$. Furthermore, the oracle returns $0$ if the machine $T^{O}$ outputs $1$ on input $x$ with probability less than $p$ when run with the oracle $O$, i.e., when $\lambda^{O}_{T}(1\mid x)<p$. To fulfill this, the oracle $O$ has to make statements about itself, since the machine $T$ from the query may again query $O$. Therefore we call oracles of this kind reflective oracles. This has to be defined very carefully to avoid the obvious diagonalization issues that are caused by programs that ask the oracle about themselves. We impose the following self-consistency constraint.

If $p$ under- or overshoots the true probability of $\lambda_{T}^{O}(\,\cdot\mid x)$, then the oracle must reveal this information. However, in the critical case when $p=\lambda_{T}^{O}(1\mid x)$, the oracle is allowed to return anything and may randomize its result. Furthermore, since $T$ might not output any symbol, it is possible that $\lambda_{T}^{O}(0\mid x)+\lambda_{T}^{O}(1\mid x)<1$. In this case the oracle can reassign the non-halting probability mass to $0$, $1$, or randomize; see Figure 1.

The following theorem establishes that reflective oracles exist.

For any probabilistic Turing machine $T\in\mathcal{T}$ we can complete the semimeasure $\lambda_{T}^{O}(\,\cdot\mid x)$ into a reflective-oracle-computable measure $\overline{\lambda}_{T}^{O}(\,\cdot\mid x)$: Using the oracle $O$ and a binary search on the parameter $p$ we search for the crossover point $p$ where $O(T,x,p)$ goes from returning $1$ to returning $0$. The limit point $p^{*}\in\mathbb{R}$ of the binary search is random since the oracle’s answers may be random. But the main point is that the expectation of $p^{*}$ exists, so $\overline{\lambda}_{T}^{O}(1\mid x)=\mathbb{E}[p^{*}]=1-\overline{\lambda}_{T}^{O}(0\mid x)$ for all $x\in\mathcal{X}^{*}$. Hence $\overline{\lambda}_{T}^{O}$ is a measure. Moreover, if the oracle is reflective, then $\overline{\lambda}_{T}^{O}(x)\geq\lambda_{T}^{O}(x)$ for all $x\in\mathcal{X}^{*}$. In this sense the oracle $O$ can be viewed as a way of ‘completing’ all semimeasures $\lambda_{T}^{O}$ to measures by arbitrarily assigning the non-halting probability mass. If the oracle $O$ is reflective this is consistent in the sense that Turing machines who run other Turing machines will be completed in the same way. This is especially important for a universal machine that runs all other Turing machines to induce a Solomonoff-style distribution.

The proof of Section 2.2 given in [FTC15a, App. B] is nonconstructive and uses the axiom of choice. In Section 2.4 we give a constructive proof for the existence of reflective oracles and show that there is one that is limit computable.

This theorem has the immediate consequence that reflective oracles cannot be used as halting oracles. At first, this result may seem surprising: according to the definition of reflective oracles, they make concrete statements about the output of probabilistic Turing machines. However, the fact that the oracles may randomize some of the time actually removes enough information such that halting can no longer be decided from the oracle output.

A similar argument can also be used to show that reflective oracles are not computable.

The idea for the proof of Section 2.3 is to construct an algorithm that outputs an infinite series of partial oracles converging to a reflective oracle in the limit.

The set of queries is countable, so we can assume that we have some computable enumeration of it:

$$\mathcal{T}\times\{0,1\}^{*}\times\mathbb{Q}=:\{q_{1},q_{2},\ldots\}$$

Let $k\in\mathbb{N}$, let $\tilde{O}$ be a $k$-partial oracle, and let $T\in\mathcal{T}$ be an oracle machine. The machine $T^{\tilde{O}}$ that we get when we run $T$ with the $k$-partial oracle $\tilde{O}$ is defined as follows (this is with slight abuse of notation since $k$ is taken to be understood implicitly).

1. 1.

   Run $T$ for at most $k$ steps.

2. 2.

   If $T$ calls the oracle on $q_{i}$ for $i\leq k$,

   1. (a)

      return $1$ with probability $\tilde{O}(q_{i})-2^{-k-1}$,

   2. (b)

      return $0$ with probability $1-\tilde{O}(q_{i})-2^{-k-1}$, and

   3. (c)

      halt otherwise.

3. 3.

   If $T$ calls the oracle on $q_{j}$ for $j>k$, halt.

Furthermore, we define $\lambda_{T}^{\tilde{O}}$ analogously to $\lambda_{T}^{O}$ as the distribution generated by the machine $T^{\tilde{O}}$.

It is important to note that we can check whether a $k$-partial oracle is $k$-partially reflective in finite time by running all machines $T$ from the first $k$ queries for $k$ steps and tallying up the probabilities to compute $\lambda_{T}^{\tilde{O}}$.

Section 2.4 only holds because we use semimeasures whose conditionals are lower semicomputable.

Now everything is in place to state the algorithm that constructs a reflective oracle in the limit. It recursively traverses a tree of partial oracles. The tree’s nodes are the partial oracles; level $k$ of the tree contains all $k$-partial oracles. There is an edge in the tree from the $k$-partial oracle $\tilde{O}_{k}$ to the $i$-partial oracle $\tilde{O}_{i}$ if and only if $i=k+1$ and $\tilde{O}_{i}$ extends $\tilde{O}_{k}$.

For every $k$, there are only finitely many $k$-partial oracles, since they are functions from finite sets to finite sets. In particular, there are exactly two $1$-partial oracles (so the search tree has two roots). Pick one of them to start with, and proceed recursively as follows. Given a $k$-partial oracle $\tilde{O}_{k}$, there are finitely many $(k+1)$-partial oracles that extend $\tilde{O}_{k}$ (finite branching of the tree). Pick one that is $(k+1)$-partially reflective (which can be checked in finite time). If there is no $(k+1)$-partially reflective extension, backtrack.

By Section 2.4 our search tree is infinitely deep and thus the tree search does not terminate. Moreover, it can backtrack to each level only a finite number of times because at each level there is only a finite number of possible extensions. Therefore the algorithm will produce an infinite sequence of partial oracles, each extending the previous. Because of finite backtracking, the output eventually stabilizes on a sequence of partial oracles $\tilde{O}_{1},\tilde{O}_{2},\ldots$. By the following lemma, this sequence converges to a reflective oracle, which concludes the proof of Section 2.3.

In reinforcement learning, an agent interacts with an environment in cycles: at time step $t$ the agent chooses an action $a_{t}\in\mathcal{A}$ and receives a percept $e_{t}=(o_{t},r_{t})\in\mathcal{E}$ consisting of an observation $o_{t}\in\mathcal{O}$ and a real-valued reward $r_{t}\in\mathbb{R}$; the cycle then repeats for $t+1$. A history is an element of $(\mathcal{A}\times\mathcal{E})^{*}$. In this section, we use $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}\in\mathcal{A}\times\mathcal{E}$ to denote one interaction cycle, and $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}$ to denote a history of length $t-1$.

We fix a discount function $\gamma:\mathbb{N}\to\mathbb{R}$ with $\gamma_{t}\geq 0$ and $\sum_{t=1}^{\infty}\gamma_{t}<\infty$. The goal in reinforcement learning is to maximize discounted rewards $\sum_{t=1}^{\infty}\gamma_{t}r_{t}$. The discount normalization factor is defined as $\Gamma_{t}:=\sum_{k=t}^{\infty}\gamma_{k}$. The effective horizon $H_{t}(\varepsilon)$ is a horizon that is long enough to encompass all but an $\varepsilon$ of the discount function’s mass:

$$H_{t}(\varepsilon):=\min\{k\mid\Gamma_{t+k}/\Gamma_{t}\leq\varepsilon\}$$

(2)

A policy is a function $\pi:(\mathcal{A}\times\mathcal{E})^{*}\to\Delta\mathcal{A}$ that maps a history $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}$ to a distribution over actions taken after seeing this history. The probability of taking action $a$ after history $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}$ is denoted with $\pi(a\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})$. An environment is a function $\nu:(\mathcal{A}\times\mathcal{E})^{*}\times\mathcal{A}\to\Delta\mathcal{E}$ where $\nu(e\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})$ denotes the probability of receiving the percept $e$ when taking the action $a_{t}$ after the history $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}$. Together, a policy $\pi$ and an environment $\nu$ give rise to a distribution $\nu^{\pi}$ over histories. Throughout this paper, we make the following assumptions.

We assumed that the discount function is summable, rewards are bounded (Section 3.1a), and actions and percepts spaces are both finite (Section 3.1b). Therefore an optimal deterministic policy exists for every environment [LH14, Thm. 10].

Fix $O$ to be a reflective oracle. From now on, we assume that the action space $\mathcal{A}:=\{\alpha,\beta\}$ is binary. We can treat computable measures over binary strings as environments: the environment $\nu$ corresponding to a probabilistic Turing machine $T\in\mathcal{T}$ is defined by

$$\nu(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t}):=\overline{\lambda}_{T}^{O}(y\mid x)=\prod_{i=1}^{k}\overline{\lambda}_{T}^{O}(y_{i}\mid xy_{1}\ldots y_{i-1})$$

where $y_{1:k}$ is a binary encoding of $e_{t}$ and $x$ is a binary encoding of $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t}$. The actions $a_{1:\infty}$ are only contextual, and not part of the environment distribution. We define

$$\nu(e_{<t}\mid a_{<t}):=\prod_{k=1}^{t-1}\nu(e_{k}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<k}).$$

Let $T_{1},T_{2},\ldots$ be an enumeration of all probabilistic Turing machines in $\mathcal{T}$. We define the class of reflective environments

$$\mathcal{M}_{\textrm{refl}}^{O}:=\left\{\overline{\lambda}_{T_{1}}^{O},\overline{\lambda}_{T_{2}}^{O},\ldots\right\}.$$

This is the class of all environments computable on a probabilistic Turing machine with reflective oracle $O$, that have been completed from semimeasures to measures using $O$.

Analogously to AIXI [Hut05], we define a Bayesian mixture over the class $\mathcal{M}_{\textrm{refl}}^{O}$. Let $w\in\Delta\mathcal{M}_{\textrm{refl}}^{O}$ be a lower semicomputable prior probability distribution on $\mathcal{M}_{\textrm{refl}}^{O}$. Possible choices for the prior include the Solomonoff prior $w\big{(}\overline{\lambda}_{T}^{O}\big{)}:=2^{-K(T)}$, where $K(T)$ denotes the length of the shortest input to some universal Turing machine that encodes $T$ [Sol78].²²2Technically, the lower semicomputable prior $2^{-K(T)}$ is only a semidistribution because it does not sum to $1$. This turns out to be unimportant. We define the corresponding Bayesian mixture

$$\xi(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t}):=\sum_{\nu\in\mathcal{M}_{\textrm{refl}}^{O}}w(\nu\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})\nu(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})$$

(3)

where $w(\nu\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})$ is the (renormalized) posterior,

$$w(\nu\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}):=w(\nu)\frac{\nu(e_{<t}\mid a_{<t})}{\overline{\xi}(e_{<t}\mid a_{<t})}.$$

(4)

The mixture $\xi$ is lower semicomputable on an oracle Turing machine because the posterior $w(\,\cdot\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})$ is lower semicomputable. Hence there is an oracle machine $T$ such that $\xi=\lambda_{T}^{O}$. We define its completion $\overline{\xi}:=\overline{\lambda}_{T}^{O}$ as the completion of $\lambda_{T}^{O}$. This is the distribution that is used to compute the posterior. There are no cyclic dependencies since $\overline{\xi}$ is called on the shorter history $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}$. We arrive at the following statement.

Moreover, since $O$ is reflective, we have that $\overline{\xi}$ dominates all environments $\nu\in\mathcal{M}_{\textrm{refl}}^{O}$:

	$\displaystyle\phantom{=}\leavevmode\nobreak\ \;\overline{\xi}(e_{1:t}\mid a_{1:t})$
	$\displaystyle=\overline{\xi}(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})\overline{\xi}(e_{<t}\mid a_{<t})$
	$\displaystyle\geq\xi(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})\overline{\xi}(e_{<t}\mid a_{<t})$
	$\displaystyle=\overline{\xi}(e_{<t}\mid a_{<t})\sum_{\nu\in\mathcal{M}_{\textrm{refl}}^{O}}w(\nu\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})\nu(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})$
	$\displaystyle=\overline{\xi}(e_{<t}\mid a_{<t})\sum_{\nu\in\mathcal{M}_{\textrm{refl}}^{O}}w(\nu)\frac{\nu(e_{<t}\mid a_{<t})}{\overline{\xi}(e_{<t}\mid a_{<t})}\nu(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})$
	$\displaystyle=\sum_{\nu\in\mathcal{M}_{\textrm{refl}}^{O}}w(\nu)\nu(e_{1:t}\mid a_{1:t})$
	$\displaystyle\geq w(\nu)\nu(e_{1:t}\mid a_{1:t})$

This property is crucial for on-policy value convergence.

This subsection is dedicated to the following result that was previously stated but not proved in [FST15, Alg. 6]. It contrasts results on arbitrary semicomputable environments where optimal policies are not limit computable [LH15b, Sec. 4].

Note that even though deterministic optimal policies always exist, those policies are typically not reflective-oracle-computable.

To prove Section 3.3 we need the following lemma.

Together, Section 3.2 and Section 3.3 provide the necessary ingredients to solve the grain of truth problem.

Hence the environment class $\mathcal{M}_{\textrm{refl}}^{O}$ contains any reflective-oracle-computable modification of the Bayes-optimal policy $\pi^{*}_{\overline{\xi}}$. In particular, this includes computable multi-agent environments that contain other Bayesian agents over the class $\mathcal{M}_{\textrm{refl}}^{O}$. So any Bayesian agent over the class $\mathcal{M}_{\textrm{refl}}^{O}$ has a grain of truth even though the environment may contain other Bayesian agents of equal power. We proceed to sketch the implications for multi-agent environments in the next section.

In a multi-agent environment there are $n$ agents each taking sequential actions from the finite action space $\mathcal{A}$. In each time step $t=1,2,\ldots$, the environment receives action $a_{t}^{i}$ from agent $i$ and outputs $n$ percepts $e_{t}^{1},\ldots,e_{t}^{n}\in\mathcal{E}$, one for each agent. Each percept $e_{t}^{i}=(o_{t}^{i},r_{t}^{i})$ contains an observation $o_{t}^{i}$ and a reward $r_{t}^{i}\in[0,1]$. Importantly, agent $i$ only sees its own action $a_{t}^{i}$ and its own percept $e_{t}^{i}$ (see Figure 2). We use the shorthand notation $a_{t}:=(a_{t}^{1},\ldots,a_{t}^{n})$ and $e_{t}:=(e_{t}^{1},\ldots,e_{t}^{n})$ and denote $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{i}=a_{1}^{i}e_{1}^{i}\ldots a_{t-1}^{i}e_{t-1}^{i}$ and $\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}=a_{1}e_{1}\ldots a_{t-1}e_{t-1}$.

We define a multi-agent environment as a function

$$\sigma:(\mathcal{A}^{n}\times\mathcal{E}^{n})^{*}\times\mathcal{A}^{n}\to\Delta(\mathcal{E}^{n}).$$

The agents are given by $n$ policies $\pi_{1},\ldots,\pi_{n}$ where $\pi_{i}:(\mathcal{A}\times\mathcal{E})^{*}\to\Delta\mathcal{A}$. Together they specify the history distribution

	$\displaystyle\sigma^{\pi_{1:n}}(\epsilon):$	$\displaystyle=1$
	$\displaystyle\sigma^{\pi_{1:n}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{1:t}):$	$\displaystyle=\sigma^{\pi_{1:n}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})\sigma(e_{t}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t})$
	$\displaystyle\sigma^{\pi_{1:n}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}a_{t}):$	$\displaystyle=\sigma^{\pi_{1:n}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t})\prod_{i=1}^{n}\pi_{i}(a_{t}^{i}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{i}).$

Each agent $i$ acts in a subjective environment $\sigma_{i}$ given by joining the multi-agent environment $\sigma$ with the policies $\pi_{1},\ldots,\pi_{i-1},\pi_{i+1},\ldots,\pi_{n}$ by marginalizing over the histories that $\pi_{i}$ does not see. Together with policy $\pi_{i}$, the environment $\sigma_{i}$ yields a distribution over the histories of agent $i$

$$\sigma_{i}^{\pi_{i}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{i}):=\sum_{\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{j},j\neq i}\sigma^{\pi_{1:n}}(\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}).$$

We get the definition of the subjective environment $\sigma_{i}$ with the identity $\sigma_{i}(e_{t}^{i}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{i}a_{t}^{i}):=\sigma_{i}^{\pi_{i}}(e_{t}^{i}\mid\mathchoice{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}{\mbox{{\ae }}}_{<t}^{i}a_{t}^{i})$. It is crucial to note that the subjective environment $\sigma_{i}$ and the policy $\pi_{i}$ are ordinary environments and policies, so we can use the formalism from Section 3.