Finite-time analysis of entropy-regularized neural natural actor-critic algorithm

We elaborate some of our contributions below.

• •

  Sharp sample complexity, convergence rate and overparameterization bounds: We prove sharp convergence guarantees in terms of sample complexity, iteration complexity and network width. Particularly, We prove that the NAC method with an adaptive step-size achieves sharp $\tilde{O}(1/\epsilon)$ iteration complexity and $\tilde{O}(1/\epsilon^{5})$ sample complexity to achieve $\epsilon$ gap with the optimal policy of the regularized MDP under mildest distribution mismatch conditions to the best of our knowledge. The required network width for both the actor and critic are $\tilde{O}(1/\epsilon^{4})$ and $\tilde{O}(1/\epsilon^{2})$, respectively. Under the standard distribution mismatch assumption as in [22], our sample complexity bound for the unregularized MDP is $\tilde{O}(1/\epsilon^{6})$, which improves the existing bounds significantly.

• •

  Stable policy optimization: Existing works on neural policy gradient methods with neural network approximation assume that the policies perform sufficient exploration to avoid instability, i.e., convergence to near-deterministic and strictly suboptimal stationary policies. In this paper, we prove that policy optimization is stabilized by incorporating entropy regularization, gradient clipping and averaging. In particular, we show that the combination of these methods leads to “persistence of excitation” condition, which ensures sufficient exploration to avoid near-deterministic and strictly suboptimal stationary policies. Consequently, we prove convergence to the globally optimal policy under the mildest concentrability coefficient assumption for on-policy NAC to the best of our knowledge.

• •

  Understanding the dynamics of neural network approximation in policy optimization: Our analysis reveals that the uniform approximation power of the actor network to approximate Q-functions throughout policy optimization steps is crucial to ensure global (near-)optimality, which is a specific feature of reinforcement learning that induces a distributional shift over time in contrast to a static supervised learning problem. To that end, we establish high-probability bounds for a two-layer feedforward actor neural network to uniformly approximate Q-functions of the policy iterates during the training.

Policy gradient and actor-critic: Policy gradient methods use a gradient-based scheme to find the optimal policy [4, 5]. [12] proposed the natural gradient method, which uses the Fisher information matrix as a pre-conditioner to fit the problem geometry better. Actor-critic method, which learns approximations to both state-action value functions and policies for variance reduction, was introduced in [6].

Neural actor-critic methods: Recently, there has been a surge of interest in direct policy optimization methods for solving MDPs with large state spaces by exploiting the representation power of deep neural networks. Particularly, deterministic policy gradient [23], trust region policy optimization (TRPO) [24], proximal policy optimization (PPO) [25], soft actor-critic (SAC) [15, 26] achieved impressive empirical success in solving complicated control tasks.

Role of regularization: Entropy regularization is an essential part of policy optimization algorithms (e.g., TRPO, PPO and SAC) to encourage exploration and achieve fast and stable convergence. It has been numerically observed that entropy regularization leads to a smoother optimization landscape, which leads to improved convergence properties in policy optimization [16]. For tabular reinforcement learning, the impact of entropy regularization was studied in [17, 20, 21]. On the other hand, the function approximation regime leads to considerably different dynamics compared to the tabular setting mainly because of the generalization over a large state space, complex exploration dynamics and distributional shift. As such, the role of regularization is very different in the function approximation regime, which we study in this paper.

Theoretical analysis of policy optimization methods: Despite the vast literature on the practical performance of PG/AC/NAC type algorithms, their theoretical understanding has remained elusive until recently. In the tabular setting, global convergence rates for PG methods were established in [17, 18, 27]. By incorporating entropy regularization, it was shown in [28, 19, 20, 29, 30] that the convergence rate can be improved significantly in the tabular setting. Finite-time performances of off-policy actor-critic methods in the tabular and linear function approximation regimes were investigated in [31, 32]. In our paper, we consider neural network approximation under entropy regularization with on-policy sampling.

On the other hand, when the controller employs a function approximator for the purpose of generalization to a large state-action space, the convergence properties of policy optimization methods radically change due to more complicated optimization landscape and distribution mismatch phenomenon in reinforcement learning [17]. Under strong assumptions on the exploratory behavior of policies throughout learning iterations, global optimality of NPG with linear function approximation up to a function approximation error was established in [17]. For actor-critic and natural actor-critic methods with linear function approximation, there are finite-time analyses in [33, 34, 35]. For general actor schemes with linear critic, convergence to stationary points was investigated in [36, 37, 38].

By incorporating entropy regularization, it was shown that improved convergence rates under much weaker conditions on the underlying controlled Markov chain can be established in [39] with linear function approximation. Our paper uses results from the drift analysis in [39], but addresses the complications due to the nonlinearity introduced by ReLU activation functions, and establishes global convergence to the optimal policies. The neural network approximation eliminates the function approximation error, which is a constant in linear function approximation, by employing a sufficiently wide actor neural network.

Neural network analysis: The empirical success of neural networks, which have more parameters than the data points, has been theoretically explained in [40, 41, 42], where it was shown that overparameterized neural networks trained by using first-order optimization methods achieve good generalization properties. The need for massive overparameterization was addressed in [43, 44], and it was shown that considerably smaller network widths can suffice to achieve good training and generalization results in structured supervised learning problems. Our analysis in this work is mainly inspired by [43]. On the other hand, reinforcement learning problem has significantly different and more challenging dynamics than the supervised learning setting as we have a dynamic optimization problem in actor-critic, where distributional shift occurs as the policies are updated. As such, uniform approximation power of the actor network in approximating various functions through policy optimization steps becomes critical, different from the supervised learning setting in [45]. Our analysis utilizes tools from [45, 43]: (i) we consider max-norm geometry to achieve mild overparameterization, (ii) we bound the distance between the neural tangent kernel (NTK) function class and the class of functions realizable by a finite-width neural network by extending the ReLU analysis in [43].

The most relevant work in the literature is [22], where the convergence of NAC with a two-layer neural network was studied without entropy regularization. It was shown that, under strong assumptions on the exploratory behavior of policies throughout the trajectory, neural-NPG achieves $\epsilon$-optimality with $O(1/\epsilon^{14})$ sample complexity and $O(1/\epsilon^{12})$ network width bounds. In this paper, we incorporate widely-used algorithmic techniques (entropy regularization, averaging and gradient clipping) to NAC with neural network approximation, and prove significantly improved sample complexity and overparameterization bounds under weaker assumptions on the concentrability coefficients. Additionally, our analysis reveals that the uniform approximation power of the actor neural network is critically important to establish global optimality, where distributional shift plays a crucial role (see Section 5.4). In another relevant work, [46] considers a single-timescale actor-critic with neural network approximation, but the function approximation error was not investigated due to the realizability assumption, which assumes that all policies throughout the policy optimization steps are realizable by the neural network. On of the main goals of our work is to study the benefits of employing neural networks in policy optimization, and we explicitly characterize the function class and approximation error that stems from the use of finite-width neural networks.

For a sequence of numbers $\{x_{i}:i\in I\}$ where $I$ is an index set, $[x_{i}]_{i\in I}$ denotes the vector obtained by concatenation of $x_{i},i\in I$. For a set $A$, $|A|$ denotes its cardinality. For two distributions $P,Q$ defined over the same probability space, Kullback-Leibler divergence is denoted as follows: $D_{KL}(P\|Q)=\mathbb{E}_{s\sim P}\Big{[}\log\frac{P(s)}{Q(s)}\Big{]}.$ For a convex set $C\subset\mathbb{R}^{d}$ and $x\in\mathbb{R}^{d}$, $\mathcal{P}_{C}(x)$ denotes the projection of $x$ onto $C$: $\mathcal{P}_{C}(x)=\arg\min_{y\in C}\|x-y\|_{2}$. For $n\in\mathbb{Z}^{+}$, $[n]=\{1,2,\ldots,n\}$. For $d,m\in\mathbb{N},$ $R>0$ and $v\in\mathbb{R}^{m\times d},$ we denote

where $v_{i}$ denotes the $i^{\rm th}$ row of $v.$

We consider a discounted Markov decision process $(\mathcal{S},\mathcal{A},P,r,\gamma)$ where $\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, $P$ is a (unknown) transition kernel, $r:\mathcal{S}\times\mathcal{A}\rightarrow[0,r_{max}]$, $0<r_{max}<\infty$ is the reward function, and $\gamma\in(0,1)$ is the discount factor. In this work, we consider a state space $\mathcal{S}$ and a finite action space $\mathcal{A}$ such that $\mathcal{S}\times\mathcal{A}\subset\mathbb{R}^{d}.$ Also, we assume that, by appropriate representation of the state and action variables, the following bound holds:

throughout the paper.

Value function: A randomized policy $\pi:\mathcal{S}\rightarrow\mathcal{A}$ assigns some probability of taking an action $a\in\mathcal{A}$ at a given state $s\in\mathcal{S}$. A policy $\pi$ introduces a trajectory by specifying $a_{t}\sim\pi(\cdot|s_{t})$ and $s_{t+1}\sim P(\cdot|s_{t},a_{t})$. For any $s_{0}\in\mathcal{S}$, the corresponding value function of a policy $\pi$ is as follows:

where $a_{t}\sim\pi(\cdot|s_{t})$ and $s_{t+1}\sim P(\cdot|s_{t},a_{t})$.

Entropy regularization: In policy optimization, in order to encourage exploration and avoid near-deterministic suboptimal policies, entropy regularization is commonly used in practice [8, 15, 9, 16]. For a policy $\pi$, let

where $\mathcal{H}(\pi(\cdot|s))=-\sum_{a\in\mathcal{A}}\pi(a|s)\log\big{(}\pi(a|s)\big{)}$ is the entropy functional. Then, for $\lambda>0$, the entropy-regularized value function is defined as follows:

Note that $\pi_{\mathbf{0}}(\cdot|\cdot)=1/|\mathcal{A}|$ maximizes the regularizer $H^{\pi}(s_{0})$ for any $s_{0}\in\mathcal{S}$. Hence, the additional $\lambda H^{\pi}(s_{0})$ term in (4) encourages exploration while introducing some bias controlled by $\lambda>0$.

Entropy-regularized objective: For a given initial state distribution $\mu$ and for a given regularization parameter $\lambda>0$, the objective in this paper is to maximize the entropy-regularized value function:

We denote the optimal policy for the regularized MDP as $\pi^{*}$ throughout the paper.

Q-function and advantage function: The entropy-regularized (or soft) Q-function $q_{\lambda}^{\pi}(s,a)$ is defined as:

Note that $q_{\lambda}^{\pi}$ is the fixed point of the Bellman equation $q(s,a)=\mathcal{T}^{\pi}q(s,a)$ where the Bellman operator $\mathcal{T}^{\pi}$ is defined as:

As we will see, for NAC algorithms, the following function, called the soft Q-function under a policy $\pi,$ turns out to be a useful quantity [20]:

Note that the two Q-functions are related as follows:

The advantage function under a policy $\pi$ is defined as follows:

Similarly, the soft advantage function is defined as follows:

Lastly, we can bound the entropy-regularized value function as follows:

for any $\lambda>0,\pi$ since $r\in[0,r_{max}]$ and $\mathcal{H}(p)\leq\log|\mathcal{A}|$ for any distribution $p$ over $\mathcal{A}$ [20].

For a given randomized policy $\pi_{\theta}$ parameterized by $\theta\in\Theta$ where $\Theta$ is a given parameter space, policy gradient methods maximize $V^{{\pi_{\theta}}}(\mu)$ by using the policy gradient $\nabla_{\theta}V^{\pi_{\theta}}(\mu)$. Natural policy gradient, as a quasi-Newton method, adjusts the gradient update to fit problem geometry by using the Fisher information matrix as a pre-conditioner [12, 20].

Let

be the Fisher information matrix under policy $\pi_{\theta}$, where

is the discounted state visitation distribution under a policy $\pi$. Then, the update rule under NPG can be expressed as

where $\eta>0$ is the step-size. Equivalently, the NPG update can be written as follows:

The above update scheme is closely related to gradient ascent and policy mirror ascent. Note that the gradient ascent for policy optimization performs the following update:

The update in (13) leads to the policy gradient algorithm [4]. Compared to (13), the natural policy gradient uses a generalized Mahalanobis distance (i.e., weighted-$\ell_{2}$ distance) as the Bregman divergence instead of $\ell_{2}$ distance, which yields significant improvements in policy optimization by avoiding the so-called vanishing gradient problem in the tabular case [20, 19, 17]. Consequently, state-of-the-art reinforcement learning algorithms such as trust region policy optimization (TRPO) [24], proximal policy optimization (PPO) [25] and soft actor-critic [15] are variants of the natural policy gradient method.

In the following, we provide necessary tools to compute the policy gradient and the update rule in (11) based on [39].

Based on Proposition 14, the gradient update of natural policy gradient can be computed by the following lemma, which is an extension of [12, 17, 39].

The above results for general policy parameterization will provide basis for the entropy-regularized natural actor-critic (NAC) with neural network approximation that we will introduce in the following section, with certain modifications for variance reduction and stability that we will describe; see Remark 2 later.

For a network width $m\in\mathbb{Z}^{+}$ and $c_{i}\in\mathbb{R}$, $\theta_{i}\in\mathbb{R}^{d}$ for $i\in[m]$, the actor network is given by the single-hidden-layer neural network:

where $c=[c_{i}]_{i\in[m]},\theta=[\theta_{i}]_{i\in[m]}$, $\sigma(x)=\max\{0,x\}$ is the ReLU activation function. As a common practice [43, 44, 42], we fix the output layer $c$ after a random initialization, and only train the weights of hidden layer, namely, $\theta\in\Theta\subset\mathbb{R}^{m\times d}$. Given a (possibly random) parameter $\theta^{0}\in\mathbb{R}^{m\times d}$, a design parameter $R>0$, regularization parameter $\lambda>0$ and network width $m\in\mathbb{Z}^{+}$, the parameter space that we consider is as follows:

For this parameter space $\Theta$, the policy class that we consider is $\Pi=\{\pi_{\theta}:\theta\in\Theta\}$, where the policy that corresponds to $\theta\in\Theta$ is as follows:

We randomly initialize the actor neural network by using the symmetric initialization in Algorithm 1, $(c,\theta(0))\sim{\tt sym\_init}(m,d)$, which was introduced in [48].

Later, we will employ a similar symmetric initialization scheme for the critic neural network.

We denote the policy at iteration $t<T$ as $\pi_{t}=\pi_{\theta(t)}$ and neural network output as $f_{t}(s,a)=f(s,a;(c,\theta(t)))$. In the absence of $\Xi_{\lambda}^{\pi_{t}}$ and $d_{\mu}^{\pi_{t}}$, we estimate

by using samples by the following actor-critic method:

• •

  Critic: Temporal difference learning algorithm (Algorithm 3 in Section 3.2), which employs a critic neural network, returns a set of neural network weights that yield a sample-based estimate for the soft advantage function $\{\widehat{\Xi}_{\lambda}^{\pi_{t}}(s,a):(s,a)\in\mathcal{S}\times\mathcal{A}\}$.

• •

  Policy update: Given this, we approximate $u_{t}^{\star}$ by using stochastic gradient descent (SGD) with $N$ iterations and step-size $\alpha_{A}>0$. Starting with $u_{0}^{(t)}=0$, an iteration of SGD is as follows

  	$\displaystyle u_{n+1/2}^{(t)}$	$\displaystyle=u_{n}^{(t)}-\alpha_{A}\Big{(}\nabla_{\theta}^{\top}\log{\pi}_{t}(a_{n}|s_{n})u_{n}^{(t)}-\widehat{\Xi}_{\lambda}^{\pi_{t}}(s_{n},a_{n})\Big{)}\nabla_{\theta}\log{\pi}_{t}(a_{n}|s_{n}),$		(22)
  	$\displaystyle u_{n+1}^{(t)}$	$\displaystyle=\mathcal{P}_{\mathcal{B}_{m,R}^{d}(0)}\left(u_{n+1/2}^{(t)}\right),$		(23)

  where $s_{n}\sim d_{\mu}^{\pi_{t}}$ and $a_{n}\sim\pi_{t}(\cdot|s_{n})$ for $n=0,1,\ldots,N-1$, $\widehat{\Xi}^{\pi_{t}}_{\lambda}$ is the output of the critic. Then, the final estimate is $u_{t}=\frac{1}{N}\sum_{n=1}^{N}u_{n}^{(t)}$. By using $u_{t}$, we perform the following update:

  $$\theta(t+1)=\theta(t)+\eta_{t}\cdot w_{t},$$

  where $w_{t}=u_{t}-\lambda\big{(}\theta(t)-\theta(0)\big{)}$.

The natural actor-critic algorithm is summarized in Algorithm 2. Below, we summarize the modifications in the algorithm that we consider in this paper with respect to the NPG described in the previous section.

In the following subsection, we describe the critic algorithm in detail.

We estimate $\Xi_{\lambda}^{\pi_{t}}$ by using the neural TD learning algorithm with max-norm regularization [49]. Note that $\Xi_{\lambda}^{\pi_{t}}$ can be directly obtained from $q_{\lambda}^{\pi_{\theta}}$ via $Q_{\lambda}^{\pi_{\theta}}(s,a)=q_{\lambda}^{\pi_{\theta}}(s,a)-\lambda\log{\pi_{\theta}}(a|s)$ and (9). Since $q_{\lambda}^{\pi_{\theta}}$ is the fixed point of the Bellman equation (6), it can be approximated by using temporal difference (TD) learning algorithms.

For the critic, we use a two-layer neural network of width $m^{\prime}$, which is defined as follows:

The critic network is initialized according to the symmetric initialization scheme in Algorithm 1. Let $(b,W(0))$ denote the initialization.

We aim to solve the following problem:

where $\mathcal{T}^{\pi}$ is the Bellman operator in (2.1).

We will consider max-norm regularization in the updates of the critic, which was shown to be effective in supervised learning and reinforcement learning (see [49, 50]). For a given $w_{0}\in\mathbb{R}^{d}$ and $R>0$, let

Under max-norm regularization, each hidden unit’s weight vector is confined within the set $\mathcal{G}_{R}(W_{i}(0))$ for a given projection radius $R$.

For $k=0,1,\ldots,T^{\prime}-1$, we assume that $(s_{k},a_{k})$ is sampled from $d_{\mu}^{\pi_{\theta}}$, i.e., $s_{k}\sim d_{\mu}^{\pi_{\theta}},a_{k}\sim{\pi_{\theta}}(\cdot|s_{k})$. Upon obtaining $(s_{k},a_{k})$, the next state-action pair is obtained by following ${\pi_{\theta}}$: $s_{k}^{\prime}\sim P(\cdot|s_{k},a_{k})$, $a_{k}^{\prime}\sim{\pi_{\theta}}(\cdot|s_{k}^{\prime})$. One can replace the i.i.d. sampling here with Markovian sampling at the cost of a more complicated analysis as in [51]. However, since experience replay is used in practice, the actual sampling procedure is neither purely Markovian or i.i.d., and here for simplicity of the analysis, we choose to model it as i.i.d. sampling.

An iteration of MN-NTD is as follows:

where $\widehat{q}_{k}(s,a)=\widehat{q}(s,a;(b,W(k)))$, $r_{k}=r(s_{k},a_{k})-\lambda\log\pi_{\theta}(a_{k}|s_{k})$ and $\mathcal{P}_{\mathcal{C}}$ is the projection operator onto a set $\mathcal{C}\subset\mathbb{R}^{d}$. The output of the critic, which approximates $q_{\lambda}^{\pi_{\theta}}$, is then obtained as:

where $T^{\prime}$ is the number of iterations of MN-NTD. We obtain an approximation of the soft Q-function as

The corresponding estimate for the soft advantage function is the following:

The critic update for a given policy $\pi_{\theta},\theta\in\Theta$ is summarized in Algorithm 3.

The following proposition implies that the persistence of excitation condition (see [52] for a discussion of the critical role of persistence of excitation in stochastic control problems) is satisfied under Algorithm 2, which implies sufficient exploration to ensure convergence to global optimality.

Proposition 32 has two critical implications:

1. (i)

   The inequality in (29) implies that any action $a\in\mathcal{A}$ is explored with strictly positive probability at any given state $s\in\mathcal{S}$, which implies that all policies throughout the policy optimization steps satisfy the “persistence of excitation” condition with high probability over the random initialization. As we will see in the convergence analysis, this property implies sufficient exploration, which ensures that near-deterministic suboptimal policies are avoided. Sufficient exploration is achieved by entropy regularization, averaging, projection of $u_{t}$, and large network width $m$ for the policy parameterization.

2. (ii)

   The inequality (29) implies that we can control the deviation of the actor network weights by $R$, $\lambda$ and $m$. This property is key for the neural network analysis in the lazy-training regime.

We first present a brief discussion on kernel approximations of neural networks, which will be useful to state our convergence results. Consider the following space of mappings:

and the function class:

Note that $\mathcal{F}_{{\bar{\nu}}}$ is a provably rich subset of the reproducible kernel Hilbert space (RKHS) induced by the neural tangent kernel, which can approximate continuous functions over a compact space [43, 40, 53]. For a given class of transportation maps $\mathcal{V}=\{v_{k}\in\mathcal{H}_{\bar{\nu}}:k\in[K]\}$ for $K\geq 1$, we also consider the following subspace of $\mathcal{F}_{{\bar{\nu}}}$:

Note that the above set depends on the choice of $\{v_{k}\}_{k\in[K]}$ but these maps can be arbitrary, The space of continuously differentiable functions $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ over a compact domain has a countable basis $\{\varphi_{k}:k=0,1,\ldots\}$ [54]. By [43, Theorem 4.3], one can find transportation mappings $v_{k}\in\mathcal{H}_{{\bar{\nu}}}$ such that $g_{k}(s,a)=\mathbb{E}[\langle v_{k}(w_{0}),(s,a)^{\top}\cdot\mathbbm{1}\{(s,a)\cdot w_{0}\geq 0\}\rangle]$ approximates $\varphi_{k}$ well. As such, $\mathcal{F}_{K,{\bar{\nu}},\mathcal{V}}$ is able to approximate a function class which contains continuously differentiable functions over a compact space as $K\rightarrow\infty$ with appropriate $V$.

We make the following realizability assumption for the Q-function.

Assumption 2 is a smoothness condition on the class of realizable functions that can be approximated by the critic network, which is dense in the space of continuous functions over $\Omega_{d}$ (see Section 4.2). ${\bar{\nu}}$, which is an upper bound on the RKHS norm, is the measure of smoothness. One can also replace the above condition by a slightly stronger condition which states that $q_{\lambda}^{\pi_{\theta}}\in\mathcal{F}_{{\bar{\nu}}}$, $\forall\theta\in\Theta.$ Note that the class of functions $\mathcal{F}_{\bar{\nu}}$ is deterministic and its approximation properties are well-known [43]. In [22], it was assumed that the state-action value functions lie in a random function class, which is obtained by shifting $\mathcal{F}_{\bar{\nu}}$ with a Gaussian process. By employing a symmetric initialization, we eliminate this Gaussian process noise, and therefore the realizable class of functions is deterministic and provably rich.

Note that in order to achieve a target error less than $\varepsilon>0$, a network width of $m^{\prime}=\widetilde{O}\Big{(}\frac{{\bar{\nu}}^{4}}{\varepsilon^{2}}\Big{)}$ and iteration complexity $T^{\prime}=O\Big{(}\frac{(1+2{\bar{\nu}})^{2}{\bar{\nu}}^{2}}{\varepsilon^{4}}\Big{)}$ suffice. The analysis of TD learning algorithm in [49] uses results from [45], which was given for classification (supervised learning) problems with logistic loss. On the other hand, TD learning requires a significantly more challenging analysis because of bootstrapping in the updates (i.e., using a stochastic semi-gradient instead of a true gradient) and quadratic loss function. Furthermore, for improved sample complexity and overparameterization bounds, max-norm regularization is employed instead of early stopping [49].

In this section, we provide the main convergence result for the entropy-regularized NAC with neural network approximation.

Note that $\mathcal{F}_{K,{\bar{\nu}},\mathcal{V}}$ approximates a rich class of functions over a compact space well for large $K$ (see Section 4.2). Also, Assumption 3 implies that there is a structure among the soft Q-functions in the policy class $\Theta$ since each $Q_{\lambda}^{\pi_{\theta}}$ can be written as a linear combination of $K$ functions that correspond to the transportation maps $v_{k}$. We consider this relatively restricted function class instead of $\mathcal{F}_{{\bar{\nu}}}$ to obtain uniform approximation error bounds to handle the dynamic structure of the policy optimization over time steps. Notably, the actor network features are expected to fit $Q_{\lambda}^{\pi_{t}}$ over all iterations, thus an inherent structure in $\{Q_{\lambda}^{\pi_{\theta}}:\theta\in\Theta\}$ appears to be necessary. For further discussion, see Section 5.4 (particularly Remark 7).