Reinforcement Learning for optimal dividend problem under diffusion model

The problem of maximizing the cumulative discounted dividend payment can be traced back to the work of de Finetti [9]. Since then the problem has been widely studied in the literature under different models, and in many cases the problem can be explicitly solved when the model parameters are known. In particular, for optimal dividend problem and its many variations in continuous time under diffusion models, we refer to the works of, among others, [1, 3, 4, 5, 8, 21, 25] and the references cited therein. The main motivation of this paper is to study the optimal dividend problems in which the model parameters are not specified so the optimal control cannot be explicitly determined. Following the recently developed method using Reinforcement Learning (RL), we shall try to find the optimal strategy for a corresponding entropy-regularized control problem and solve it along the lines of both policy improvement and evaluation schemes.

The method of using reinforcement learning to solve discrete Markov decision problems has been well studied, but the extension of these concepts to the continuous time and space setting is still fairly new. Roughly speaking, in RL the learning agent uses a sequence of trial and errors to simultaneously identify the environment or the model parameters, and to determine the optimal control and optimal value function. Such a learning process has been characterized by a mixture of exploration and exploitation, which repeatedly tries the new actions to improve the collective outcomes. A critical point in this process is to balance the exploration and exploitation levels since the former is usually computationally expensive and time consuming, while the latter may lead to sub optimums. In RL theory a typical idea to balance the exploration and exploitation in an optimal control problem is to “randomize” the control action and add a (Shannon’s) entropy term in the cost function weighted by a temperature parameter. By maximizing the entropy one encourages the exploration and by decreasing the temperature parameter one gives more weight to exploitation. The resulting optimal control problem is often refer to as the entropy-regularized exploratory optimal control problem, which will be the starting point of our investigation.

As any reinforcement learning scheme, we shall solve the entropy-regularized exploratory optimal dividend problem via a sequence of Policy Evaluation (PE) and Policy Improvement (PI) procedures. The former evaluates the cost functional for a given policy, and the latter produces new policy that is “better” than the current one. We note that the idea of Policy Improvement Algorithms (PIA) as well as their convergence analysis is not new in the numerical optimal control literature (see, e.g., [13, 17, 16, 19]). The main difference in the current RL setting is the involvement of the entropy regularization, which causes some technical complications in the convergence analysis. In the continuous time entropy-regularized exploratory control problem with diffusion models a successful convergence analysis of PIA was first established for a particular Linear-Quadratic (LQ) case in [23], in which the exploratory HJB equation (i.e., HJB equation corresponding to the entropy regularized problem) can be directly solved, and the Gaussian nature of the optimal exploratory control is known. A more general case was recently investigated in [12], in which the convergence of PIA is proved in a general infinite horizon setting, without requiring the knowledge of the explicit form of the optimal control. The problem studied in this paper is very close to the one in [12], but not identical. While some of the analysis in this paper is benefitted from the fact that the spatial variable is one dimensional, but there are particular technical subtleties because of the presence of ruin time, although the problem is essentially an infinite horizon one, like the one studied in [12].

There are two main issues that this paper will focus on. The first is to design the PE and PI algorithms that are suitable for the continuous time optimal dividend problems. We shall follow some of the “popular” schemes in RL, such as the well-understood Temporal Difference (TD) methods, combined with the so-called martingale approach to design the PE learing procedure. Two technical points are worth noting: 1) since the cost functional involves ruin time, and the observation of the ruin time of the state process is sometimes practically impossible (especially in the cases where ruin time actually occurs beyond the time horizon we can practically observe), we shall propose algorithms that are insensitive to the ruin time; 2) although the infinite horizon nature of the problem somewhat prevents the use of the so-called “batch” learning method, we shall nevertheless try to study the temporally “truncated” problem so that the batch learning method can be applied. It should also be noted that one of the main difficulties in PE methods is to find an effective parameterization family of functions from which the best approximation for the cost functional is chosen, and the choice of the parameterization family directly affects the accuracy of the approximation. Since there are no proven standard methods of finding a suitable parameterization family, except for the LQ (Gaussian) case when the optimal value function is explicitly known, we shall use the classical “barrier”-type (restricted) optimal dividend strategy in [1] to propose the parametrization family, and carry out numerical experiments using the corresponding families.

The second main issue is the convergence analysis of the PIA. Similar to [12], in this paper we focus on the regularity analysis on the solution to the exploratory HJB equation and some related PDEs. Compared to the heavy PDE arguments as in [12], we shall take advantage of the fact that in this paper the state process is one dimensional taking nonnegative values, so that some stability arguments for 2-dimensional first-order nonlinear systems can be applied to conclude that the exploratory HJB equation has a concave, bounded classical solution, which would coincide with the viscosity solution (of class (L)) of HJB equation and the value function of the optimal dividend problem. With the help of these regularity results, we prove the convergence of PIA to the value function along the line of [12] but with a much simpler argument.

The rest of the paper is organized as follows. In §2 we give the preliminary description of the problem and all the necessary notations, definitions, and assumptions. In §3 we study the value function and its regularity, and prove that it is a concave, bounded classical solution to the exploratory HJB equation. In §4 we study the issue of policy update. We shall introduce our PIA and prove its convergence. In §5 and§6 we discuss the methods of Policy Evaluation, that is, the methods for approximating the cost functional for a given policy, using a martingale lost function based approach and (online) CTD($\gamma$) methods, respectively. In §7 we propose parametrization families for PE and present numerical experiments using the proposed PI and PE methods.

Throughout this paper we consider a filtered probability space $(\Omega,{\cal F},\{{\cal F}_{t}\}_{t\geq 0},\mathbb{P})$ on which is defined a standard Brownian motion $\{W_{t},t\geq 0\}$. We assume that the filtration $\mathbb{F}:=\{{\cal F}_{t}\}=\{{\cal F}^{W}_{t}\}$, with the usual augmentation so that it satisfies the usual conditions. For any metric space $\mathbb{X}$ with topological Borel sets $\mathscr{B}(\mathbb{X})$, we denote $\mathbb{L}^{0}(\mathbb{X})$ to be all $\mathscr{B}(\mathbb{X})$-measurable functions, and $\mathbb{L}^{p}(\mathbb{X})$, $p\geq 1$, to be the space of $p$-th integrable functions. The spaces $\mathbb{L}^{0}_{\mathbb{F}}([0,T];\mathbb{R})$ and $\mathbb{L}^{p}_{\mathbb{F}}([0,T];\mathbb{R})$, $p\geq 1$, etc., are defined in the usual ways. Furthermore, for a given domain $\mathbb{D}\subset\mathbb{R}$, we denote $\mathbb{C}^{k}(\mathbb{D})$ to be the space of all $k$-th order continuously differentiable functions on $\mathbb{D}$, and $\mathbb{C}(\mathbb{D})=\mathbb{C}^{0}(\mathbb{D})$. In particular, for $\mathbb{R}_{+}:=[0,\infty)$, we denote $\mathbb{C}^{k}_{b}(\mathbb{R}_{+})$ to be the space of all bounded and $k$-th continuously differentiable functions on $\mathbb{R}_{+}$ with all derivatives being bounded.

Consider the following simplest diffusion approximation of a Cramer-Lundberg model with dividend:

$\displaystyle dX_{t}=(\mu-\alpha_{t})dt+\sigma dW_{t},\quad t>0,\quad X_{0}=x\in\mathbb{R},$

(2.1)

where $x$ is the initial state, $\mu$ and $\sigma$ are constants determined by the premium rate and the claim frequency and size (cf., e.g., [1]), and $\alpha_{t}$ is the dividend rate at time $t\geq 0$. We denote $X=X^{\alpha}$ if necessary, and say that $\alpha=\{\alpha_{t},t\geq 0\}$ is admissible if it is $\mathbb{F}$-adapted and takes values in a given “action space” $[0,a]$. Furthermore, let us define the ruin time to be $\tau^{\alpha}_{x}:=\inf\{t>0:X^{\alpha}_{t}<0\}$. Clearly, $X^{\alpha}_{\tau^{\alpha}}=0$, and the problem is considered “ruined” as no dividend will be paid after $\tau^{\alpha}$. Our aim is to maximize the expected total discounted dividend given the initial condition $X^{\alpha}_{0}=x\in\mathbb{R}$:

$\displaystyle\qquad V(x):=\sup_{\alpha\in\mathscr{U}[0,a]}\mathbb{E}_{x}\Big{[}\int_{0}^{\tau_{x}^{\alpha}}e^{-ct}\alpha_{t}dt\Big{]}:=\sup_{\alpha\in\mathscr{U}[0,a]}\mathbb{E}\Big{[}\int_{0}^{\tau^{\alpha}_{x}}e^{-ct}\alpha_{t}dt\Big{|}x_{0}=x\Big{]},$

(2.2)

where $c>0$ is the discount rate, and $\mathscr{U}[0,a]$ is the set of admissible dividend rates taking values in $[0,a]$. The problem (2.1)-(2.2) is often referred to as the classical optimal restricted dividend problem, meaning that the dividend rate is restricted in a given interval $[0,a]$.

It is well-understood that when the parameters $\mu$ and $\sigma$ are known, then the optimal control is of the “feedback” form: $\alpha_{t}^{*}=\boldsymbol{a}^{*}(X_{t}^{*})$, where $X_{t}^{*}$ is the corresponding state process and $\boldsymbol{a}^{*}(\cdot)$ is a deterministic function taking values in $[0,a]$, often in the form of a threshold control (see, e.g., [1]). However, in practice the exact form of $\boldsymbol{a}^{*}(\cdot)$ is not implementable since the model parameters are usually not known, thus the “parameter insensitive” method through Reinforcement Learning (RL) becomes much more desirable alternative, which we now elaborate.

In the RL formulation, the agent follows a process of exploration and exploitation via a sequence of trial and error evaluation. A key element is to randomize the control action as a probability distribution over $[0,a]$, similar to the notion of relaxed control in control theory, and the classical control is considered as a special point-mass (or Dirac $\delta$-measure) case. To make the idea more accurate mathematically, let us denote $\mathscr{B}([0,a])$ to be the Borel field on $[0,a]$, and $\mathscr{P}([0,a])$ to be the space of all probability measure on $([0,a],\mathscr{B}([0,a]))$, endowed with, say, the Wasserstein metric. A “relaxed control” is a randomized policy defined as a measure-valued progressively measurable process $(t,\omega)\mapsto{\pi}(\cdot;t,\omega)\in\mathscr{P}([0,a])$. Assuming that ${\pi}(\cdot;t,\omega)$ has a density, denoted by $\pi_{t}(\cdot,\omega)\in\mathbb{L}^{1}_{+}([0,a])\subset\mathbb{L}^{1}([0,a])$, $(t,\omega)\in[0,T]\times\Omega$, then we can write

$${\pi}(A;t,\omega)=\int_{A}\pi_{t}(w,\omega)dw,\qquad A\in\mathscr{B}([0,a]),\quad(t,\omega)\in[0,T]\times\Omega.$$

In what follows we shall often identify a relaxed control with its density process $\pi=\{\pi_{t},t\geq 0\}$. Now, for $t\in[0,T]$, we define a probability measure on $([0,a]\times\Omega,\mathscr{B}([0,a])\otimes{\cal F})$ as follows: for $A\in\mathscr{B}([0,a])$ and $B\in{\cal F}$,

$\displaystyle\mathbb{Q}_{t}(A\times B):=\int_{A}\int_{B}\pi(dw;t,\omega)\mathbb{P}(d\omega)=\int_{A}\int_{B}\pi_{t}(w,\omega)dw\mathbb{P}(d\omega).$

(2.3)

We call a function $A^{\pi}:[0,T]\times[0,a]\times\Omega\mapsto[0,a]$ the “canonical representation” of a relaxed control $\pi=\{\pi(\cdot,t,\cdot)\}_{t\geq 0}$, if $A^{\pi}_{t}(w,\omega)=w$. Then, for $t\geq 0$ we have

$\displaystyle\mathbb{E}^{\mathbb{Q}_{t}}[A^{\pi}_{t}]=\int_{\Omega}\int_{0}^{a}A^{\pi}_{t}(w,\omega)\pi(dw;t,\omega)\mathbb{P}(d\omega)=\mathbb{E}^{\mathbb{P}}\Big{[}\int_{0}^{a}w\pi_{t}(w)dw\Big{]}.$

(2.4)

We can now derive the exploratory dynamics of the state process $X$ along the lines of entropy-regularized relaxed stochastic control arguments (see, e.g., [22]). Roughly speaking, consider the discrete version of the dynamics (2.1): for small $\Delta t>0$,

$\displaystyle\Delta x_{t}:=x_{t+\Delta t}-x_{t}\approx(\mu-a_{t})\Delta t+\sigma(W_{t+\Delta t}-W_{t}),\qquad t\geq 0.$

(2.5)

Let $\{a_{t}^{i}\}_{i=1}^{N}$ and $\{(x^{i}_{t},W_{t}^{i})\}_{i=1}^{N}$ be $N$ independent samples of $(a_{t})$ under the distribution $\pi_{t}$, and the corresponding samples of $(X_{t}^{\pi},W_{t})$, respectively. Then, the law of large numbers and (2.4) imply that

$\displaystyle\qquad\sum_{i=1}^{N}\frac{\Delta x_{t}^{i}}{N}\approx\sum_{i=1}^{N}(\mu-a^{i}_{t})\frac{\Delta t}{N}\approx\mathbb{E}^{\mathbb{Q}_{t}}[\mu-A^{\pi}_{t}]\Delta t=\mathbb{E}^{\mathbb{P}}\Big{[}\mu\negthinspace-\negthinspace\int_{0}^{a}w\pi_{t}(w,\cdot)dw\Big{]}\Delta t,$

(2.6)

as $N\to\infty$. This, together with the fact $\frac{1}{N}\sum_{i=1}^{N}(\Delta x_{t}^{i})^{2}\approx\sigma^{2}\Delta t$, leads to the follow form of the exploratory version of the state dynamics:

$\displaystyle dX_{t}=\Big{(}\mu-\int_{0}^{a}w\pi_{t}(w,\cdot)dw\Big{)}dt+\sigma dW_{t},\quad X_{0}=x,$

(2.7)

where $\{\pi_{t}(w,\cdot)\}$ is the (density of) relaxed control process, and we shall often denote $X=X^{\pi,0,x}=X^{\pi,x}$ to specify its dependence on control $\pi$ and the initial state $x$.

To formulate the entropy-regularized optimal dividend problem, we first give a heuristic argument. Similar to (2.6), for $N$ large and $\Delta t$ small we should have

$\displaystyle\frac{1}{N}\sum_{i=1}^{N}e^{-ct}a_{t}^{i}{\bf 1}_{[t\leq\tau^{i}]}\Delta t\approx\mathbb{E}^{\mathbb{Q}_{t}}\Big{[}e^{-ct}A_{t}^{\pi}{\bf 1}_{[t\leq\tau^{\pi}_{x}]}\Delta t\Big{]}=\mathbb{E}^{\mathbb{P}}\Big{[}{\bf 1}_{[t\leq\tau^{\pi}_{x}]}e^{-ct}\int_{0}^{a}w\pi_{t}(w)dw\Delta t\Big{]}.$

Therefore, in light of [22] we shall define the entropy-regularized cost functional of the optimal expected dividend control problem under the relaxed control ${\pi}$ as

$\displaystyle J(x,{\pi)}=\mathbb{E}_{x}\Big{[}\int_{0}^{\tau_{x}^{{\pi}}}e^{-ct}{\cal H}_{\lambda}^{\pi}(t)dt\Big{]},$

(2.8)

where ${\cal H}_{\lambda}^{\pi}(t):=\int_{0}^{a}(w-\lambda\ln\pi_{t}(w))\pi_{t}(w)dw$, ${\tau^{\pi}_{x}}=\inf\{t>0:X_{t}^{\pi,x}<0\}$, and $\lambda>0$ is the so-called temperature parameter balancing the exploration and exploitation.

We now define the set of open-loop admissible controls as follows.

Consequently, the value function (2.2) now reads

$\displaystyle V(x)=\sup_{{\pi}\in\mathscr{A}(x)}\mathbb{E}_{x}\Big{\{}\int_{0}^{\tau_{x}^{\pi}}e^{-ct}{\cal H}_{\lambda}^{\pi}(t)dt\Big{\}},\qquad x\geq 0.$

(2.9)

An important type of $\pi\in\mathscr{A}(x)$ is of the “feedback” nature, that is, $\pi_{t}(w,\omega)=\boldsymbol{\pi}(w,X^{\boldsymbol{\pi},x}_{t}(\omega))$ for some deterministic function $\boldsymbol{\pi}$, where $X^{\boldsymbol{\pi},x}$ satisfies the SDE:

$\displaystyle dX_{t}=\Big{(}\mu-\int_{0}^{a}w{\boldsymbol{\pi}}(w,X_{t})dw\Big{)}dt+\sigma dW_{t},\quad t\geq 0;\quad X_{0}=x.$

(2.10)

The following properties of the value function is straightforward.

Proof. (1) Let $x\geq y$, and $\pi\in\mathscr{A}(y)$. Consider $\hat{\pi}_{t}(w,\omega):=\pi_{t}(w,\omega){\bf 1}_{\{t<\tau^{\pi}_{y}(\omega)\}}+\frac{e^{\frac{w}{\lambda}}}{\lambda(e^{\frac{a}{\lambda}}-1)}{\bf 1}_{\{t\geq\tau^{\pi}_{y}(\omega)\}}$, $(t,w,\omega)\in[0,\infty)\times[0,a]\times\Omega$. Then, it is readily seen that $J(x,\hat{\pi})\geq J(y,\pi)$, for $a>1$. Thus $V(x)\geq J(x,\hat{\pi})\geq V(y)$, proving (1), as $\pi\in\mathscr{A}(y)$ is arbitrary.

(2) By definition $\int_{0}^{a}w\pi_{t}(w)dw\leq a$ and $-\int_{0}^{a}\ln\pi_{t}(w))\pi_{t}(w)dw\leq\ln a$ by the well-known Kullback-Leibler divergence property. Thus, ${\cal H}_{\lambda}^{\pi}(t)\leq\lambda\ln a+a$, and then $V(x)\leq\frac{\lambda\ln a+a}{c}$. On the other hand, since $\hat{\boldsymbol{\pi}}(w,x)\equiv\frac{1}{a}$, $(x,w)\in\mathbb{R}_{+}\times[0,a]$, is admissible and $J(x,\hat{\boldsymbol{\pi}})\geq 0$ for $a>1$, the conclusion follows.  

We remark that in optimal dividend control problems it is often assumed that the maximal dividend rate is greater than the average return rate (that is, $a>2\mu$), and that the average return of a surplus process $X$, including the safety loading, is higher than the interest rate $c$. These, together with Proposition 2.3, lead to the following standing assumption that will be used throughout the paper.

In this section we study the value function of the relaxed control problem (2.9). We note that while most of the results are well-understood, some details still require justification, especially concerning the regularity, due particularly to the non-smoothness of the exit time $\tau_{x}$.

We begin by recalling the Bellman optimality principle (cf. e.g., [24]):

$\displaystyle V(x)$

$\displaystyle=$

$\displaystyle\sup_{{\pi}(\cdot)\in\mathscr{A}(x)}\mathbb{E}_{x}\Big{[}\int_{0}^{s\wedge\tau^{\pi}}e^{-ct}{\cal H}_{\lambda}^{\pi}(t)dt+e^{-c(s\wedge\tau^{\pi})}V(X_{s\wedge\tau^{\pi}}^{\pi})\Big{]},\quad s>0.$

Noting that $V(0)=0$, we can (formally) argue that $V$ satisfies the HJB equation:

$\displaystyle\qquad\quad\begin{cases}\displaystyle cv(x)\negthinspace=\negthinspace\sup_{{\pi}\in\mathbb{L}^{1}[0,a]}\int_{0}^{a}\negthinspace\Big{[}w\negthinspace-\negthinspace\lambda\ln\pi(w)+\frac{1}{2}\sigma^{2}v^{\prime\prime}(x)\negthinspace+\negthinspace(\mu-w)v^{\prime}(x)\Big{]}\pi(w)dw;\\ v(0)=0.\end{cases}$

(3.1)

Next, by an argument of Lagrange multiplier and the calculus of variations (see [10]), we can find the maximizer of the right hand side of (3.1) and obtain the optimal feedback control which has the following Gibbs form, assuming all derivatives exist:

$\displaystyle{\boldsymbol{\pi}}^{*}(w,x)=G(w,1-v^{\prime}(x)),$

(3.2)

where $G(w,y)=\frac{y}{\lambda[e^{\frac{a}{\lambda}y}-1]}\cdot e^{\frac{w}{\lambda}y}{\bf 1}_{\{y\neq 0\}}+\frac{1}{a}{\bf 1}_{\{y=0\}}$ for $y\in\mathbb{R}$. Plugging (3.2) into (3.1), we see that the HJB equation (3.1) becomes the following second order ODE:

$\displaystyle\frac{1}{2}\sigma^{2}v^{\prime\prime}(x)+f(v^{\prime}(x))-cv(x)=0,\quad x\geq 0;\qquad v(0)=0,$

(3.3)

where the function $f$ is defined by

$\displaystyle f(z):=\Big{\{}\mu z+\lambda\ln\Big{[}\frac{\lambda(e^{\frac{a}{\lambda}(1-z)}-1)}{1-z}\Big{]}\Big{\}}{\bf 1}_{\{z\neq 1\}}+[\mu+\lambda\ln a]{\bf 1}_{\{z=1\}}.$

(3.4)

The following result regarding the function $f$ is important in our discussion.

Proof. (1) Since the function $w(z)$ is an entire function and $w(z)>0$ for $z\neq 1$, $\ln[w(z)]$ is infinitely many times differentiable for all $z\neq 1$. On the other hand, since $w(1)=a>1$, by the continuity of $w(z)$, there exists $r>0$ such that $w(z)\in(\frac{a}{2},\frac{3a}{2})$, whenever $|z-1|<r$. Thus $\ln[w(z)]$ is infinitely many times differentiable for $|z-1|<r$ as well. Consequently, $\ln[w(z)]\in\mathbb{C}^{\infty}(\mathbb{R})$, whence $f(z)\in\mathbb{C}^{\infty}(\mathbb{R})$ by extension.

(2) The convexity of the function $f$ follows from a direct calculation of $f^{\prime\prime}(z)$ for $z\in\mathbb{R}$. Define $\tilde{w}(z):=\lambda\ln[w(z)]=f(z)-\mu z$. It is straightforward to show that $\tilde{w}(1)>0$, $\tilde{w}(1+\lambda)<0$, and $\tilde{w}^{\prime}(z)<0$. Thus $\tilde{w}(H)=0$ for some (unique) $H\in(1,1+\lambda)$, proving (2).  

We should note that (3.3) can be viewed as either a boundary value problem of an elliptic PDE with unbounded domain $[0,\infty)$ or a second order ODE defined on $[0,\infty)$. But in either case, there is missing information on boundary/initial conditions. Therefore the well-posedness of the classical solution is actually not trivial.

Let us first consider the equation (3.3) as an ODE defined on $[0,\infty)$. Since the value function is non-decreasing by Proposition 2.3, for the sake of argument let us first consider (3.3) as an ODE with initial condition $v(0)=0$ and $v^{\prime}(0)=\alpha>0$. By denoting $X_{1}(x)=v(x)$ and $X_{2}(x)=v^{\prime}(x)$, we see that (3.3) is equivalent to the following system of first order ODEs: for $x\in[0,\infty)$,

$\displaystyle\begin{cases}X_{1}^{\prime}=X_{2},\qquad\qquad\qquad\qquad&X_{1}(0)=v(0)=0;\\ X_{2}^{\prime}=\frac{2c}{\sigma^{2}}X_{1}-\frac{2}{\sigma^{2}}f(X_{2}),&X_{2}(0)=v^{\prime}(0).\end{cases}$

(3.5)

Here $f$ is an entire function. Let us define $\tilde{X}_{1}:=X_{1}-\frac{f(0)}{c}$, $X:=(\tilde{X}_{1},X_{2})^{T}$, $A:=\begin{bmatrix}0&1\\ \frac{2c}{\sigma^{2}}&-\frac{2}{\sigma^{2}}h^{\prime}(0)\end{bmatrix}$ and $q(X)=\begin{bmatrix}0\\ -\frac{2}{\sigma^{2}}k(X_{2})\end{bmatrix}$ where $h(y):=f(y)-f(0)=yh^{\prime}(0)+\sum_{n=2}^{\infty}\frac{h^{(n)}(0)y^{n}}{n!}=yh^{\prime}(0)+k(y)$. Then, $X$ satisfies the following system of ODEs:

$\displaystyle X^{\prime}=AX+q(X),\qquad X(0)=(-f(0)/c,v^{\prime}(0))^{T}.$

(3.6)

It is easy to check $A$ has eigenvalues $\lambda_{1,2}=\frac{-h^{\prime}(0)\mp\sqrt{2c\sigma^{2}+h^{\prime}(0)^{2}}}{\sigma^{2}}$, with $\lambda_{1}<0<\lambda_{2}$. Now, let $Y=PX$, where $P$ is such that $PAP^{-1}=\mbox{\rm diag}[\lambda_{1},\lambda_{2}]:=B$. Then $Y$ satisfies

$\displaystyle Y^{\prime}=BY+g(Y),\quad Y(0)=PX(0),$

(3.7)

where $g(Y)=Pq(P^{-1}Y)$. Since $\nabla_{Y}g(Y)$ exists and tends to 0 as $|Y|\to 0$, and $\lambda_{1}<0<\lambda_{2}$, we can follow the argument of [6, Theorem 13.4.3] to construct a solution $\tilde{\phi}$ to (3.7) such that $|\tilde{\phi}(x)|\leq Ce^{-\alpha x}$ for some constant $C>0$, so that $|\tilde{\phi}(x)|\to 0$, as $x\to\infty$. Consequently, the function $\phi(x):=P^{-1}\tilde{\phi}(x)$ is a solution to (3.6) satisfying $|\phi(x)|\to 0$, as $x\to\infty$. In other words, (3.5) has a solution such that $(X_{1}(x),X_{2}(x))\to(0+\frac{f(0)}{c},0)=(\frac{f(0)}{c},0)$ as $x\to\infty$. We summarize the discussion above as the following result.

Proof. Following the discussion proceeding the proposition we know that the classical solution $v$ to (3.3) satisfying (i) and (ii) exists. We need only check (iii).

To this end, we shall follow an argument of [20]. Let us first formally differentiate (3.3) to get $v^{\prime\prime\prime}(x)=\frac{2c}{\sigma^{2}}v^{\prime}(x)-\frac{2}{\sigma^{2}}f^{\prime}(v^{\prime}(x))v^{\prime\prime}(x)$, $x\in[0,\infty)$. Since $v\in\mathbb{C}^{2}_{b}([0,\infty))$, denoting $m(x):=v^{\prime}(x)$, we can write

$\displaystyle m^{\prime\prime}(x)=\frac{2c}{\sigma^{2}}m(x)-\frac{2}{\sigma^{2}}f^{\prime}(m(x))m^{\prime}(x)\qquad x\in[0,\infty).$

Now, noting Proposition 3.1, we define a change of variables such that for $x\in[0,\infty)$, $\varphi(x):=\int_{0}^{x}\exp\left[\int_{0}^{v}-\frac{2}{\sigma^{2}}f^{\prime}(m(w))dw\right]dv$, and denote $l(y)=m(\varphi^{-1}(y))$, $y\in(0,\infty)$. Since $\varphi(0)=0$, and $\varphi^{\prime}(0)=1$, we can define $\varphi^{-1}(0)=0$ as well. Then we see that,

$\displaystyle\quad\quad\quad l^{\prime\prime}(y)=[\varphi^{\prime}(\varphi^{-1}(y))]^{-2}\frac{2c}{\sigma^{2}}l(y),~{}y\in(0,\infty);\quad l(0)=m(0)=v^{\prime}(0)=\alpha>0.$

(3.8)

Since (3.8) is a homogeneous linear ODE, by uniqueness $l(0)=\alpha>0$ implies that $l(y)>0$, $y\geq 0$. That is, $m(x)=v^{\prime}(x)>0$, $x\geq 0$, and $v$ is (strictly) increasing.

Finally, from (3.8) we see that $l(y)>0$, $y\in[0,\infty)$ also implies that $l^{\prime\prime}(y)>0$, $y\in[0,\infty)$. Thus $l(\cdot)$ is convex on $[0,+\infty)$, and hence would be unbounded unless $l^{\prime}(y)\leq 0$ for all $y\in[0,\infty)$. This, together with the fact that $v(x)$ is a bounded and increasing function, shows that $l(\cdot)$ (i.e., $v^{\prime}(\cdot)$) can only be decreasing and convex, thus $v^{\prime\prime}(x)$ (i.e., $l^{\prime}(y)$) $\leq 0$, proving the concavity of $v$, whence the proposition.  

Viscosity Solution of (3.3). We note that Proposition 3.2 requires that $v^{\prime}(0)$ exists, which is not a priorily known. We now conside (3.3) as an elliptic PDE defined on $[0,\infty)$, and argue that it possesses a unique bounded viscosity solution. We will then identify its value $v^{\prime}(0)$ and argue that it must coincide with the classical solution identified in Proposition 3.2.

To begin with, let us first recall the notion of viscosity solution to (3.3). For $\mathbb{D}\subseteq\mathbb{R}$, we denote the set of all upper (resp. lower) semicontinuous function in $\mathbb{D}$ by USC$(\mathbb{D})$ (resp. LSC$(\mathbb{D})$).

We first show that both viscosity subsolution and viscosity supersolution to (3.3) exist. To see this, for $x\in[0,\infty)$, consider the following two functions:

$\displaystyle\underline{\psi}(x):=1-e^{-x},\quad\overline{\psi}(x):=\frac{A}{M}(1-e^{-M(x\wedge b)}),$

(3.9)

where $A$, $M$, $b>0$ are constants satisfying $M>2\mu/\sigma^{2}$ and the following constraints:

$\displaystyle\left\{\begin{array}[]{lll}\frac{1}{M}\Big{\{}\ln\Big{(}\frac{A}{A-M}\Big{)}\vee\ln\Big{(}\frac{A}{A-\frac{f(0)}{c}M}\Big{)}\Big{\}}<b<\frac{1}{M}\Big{\{}\ln\frac{A}{H}\wedge\ln\Big{(}\frac{\sigma^{2}}{2\mu}M\Big{)}\Big{\}};\vskip 6.0pt plus 2.0pt minus 2.0pt\\ A>\max\left\{M+H,~{}\frac{f(0)}{c}M+H,~{}\frac{\sigma^{2}M^{2}}{\sigma^{2}M-2\mu},~{}\frac{f(0)}{c}\cdot\frac{\sigma^{2}M^{2}}{\sigma^{2}M-2\mu}\right\}.\end{array}\right.$

(3.12)

Proof. We first show that $\underline{\psi}$ is a viscosity subsolution. To see this, note that $\underline{\psi}^{\prime}(x)=e^{-x}$ and $\underline{\psi}^{\prime\prime}(x)=-e^{-x}$ on $(0,+\infty)$. By Assumption 2.4, Proposition 3.1, and the fact $f^{\prime}(1)<0$, we have

			$\displaystyle\frac{1}{2}\sigma^{2}\underline{\psi}^{\prime\prime}(x)+f(\underline{\psi}^{\prime}(x))-c(\underline{\psi}(x))=-\frac{1}{2}\sigma^{2}e^{-x}+f(e^{-x})-c(1-e^{-x})$
		$\displaystyle\geq$	$\displaystyle-\frac{1}{2}\sigma^{2}e^{-x}+\mu+\lambda\ln a-c(1-e^{-x})=(c-\frac{1}{2}\sigma^{2})e^{-x}+\mu+\lambda\ln a-c>0.$

That is, $\underline{\psi}(x)$ is a viscosity subsolution of (3.3) on $[0,\infty)$.

To prove that $\overline{\psi}$ is a supersolution of (3.3), we take the following three steps:

(i) Note that $\overline{\psi}^{\prime}(x)=Ae^{-Mx}$ and $\overline{\psi}^{\prime\prime}(x)=-AMe^{-Mx}$ for all $x\in(0,b)$. Let $H$ be the abscissa value of intersection point of $f(x)$ and $k(x)$, then $H\in(1,1+\lambda)$, thanks to Proposition 3.1.

Since $A>H$ and $b<(1/M)\ln(A/H)$ (i.e. $Ae^{-Mb}>H$), we have $f(A)<\mu A$ and $f(Ae^{-Mb})<\mu Ae^{-Mb}$. Also since $M>(2\mu/\sigma^{2})$ and $b<(1/M)\ln(\sigma^{2}M/2\mu)$, we have $f(A)<\frac{1}{2}\sigma^{2}AMe^{-Mb}$ and $f(Ae^{-Mb})<\frac{1}{2}\sigma^{2}AMe^{-Mb}$.

By Assumption 2.4, $A>\max\left\{M,H\right\}$, $M>\frac{2\mu}{\sigma^{2}}$ and $b<\min\left\{\frac{\ln\left(\frac{A}{H}\right)}{M},\frac{\ln\left(\frac{\sigma^{2}}{2\mu}M\right)}{M}\right\}$, we have $-\frac{1}{2}\sigma^{2}AMe^{-Mx}+f(Ae^{-Mx})-c\cdot\frac{A}{M}(1-e^{-Mx})<0$ for $x\in(0,b)$. That is, $\overline{\psi}(x)$ is a viscosity supersolution of (3.3) on $[0,b)$.

(ii) Next, note that $A>\frac{f(0)M}{c}$ and $b>\frac{1}{M}\ln\big{(}\frac{A}{A-\frac{f(0)M}{c}}\big{)}$, we see that $f(0)-c\cdot\frac{A}{M}(1-e^{-Mb})<0$ for $x\in(b,\infty)$, and it follows that $\overline{\psi}(x)$ is a viscosity supersolution of (3.3) on $(b,\infty)$.