p39

1.随机最优控制问题

考虑一个问题：An agent needs to make a decision on consumption and investment.
Time: $[0,T]$;
Risk-free interest rate: $r$;
Price of a risky asset: {St:t∈[0,T]}\{ S_t : t \in [0,T] \}{St​:t∈[0,T]},
$$dS(t)=\mu S(t)dt+\sigma S(t)dW(t)$$
Initial wealth: $X_0=x$
Wealth: {Xt:t∈[0,T]}\{ X_t : t \in [0,T] \}{Xt​:t∈[0,T]}
Consumption: {ct:t∈[0,T]}\{ c_t : t \in [0,T] \}{ct​:t∈[0,T]}（考虑投资+消费）
Portfolio weights: (1)risky asset ${\omega }_t$; (2)money account $1-{\omega }_t$
$$dX(t)=\frac{{\omega }_t{X}_t}{S_t}dS(t)+(1-{\omega }_t)X_{t}rdt-c_{t}dt$$

$=\left[r{X}_t-c_t+{\omega }_t{X}_t(\mu -r)\right]dt+{\omega }_t{X}_t\sigma dW(t)$

效用函数
The agent chooses consumption and investment to maximize
$$\mathbb{E}\left[{\int }_0^{T}U(t,c_t)dt+\Phi (X_T)\right]$$
注：这是经典的连续时间消费—投资问题（Merton problem）的标准形式。 1.经济学含义：效用来自“消费”，不是股票价格这个模型里： 股票价格St本身并不会直接给人带来快乐 真正给人带来满足的是： 消费（吃饭、住房、娱乐） 最终财富（遗产、退休资产等）。 2.Merton 模型的核心目标是：“如何在风险资产和消费之间最优分配财富”因此St已经进入财富动态

Phi_T是终端效用（terminal utility）即到最终时刻T，剩下的钱也有价值。

Utility maximization problem
max⁡{ct,ωt}t∈[0,T]E[∫0TU(t,ct) dt+Φ(XT)]s.t.dXt=[rXt−ct+ωtXt(μ−r)]dt+ωtXtσdW(t),X0=x,ct≥0,0≤ωt≤1. \begin{aligned} \max_{\{c_{t},\omega _{t}\}_{t\in \lbrack 0,T]}}\quad &\mathbb{E}\left[ \int_{0}^{T}U(t,c_{t})\,dt+\Phi (X_{T})\right] \\ \text{s.t.}\quad &dX_{t} = \left[ r X_t - c_t + \omega_t X_t (\mu -r) \right] dt +\omega_t X_t \sigma dW(t), \\ &X_{0}=x, \\ &c_{t}\geq 0, \\ &0\leq \omega _{t}\leq 1. \end{aligned} {ct​,ωt​}t∈[0,T]​max​s.t.​E[∫0T​U(t,ct​)dt+Φ(XT​)]dXt​=[rXt​−ct​+ωt​Xt​(μ−r)]dt+ωt​Xt​σdW(t),X0​=x,ct​≥0,0≤ωt​≤1.​
stochastic optimal control problem（一般化）
sup⁡{ut}t∈[0,T]∈UE[∫0TF(s,Xsu,us) ds+G(XTu)]s.t.dXtu=μ(t,Xsu,ut)dt+σ(t,Xtu,ut)dW(t),X0u=x. \begin{aligned} \sup_{\{u_{t}\}_{t\in \lbrack 0,T]}\in U}\quad &\mathbb{E}\left[ \int_{0}^{T}F(s,X_{s}^{u},u_{s})\,ds+G(X_{T}^{u})\right] \\ \text{s.t.}\quad &dX_{t}^{u}=\mu \left( t,X_{s}^{u},u_{t}\right) dt+\sigma \left( t,X_{t}^{u},u_{t}\right) dW(t), \\ &X_{0}^{u}=x. \end{aligned} {ut​}t∈[0,T]​∈Usup​s.t.​E[∫0T​F(s,Xsu​,us​)ds+G(XTu​)]dXtu​=μ(t,Xsu​,ut​)dt+σ(t,Xtu​,ut​)dW(t),X0u​=x.​
注记：

• max改为sup的原因是有可能取不到max。
• 这里{ut}t∈[0,T]\{u_{t}\}_{t\in [ 0,T]}{ut​}t∈[0,T]​可以是一个决策向量，如上包含了$c_t$和${\omega }_t$，control process。
• $U$是一个决策空间，控制约束。control constraints。
• $X_t^u$就是一个状态过程，如股票价格状态，依赖于$x$ and $u$，状态由控制决定。state process，depends on $x$ and $u$。
• $G$终点时候，有可能是增益，有可能是惩罚。terminal reward/penalty。
• $F$瞬间的收益或惩罚。running reward/penalty。

p40

2.贝尔曼最优条件

每个瞬间+最终收益
引入几个概念
1.performance criteria，表现，某一个过程对应的目标函数值是多少。
假设现在是$t$时刻，状态$x$，控制$u$。从$t$时刻往后延伸到$T$往后产生的收益。
$J(t,x,u)={\mathbb{E}}_{t,x}\left[{\int }_t^{T}F(s,X_s^u,u_s)ds+G(X_T^u)\right]$
注：与此同时，$J$是泛函（functional），输入是$u=(u_s{)}_{s\in [t,T]}$整一个控制过程，输出是一个数，即$u(\cdot )\to J(u)$，这就是函数的函数概念。
2.值函数 Value function，从$t$时刻往后延伸到$T$往后的收益加总。
V(t,x)=sup⁡{uτ}τ∈[t,T]∈UJ(t,x,u)V(t,x)=\sup_{\{u_{\tau}\}_{ \tau \in \lbrack t,T]}\in U} J(t,x,u)V(t,x)=sup{uτ​}τ∈[t,T]​∈U​J(t,x,u)

贝尔曼最优条件
Preliminary: Law of Iterated Expectations, LIE; Tower Property. $\mathbb{E}\left[Y\right]=\mathbb{E}\left[\mathbb{E}\left(Y|X\right)\right]$

Remark: The third equality cannot be written as $\mathbb{E}\left[{\int }_{\tau }^{T}F(s,X_s^u,u_s)ds+G(X_T^u)\right]$. The underlying reason in
stochastic control problems is as follows: $Y$ represents the future payoff $[\tau ,T]$. From the perspective of time $t$, it is a completely random
path. However, at time $\tau$, the system state is already deterministic
conditional on ${\mathcal{F}}_{\tau }$. (Accordingly, we first condense the
future random payoff into an ${\mathcal{F}}_{\tau }$-measurable expectation
observable at time $\tau$.) Therefore we first condense the future random
payoff into an expectation that is observable at time $\tau$.

$J(t,x,u)={\mathbb{E}}_{t,x}\left[{\int }_t^{T}F(s,X_s^u,u_s)ds+G(X_T^u)\right]$
首先拆分时间段$[t,T]$拆分为$[t,\tau ]$和$[\tau ,T]$
$={\mathbb{E}}_{t,x}\left[{\int }_t^{\tau }F(s,X_s^u,u_s)ds+{\int }_{\tau }^{T}F(s,X_s^u,u_s)ds+G(X_T^u)\right]$
这里为什么要加期望，和
1.从时刻$t$看，$\mathcal{F}$是整个未来完全随机，所以最外层套一个${\mathbb{E}}_{t,x}$没问题。2.${\mathcal{F}}_{\tau }$表示的$\tau$时刻以前的信息，但是$t->\tau$这事情确定的，所以写为$\mathbb{E}\left[{\int }_{\tau }^{T}Fds+G\mid {\mathcal{F}}_{\tau }\right]$。
$={\mathbb{E}}_{t,x}\left\{{\int }_t^{\tau }F(s,X_s^u,u_s)ds+\mathbb{E}\left[{\int }_{\tau }^{T}F(s,X_s^u,u_s)ds+G(X_T^u)\mid {\mathcal{F}}_{\tau }\right]\right\}$

$={\mathbb{E}}_{t,x}\left[{\int }_t^{\tau }F(s,X_s^u,u_s)ds+J(\tau ,X_{\tau }^u,u)\right]$

$\le {\mathbb{E}}_{t,x}\left[{\int }_t^{\tau }F(s,X_s^u,u_s)ds\right]+V(\tau ,X_{\tau }^u)$

≤sup⁡{us}s∈[0,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)]\leq \sup_{\{u_{s}\}_{s\in \lbrack 0,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right]≤sup{us​}s∈[0,T]​​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]

sup⁡{us}s∈[t,T]J(t,x,u)≤sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)]V(t,x)≤sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)] \begin{aligned} \sup_{\{u_{s}\}_{s\in \lbrack t,T]}}J(t,x,u) \leq &\sup_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right] \\ V(t,x) \leq &\sup_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right] \end{aligned} {us​}s∈[t,T]​sup​J(t,x,u)≤V(t,x)≤​{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]​
接下来需要证明$\ge$也成立。
我们定义一个$ϵ$-最优控制为$v^{ϵ}$。
Define $ϵ$-optimal control as $v^{ϵ}$ such that

Define $ϵ$-optimal control as $v^{ϵ}$ such that $V(t,x)\ge J(t,x,v^{ϵ})\ge V(t,x)-ϵ$. And then define a control ${\tilde{v}}^{ϵ}$ such that v~ϵ=1({t≤τ})ut,1({t>τ})vϵ\tilde{v}^{\epsilon }=\mathbf{1(\{}t\leq \tau \mathbf{\})}u_{t},\mathbf{1(\{}t>\tau \mathbf{\})}v^{\epsilon }v~ϵ=1({t≤τ})ut​,1({t>τ})vϵ.
abitrary control and $ϵ$-optimal control

Then we have
$$\begin{array}{c}V(t,x)\ge J(t,x,{\tilde{v}}^{ϵ})\ge {\mathbb{E}}_{t,x}\left[{\int }_t^{T}F(s,X_s^{{\tilde{v}}^{ϵ}},{\tilde{v}}^{ϵ})ds+G(X_T^{{\tilde{v}}^{ϵ}})\right]\end{array}$$
$$\begin{array}{c}={\mathbb{E}}_{t,x}\left[{\int }_t^{\tau }F(s,X_s^u,u_s)ds+{\int }_{\tau }^{T}F(s,X_s^{{\tilde{v}}^{ϵ}},v^{ϵ})ds+G(X_T^{{\tilde{v}}^{ϵ}})\right]\end{array}$$
$$\begin{array}{c}\ge {\mathbb{E}}_{t,x}\left[{\int }_t^{\tau }F(s,X_s^u,u_s)ds+V(\tau ,X_{\tau }^u)-ϵ\right]\end{array}$$
because abitrary between $t$ to $\tau$
≥sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)−ϵ] \begin{equation*} \geq \sup_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})-\epsilon % \right] \end{equation*} ≥{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)−ϵ]​

Let $ϵ\to 0$, then we have

V(t,x)≥sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)] \begin{equation*} V(t,x)\geq \sup_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right] \end{equation*} V(t,x)≥{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]​

{V(t,x)≤sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)]V(t,x)≥sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)] \begin{cases} V(t,x) \leq \sup\limits_{\{u_{s}\}_{s\in [t,T]}} \mathbb{E}_{t,x} \left[ \int_{t}^{\tau} F(s,X_{s}^{u},u_{s})\,ds + V(\tau,X_{\tau}^{u}) \right] \\[6pt] V(t,x) \geq \sup\limits_{\{u_{s}\}_{s\in [t,T]}} \mathbb{E}_{t,x} \left[ \int_{t}^{\tau} F(s,X_{s}^{u},u_{s})\,ds + V(\tau,X_{\tau}^{u}) \right] \end{cases} ⎩ ⎨ ⎧​V(t,x)≤{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]V(t,x)≥{us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]​

V(t,x)=sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)] \begin{equation*} V(t,x)=\sup_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right] \end{equation*} V(t,x)={us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]​

This is called the Bellman optimality condition.

意即决策一半时候，更新到最优策略，收益也可得到最优

注：从“$[t,T]$的最优价值”=先做“$t\to \tau$”这一小段决策得到即时收益+到“$\tau$之后”继续采用最优策略得到未来最优的价值。
如果一个策略在整体上最优，那么它从任意未来时刻开始看，后半段也必须是最优的。否则后半段还能改进，那整体就不是最优。

注2：在 $t$ 时刻、系统状态为 $x$ 时，从现在开始一直到终点 $T$，所能够获得的最大期望总收益。

3.HJB方程

method: 1.guess and verify 猜一个值函数，然后检验。2.值函数迭代
目前有了贝尔曼最优条件还不够，需要进一步处理，即HJB方程。

apply Ito’s formula to $V(\tau ,X_{\tau }^u)$ with arbitrary $u$
$$\begin{array}{c}d\left[V(\tau ,X_{\tau }^u)\right]=\frac{\partial V(t,X_{\tau }^u)}{\partial t}d\tau +\frac{\partial V(t,X_{\tau }^u)}{\partial X}d{X}_{\tau }^u+\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}(d{X}_{\tau }^u{)}^2\end{array}$$
因为$[t,T]$划分为$[t,\tau ]$和$(\tau ,T]$，并且各自对应的策略是“固定”和“最优”。而且第一段已经固定了，变化在于第二段。
$\frac{\partial V(t,X_{\tau }^u)}{\partial t}$表示value function 对“时间变量”的变化率。真正变化的是$\tau$。
$$\begin{array}{rl}=& \left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \\ & +\frac{\partial V(t,X_{\tau }^u)}{\partial X}\sigma (\tau ,X_{\tau }^u,u_{\tau })d{W}_{\tau }\end{array}$$

🔵 calculate

$$d{X}_t^u=\mu (t,X_s^u,u_t)dt+\sigma (t,X_t^u,u_t)dW(t)$$
simply, $d{X}_t^u=\mu \left(\cdot \right)dt+\sigma \left(\cdot \right)dW(t),(d{X}_t^u{)}^2={\sigma }^2\left(\cdot \right)d\tau$. $V_x[\mu \left(\cdot \right)dt+\sigma \left(\cdot \right)dW(t)]$

$$\begin{array}{rl}d\left[V(\tau ,X_{\tau }^u)\right]& =V_{t}d\tau +V_x\mu \left(\cdot \right)d\tau +V_x\sigma \left(\cdot \right)dW(t)+\frac{1}{2}{V}_{xx}{\sigma }^2\left(\cdot \right)d\tau \\ & =\left[V_t+V_x\mu \left(\cdot \right)+\frac{1}{2}{V}_{xx}{\sigma }^2\left(\cdot \right)\right]d\tau +V_x\sigma \left(\cdot \right)dW(t)\end{array}$$

Integrate both sides from $t\to t+h$
$$\begin{array}{rl}& V(t+h,X_{t+h}^u)-V(t,X_t^u)\\ =& {\int }_t^{t+h}\left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \\ & +{\int }_t^{t+h}\frac{\partial V(t,X_{\tau }^u)}{\partial X}\sigma (\tau ,X_{\tau }^u,u_{\tau })d{W}_{\tau }\end{array}$$

注：

在随机过程中，伊藤积分通常写作：
$$I_t={\int }_0^t{X}_{s}d{W}_s$$

其中 $W_s$ 是标准布朗运动（Brownian Motion）。布朗运动可以看作是无数个微小的、无方向的随机震荡。

2. “Martingale increment”（鞅增量）是什么意思？

• Martingale（鞅）：在概率论中，鞅代表一个“公平游戏”。这意味着如果你知道了直到今天为止的所有信息，你对明天财富的最佳预测就是你今天的财富。
• Increment（增量）：即这一小段时间内的变化量 $d{I}_t=X_{t}d{W}_t$。
• 结论：说伊藤积分是“鞅增量”，意味着这一小段随机积分的变化是不带“趋势”的，它完全是由随机扰动驱动的，没有偏向增加或减少的预设动力。

3. “条件期望为 0”意味着什么？

$$\mathbb{E}[X_{t}d{W}_t\mid {\mathcal{F}}_s]=0(\text{对于 }s<t)$$

这里的 ${\mathcal{F}}_s$ 代表直到 $s$ 时刻为止的所有已知信息。

• 直观理解：虽然我们不知道 $d{W}_t$ 具体会跳向哪里，但在已知当前所有信息的情况下，它向上跳和向下跳的概率是平衡的。

4. 这句话的实际用途

在推导伊藤引理（Itô’s Lemma）或者求解随机微分方程（SDE）时，这个性质非常强大：

1. 简化计算：当我们对一个随机微分方程两边取期望时，所有的伊藤积分项（鞅增量项）都会直接消失（变成 0）。
2. 提取趋势：这能帮助研究者把“确定性的趋势（Drift）”从“纯粹的随机波动（Diffusion）”中分离出来。

---

总结一下：
这句话的意思是，伊藤积分所代表的随机波动部分是纯粹的噪声，它不包含任何可以被提前预知的系统性偏差。 如果你在这一刻预测下一刻这段积分的变化，你的最优估计只能是 0。

Apply the expectation operator ${\mathbb{E}}_{t,x}$
$$\begin{array}{c}{\mathbb{E}}_{t,x}\left[V(t+h,X_{\tau +h}^u)\right]-V(t,X_t^u)\end{array}$$
$$\begin{array}{c}={\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}\left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \right]\end{array}$$

注：$V(t+h,X_{t+h}^u)-V(t,X_t^u)$，这里最主要是$V(t+h,X_{t+h}^u)$中$X_{t+h}^u$是随机变量。在$t$时刻的时候，时间和状态都是确定的。但是在未来时刻$t+h$时刻，未知。

$$\begin{array}{c}{\mathbb{E}}_{t,x}\left[V(t+h,X_{t+h}^u)\right]=V(t,X_t^u)+{\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}\left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \right]\end{array}$$
$$\begin{array}{c}V(t+h,X_{t+h}^u)=V(t,X_t^u)+{\int }_t^{t+h}\left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \end{array}$$

by Bellman optimality condition

V(t,x)=sup⁡{us}s∈[t,T]Et,x[∫tτF(s,Xsu,us) ds+V(τ,Xτu)] \begin{equation*} V(t,x)=\sup\limits_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{\tau }F(s,X_{s}^{u},u_{s})\,ds+V(\tau ,X_{\tau }^{u})\right] \end{equation*} V(t,x)={us​}s∈[t,T]​sup​Et,x​[∫tτ​F(s,Xsu​,us​)ds+V(τ,Xτu​)]​
这里我们把原来的 $\tau$ 改为 $t+h$，然后去掉$\sup$，但是注意变为大于等于。
V(t,x)=sup⁡{us}s∈[t,T]Et,x[∫tt+hF(s,Xsu,us) ds+V(t+h,Xt+hu)] \begin{equation*} V(t,x)=\sup\limits_{\{u_{s}\}_{s\in \lbrack t,T]}}\mathbb{E}_{t,x}\left[ \int_{t}^{t+h}F(s,X_{s}^{u},u_{s})\,ds+V(t+h,X_{t+h}^{u})\right] \end{equation*} V(t,x)={us​}s∈[t,T]​sup​Et,x​[∫tt+h​F(s,Xsu​,us​)ds+V(t+h,Xt+hu​)]​
$$\begin{array}{c}\ge {\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}F(s,X_s^u,u_s)ds+V(t+h,X_{t+h}^u)\right]\end{array}$$
$$\begin{array}{rl}V(t,x)\ge & {\mathbb{E}}_{t,x}[{\int }_t^{t+h}F(s,X_s^u,u_s)ds+V(t,x)+\\ & {\int }_t^{t+h}\left(\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \end{array}$$

$$\begin{array}{c}=V(t,X_t^u)+{\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}\left(F(\tau ,X_{\tau }^u,u_{\tau })d\tau +\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \right]\end{array}$$
then we have
$$\begin{array}{c}0\ge {\mathbb{E}}_{t,x}\left[{\int }_t^{t+h}\left(F(\tau ,X_{\tau }^u,u_{\tau })+\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \right]\end{array}$$

divide both sides by $h$
$$\begin{array}{c}0\ge {\mathbb{E}}_{t,x}\left[\frac{1}{h}{\int }_t^{t+h}\left(F(\tau ,X_{\tau }^u,u_{\tau })d\tau +\frac{\partial V(t,X_{\tau }^u)}{\partial t}+\frac{\partial V(t,X_{\tau }^u)}{\partial X}\mu (\tau ,X_{\tau }^u,u_{\tau })+\frac{1}{2}\frac{{\partial }^{2}V(t,X_{\tau }^u)}{\partial X^2}{\sigma }^2(\tau ,X_{\tau }^u,u_{\tau })\right)d\tau \right]\end{array}$$
let $h\to 0$, then
这里简单说明就是积分中值定理+连续性，最后令中指epsilon取t
补充：
积分中值定理：$\frac{1}{h}{\int }_t^{t+h}f(\tau )d\tau =f(\xi ),\xi \in [t,t+h]$。
所以简写一下，$0\ge \mathbb{E}\frac{1}{h}{\int }_t^{t+h}H(\tau ,X_{\tau }^u,u_{\tau })d\tau$，
$\frac{1}{h}{\int }_t^{t+h}H(\tau ,X_{\tau }^u,u_{\tau })d\tau =H({\xi }_h,X_{{\xi }_h}^u,u_{{\xi }_h})$
然后我们取值${\xi }_h=t$，$H(t,X_t^u,u_t)$，
${\mathbb{E}}_{t,x}[H(t,X_t^u,u_t)]\le 0$
这里${\mathbb{E}}_{t,x}$，也就是在$t$时刻，系统的状态时刻为$x$。也就是在$t$时刻，信息是已知的$\mathbb{E}(\cdot |X_t=x)$。
然后$H(t,X_t^u,u_t)$就是常数(已知)了，${\mathbb{E}}_{t,x}[H(t,X_t^u,u_t)]=H(t,X_t^u,u_t)$，然后把$H(t,X_t^u,u_t)$复原。

$$\begin{array}{c}0\ge F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial t}+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)\end{array}$$
the inequality becomes an equality when $u$ is optimal
$$\begin{array}{c}\underset{u_t}{\sup }F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial t}+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)\end{array}$$
$$\begin{array}{c}\frac{\partial V(t,X_t^u)}{\partial t}+\underset{u_t}{\sup }F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)=0\end{array}$$
这里是终端条件，终端收益
Also, notice that
$$\begin{array}{c}V(T,x)=G(x)\end{array}$$
then we have the following partial differential equation
$$\{\begin{array}{l}\frac{\partial V(t,X_t^u)}{\partial t}+\underset{u_t}{\sup }F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)=0\\ V(T,x)=G(x)\end{array}$$
This is called Hamiton-Jacobi-Bellman equation(HJB).

4.运用HJB方程求解最优消费投资问题

怎么写
How to solve a stochastic optimal control problem?
sup⁡{ut}t∈[0,T]∈UE[∫0TF(s,Xsu,us) ds+G(XTu)]s.t.dXtu=μ(t,Xtu,ut)dt+σ(t,Xtu,ut)dW(t),X0u=x. \begin{aligned} \sup_{\{u_{t}\}_{t\in \lbrack 0,T]}\in U}\quad &\mathbb{E}\left[ \int_{0}^{T}F(s,X_{s}^{u},u_{s})\,ds+G(X_{T}^{u})\right] \\ \text{s.t.}\quad &dX_{t}^{u}=\mu \left( t,X_{t}^{u},u_{t}\right) dt+\sigma \left( t,X_{t}^{u},u_{t}\right) dW(t), \\ &X_{0}^{u}=x. \end{aligned} {ut​}t∈[0,T]​∈Usup​s.t.​E[∫0T​F(s,Xsu​,us​)ds+G(XTu​)]dXtu​=μ(t,Xtu​,ut​)dt+σ(t,Xtu​,ut​)dW(t),X0u​=x.​
①write down HJB equation
$$\{\begin{array}{l}\frac{\partial V(t,X_t^u)}{\partial t}+\underset{u_t}{\sup }F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)=0\\ V(T,x)=G(x)\end{array}$$
②solve for $u^{\ast }$ in terms of $V$
$$\underset{u_t}{\sup }F(t,X_t^u,u_t)+\frac{\partial V(t,X_t^u)}{\partial X}\mu (t,X_t^u,u_t)+\frac{1}{2}\frac{{\partial }^{2}V(t,X_t^u)}{\partial X^2}{\sigma }^2(t,X_t^u,u_t)$$
③plug $u^{\ast }$ back to HJB equation, and then solve $V$。难
数值解，或验证