15.2.1.3 Variants of the HJB equation

Several versions of the HJB equation exist. The one presented in (15.14) is suitable for planning problems such as those expressed in Chapter 14. If the cost-to-go functions are time-dependent, then the HJB equation is

$$\min_{u \in U(x)} \left\{ l(x,u,t) + \frac{\partial G^*}{\partial t} + \sum_{i = 1}^n \frac{\partial G^*}{\partial x_i} f_i(x,u,t) \right\} = 0 ,$$ (15.16)

and $G^*$ is a function of both $x$ and $t$. This can be derived again using a Taylor expansion, but with $x$ and $t$ treated as the variables. Most textbooks on optimal control theory present the HJB equation in this form or in a slightly different form by pulling $\partial G^*/\partial t$ outside of the $\min$ and moving it to the right of the equation:

$$\min_{u \in U(x)} \left\{ l(x,u,t) + \sum_{i = 1}^n \frac{\partial G^*}{\partial x_i} f_i(x,u,t) \right\} = -\frac{\partial G^*}{\partial t} .$$ (15.17)

In differential game theory, the HJB equation generalizes to the Hamilton-Jacobi-Isaacs (HJI) equations [59,477]. Suppose that the system is given as (13.203) and a zero-sum game is defined using a cost term of the form $l(x,u,v,t)$. The HJI equations characterize saddle equilibria and are given as

$$\min_{u \in U(x)} \max_{v \in V(x)} \left\{ l(x,u,v,t) + \frac{\p... ...t} + \sum_{i = 1}^n \frac{\partial G^*}{\partial x_i} f_i(x,u,v,t) \right\} = 0$$ (15.18)

and

$$\max_{v \in V(x)} \min_{u \in U(x)} \left\{ l(x,u,v,t) + \frac{\p... ... + \sum_{i = 1}^n \frac{\partial G^*}{\partial x_i} f_i(x,u,v,t) \right\} = 0 .$$ (15.19)

There are clear similarities between these equations and (15.16). Also, the swapping of the $\min$ and $\max$ operators resembles the definition of saddle points in Section 9.3.