Now consider adapting to the continuous case. Suppose and are both continuous, but discrete stages remain, and verify that (15.5) to (15.8) still hold true. Their present form can be used for any system that is approximated by discrete stages. Suppose that the discrete-time model of Section 14.2.2 is used to approximate a system on a state space that is a smooth manifold. In that model, was discretized to , but here it will be left in its original form. Let represent the time discretization.

The HJB equation will be obtained by approximating (15.6) with the discrete-time model and letting approach zero. The arguments here are very informal; see [95,570,912] for more details. Using discrete-time approximation, the dynamic programming recurrence is

in which is a discrete-time approximation to the cost that accumulates over stage and is given as

It is assumed that as approaches zero, the total discretized cost converges to the integrated cost of the continuous-time formulation.

Using the linear part of a Taylor series expansion about , the term can be approximated as

This approximates by its tangent plane at . Substitution of (15.11) and (15.10) into (15.9) yields

Subtracting from both sides of (15.12) yields

Taking the limit as approaches zero and then dividing by yields the

Compare the HJB equation to (15.6) for the discrete-time case. Both indicate how the cost changes when moving in the best direction. Substitution of for the optimal action into (15.14) yields

This is just the continuous-time version of (15.8). In the current setting, the left side indicates the derivative of the cost-to-go function along the direction obtained by applying the optimal action from .

The HJB equation, together with a boundary condition that specifies the final-stage cost, sufficiently characterizes the optimal solution to the planning problem. Since it is expressed over the whole state space, solutions to the HJB equation yield optimal feedback plans. Unfortunately, the HJB equation cannot be solved analytically in most settings. Therefore, numerical techniques, such as the value iteration method of Section 14.5, must be employed. There is, however, an important class of problems that can be directly solved using the HJB equation; see Section 15.2.2.

Steven M LaValle 2020-08-14