15.2.2 Linear-Quadratic Problems

This section briefly describes a problem for which the HJB equation can be directly solved to yield a closed-form expression, as opposed to an algorithm that computes numerical approximations. Suppose that a linear system is given by (13.37), which requires specifying the matrices $A$ and $B$. The task is to design a feedback plan that asymptotically stabilizes the system from any initial state. This is an infinite-horizon problem, and no termination action is applied.

An optimal solution is requested with respect to a cost functional based on matrix quadratic forms. Let $Q$ be a nonnegative definite^15.4 $n \times n$ matrix, and let $R$ be a positive definite $n \times n$ matrix. The quadratic cost functional is defined as

$$L({\tilde{x}},{\tilde{u}}) = \frac{1}{2} \int_0^\infty \big( x(t)^T Q x(t) + u(t)^T R u(t) \big)\, dt .$$ (15.20)

To guarantee that a solution exists that yields finite cost, several assumptions must be made on the matrices. The pair $(A,B)$ must be stabilizable, and $(A,Q)$ must be detectable; see [28] for specific conditions and a full derivation of the solution presented here.

Although it is not done here, the HJB equation can be used to derive the algebraic Riccati equation,

$$SA + A^T S - S B R^{-1} B^T S + Q = 0 ,$$ (15.21)

in which all matrices except $S$ were already given. Methods exist that solve for $S$, which is a unique solution in the space of nonnegative definite $n \times n$ matrices.

The linear vector field

$${\dot x}= \big( A - B R^{-1} B^T S \big) x$$ (15.22)

is asymptotically stable (the real parts of all eigenvalues of the matrix are negative). This vector field is obtained if $u$ is selected using a feedback plan $\pi$ defined as

$$\pi (x) = -R^{-1} B^T S x .$$ (15.23)

The feedback plan $\pi$ is in fact optimal, and the optimal cost-to-go is simply

$$G^*(x) = \begin{matrix}\frac{1}{2} \end{matrix} x^T S x .$$ (15.24)

Thus, for linear systems with quadratic cost, an elegant solution exists without resorting to numerical approximations. Unfortunately, the solution techniques do not generalize to nonlinear systems or linear systems among obstacles. Hence, the planning methods of Chapter 14 are justified.

However, many variations and extensions of the solutions given here do exist, but only for other problems that are expressed as linear systems with quadratic cost. In every case, some variant of Riccati equations is obtained by application of the HJB equation. Solutions to time-varying systems are derived in [28]. If there is Gaussian uncertainty in predictability, then the linear-quadratic Gaussian (LQG) problem is obtained [564]. Linear-quadratic problems and solutions even exist for differential games of the form (13.204) [59].