Question

Find the optimal paths of the control, state, and costate variables that will $$\begin{array}{ll} \text { Maximize } & \int_{0}^{1}\left(y-u^{2}\right) d t \\ \text { subject to } & y^{\prime}=u \\ \text { and } & y(0)=2 \quad y(1) \text { free } \end{array}$$

Short answer

Optimal control: \( u = \frac{-t + 1}{2} \). Optimal state path: \( y(t) = \frac{-t^2}{4} + \frac{t}{2} + 2 \).

Step-by-step solution

Define the Hamiltonian

The Hamiltonian function combines the integrand with the constraints. Define the Hamiltonian for this problem as \( H = y - u^2 + u u \), where \( u \) is the costate variable.

Derive the Costate Equation

Differentiate the Hamiltonian with respect to \( y \) to obtain the costate equation. This gives \( u' = -\frac{\partial H}{\partial y} = -1 \). Thus, \( u(t) = -t + C_1 \) for some constant \( C_1 \).

Determine the Optimal Control

Find the control \( u \) that maximizes the Hamiltonian. Differentiate \( H \) with respect to \( u \) and set it to zero: \( \frac{\partial H}{\partial u} = -2u + u = 0 \). Thus, the optimal control is \( u = \frac{u}{2} \).

Solve the State Equation

The state equation is \( y' = u \). Substitute \( u = \frac{u}{2} \) and solve: \( y' = \frac{-t + C_1}{2} \). Integrate to find \( y(t) = \frac{-t^2}{4} + \frac{C_1 t}{2} + C_2 \).

Apply Boundary Conditions

Use the boundary condition \( y(0) = 2 \) to find \( C_2 \). Substitute to get \( 2 = \frac{0}{4} + \frac{C_1 \cdot 0}{2} + C_2 \), hence \( C_2 = 2 \).

Evaluate the Free Boundary Condition

To determine \( C_1 \), use the transversality condition since \( y(1) \) is free. At the endpoint \( t = 1 \), the Hamiltonian should not depend on \( u \), which implies that \( C_1 \) should satisfy the endpoint conditions of \( y(1) \).

Solve for Specific Constants

From the Hamiltonian condition and free boundary of \( y(1) \), solve for \( C_1 \) to complete the conditions for optimal paths. For the optimal solution \( u = -t + 1 \), substitute into control equation and solve for specifics if necessary.

Key concepts

Hamiltonian Function

Optimal control theory is a mathematical framework used to determine the best possible control laws for a given system. At the heart of this framework is the Hamiltonian Function, an essential tool for formulating and solving optimization problems.
In the context of our example, the Hamiltonian combines both the objective we wish to maximize and the dynamic constraints of the system. The function is designed to encapsulate the entire problem in a single expression:

• The integrand of the objective function, which is what we try to optimize, contributes directly to the Hamiltonian.
• The constraints of the system are incorporated into the Hamiltonian using a technique similar to the Lagrange multipliers in constrained optimization.

In our specific case, the Hamiltonian is given by:\[H = y - u^2 + u \cdot u\]This represents both the benefit of achieving higher values of the state \(y\), and the cost associated with the control effort \(u^2\). The optimal control is determined by finding values of \(u\) that maximize this Hamiltonian function.

Costate Equation

The Costate Equation is closely tied to the concept of optimal control and is crucial for solving dynamic optimization problems. It represents the rate of change of the costate variable, which can be thought of as a form of shadow price or Lagrange multiplier.
To derive the costate equation, we need to differentiate the Hamiltonian with respect to the state variable. In our example, the state variable is \(y\), and differentiating gives:

• \( u' = -\frac{\partial H}{\partial y} = -1 \)

This equation indicates that the costate \(u'\) decreases at a constant rate of \(-1\). Integrating this gives the costate trajectory:\[u(t) = -t + C_1\]where \(C_1\) is a constant determined by boundary conditions. The costate equation helps ensure that the chosen control leads to optimal paths by adjusting \(u(t)\) in response to changes in system dynamics.

Optimal Control

Finding the Optimal Control is a crucial step in solving an optimal control problem. This involves identifying the control variable \( u \) that maximizes the Hamiltonian function. Essentially, this step ensures that for each point in time, the chosen control is making the best possible impact on the system's performance.
To find the optimal control, we differentiate the Hamiltonian with respect to the control variable and set this derivative to zero:

• \( \frac{\partial H}{\partial u} = -2u + u = 0 \)

This simplifies to the optimality condition for the control:\[u = 2u\]From our earier calculations, replacing \( u = -t + C_1 \), gives:\[2u = -t + C_1\]The solution here is to continuously adjust \( u(t) \) such that this relationship holds at all times, ensuring system behavior remains on the optimal path.

State Equation

The State Equation describes how the state of the system evolves over time, influenced by the chosen control. It is an integral part of the optimal control framework, ensuring that the system dynamics are consistent with the chosen path.
Our state equation is relatively simple:

• \( y'(t) = u(t) \)

This says the rate of change of the state variable \( y(t) \) is directly driven by \( u(t) \), our control variable.
Upon substituting our expression for \( u(t) \), which was determined from the optimal control condition, we solve the state equation:\[y'(t) = \frac{-t + C_1}{2}\]
Integrating gives the path of the state variable:\[y(t) = \frac{-t^2}{4} + \frac{C_1 t}{2} + C_2\]Finally, applying the boundary condition \( y(0) = 2 \), we find that \( C_2 = 2 \). Together with the transversality condition, these help in fully determining the solution trajectory for the state, ensuring the system follows the optimal path from start to finish.