Heuristic Dynamic Programming (HDP)¶

HDP (Heuristic Dynamic Programming) is a model-based member of the Adaptive Critic Designs (ACD) family. Unlike model-free approaches (like DDPG or SAC), HDP leverages a known or learned linearized system model (matrices A, B) to perform one-step lookahead for actor improvement. The critic network learns the scalar cost-to-go function \( J(R) \), and the actor is optimized by backpropagating through the model to minimize expected future cost.

Key Ideas¶

Model-Based Critic: The critic \( J(R) \) estimates the cost-to-go as a function of the observable state \( R(t) = [x(t), \theta_{ref}(t), q_{ref}(t)] \)
One-Step Lookahead: The actor is improved by minimizing \( U(t) + \gamma J(R_{t+1}) \) where \( R_{t+1} \) is predicted using the linearized model
Temporal Difference Learning: The critic is trained via TD(0) on \( J(R_t) \approx U_t + \gamma J(R_{t+1}) \)
No Action Input to Critic: Unlike ADHDP or DDPG, HDP's critic does not take the action as input — it only depends on the state \( R \)

Architecture¶

Component	Role	Implementation
Actor \( \pi(R) \)	Generates control signal \( u(t) \)	`DeterministicActor` (MLP with tanh output)
Critic \( J(R) \)	Estimates scalar cost-to-go	`JCritic` (MLP → scalar)
Model \( (A, B) \)	Linearized dynamics for lookahead	Matrices from `env.model.filt_A`, `env.model.filt_B`

Algorithm¶

Training Loop¶

For each episode:
    Reset environment → x(0)
    For each step t:
        1. Construct R(t) = [x(t), θ_ref(t), q_ref(t)]
        2. Actor: u(t) = π(R(t)) [+ exploration noise]
        3. Execute u(t) in environment → x(t+1), U(t)
        4. Construct R(t+1) = [x(t+1), θ_ref(t+1), q_ref(t+1)]

        # Critic Update (TD Learning)
        5. J_target = U(t) + γ · J(R(t+1))   [bootstrap if not terminal]
        6. L_critic = MSE(J(R(t)), J_target)
        7. Update critic via gradient descent

        # Actor Update (Model-Based Lookahead)
        8. R'(t+1) = A · R(t) + B · π(R(t))   [model prediction]
        9. L_actor = U(t) + γ · J(R'(t+1))
        10. Update actor via gradient descent (through model & critic)

Mathematical Formulation¶

Critic Loss (TD Target):

\[ \mathcal{L}_{\text{critic}} = \mathbb{E}\left[ \left( J(R_t) - \left( U_t + \gamma J(R_{t+1}) \right) \right)^2 \right] \]

Actor Loss (One-Step Lookahead):

\[ \mathcal{L}_{\text{actor}} = \mathbb{E}\left[ U_t + \gamma J\left( A \cdot R_t + B \cdot \pi(R_t) \right) \right] \]

Where: - \( U_t \) is the immediate cost (negative reward) - \( \gamma \) is the discount factor - \( A, B \) are the linearized system matrices

Cost Function¶

The utility \( U(t) \) is typically a quadratic tracking cost:

\[ U(t) = w_\theta (\theta - \theta_{ref})^2 + w_q (q - q_{ref})^2 + w_u \|u\|^2 + w_{\Delta u} \|\Delta u\|^2 \]

Weight	Meaning
\( w_\theta \)	Pitch angle tracking penalty
\( w_q \)	Pitch rate tracking penalty
\( w_u \)	Control effort penalty
\( w_{\Delta u} \)	Control smoothness penalty

Quick Start¶

import numpy as np
from tensoraerospace.agent.hdp import HDP
from tensoraerospace.envs.b747 import ImprovedB747Env

def step_reference(steps: int, deg: float = 5.0) -> np.ndarray:
    """Generate step reference signal for pitch tracking."""
    ref = np.zeros((1, steps), dtype=np.float32)
    ref[:, steps // 5:] = np.deg2rad(deg)
    return ref

num_steps = 800

env = ImprovedB747Env(
    initial_state=np.array([0.0, 0.0, 0.0, 0.0], dtype=float),
    reference_signal=step_reference(num_steps, deg=5.0),
    number_time_steps=num_steps,
    dt=0.02,
)

agent = HDP(
    env,
    gamma=0.99,
    actor_lr=3e-4,
    critic_lr=3e-4,
    hidden_size=256,
    exploration_std=0.1,
    device="cpu",
    # Tracking cost weights
    dhp_w_theta=5.0,
    dhp_w_q=0.2,
    dhp_w_u=0.01,
    dhp_w_du=0.02,
    # Optional: use a PD baseline for stability
    dhp_use_baseline=False,
)

# Train the agent
agent.train(num_episodes=100, max_steps=num_steps)

# Save the trained model
agent.save("./hdp_b747_model")

Hyperparameters¶

Core Parameters¶

Parameter	Default	Description
`gamma`	0.99	Discount factor for future costs
`actor_lr`	3e-4	Actor network learning rate
`critic_lr`	3e-4	Critic network learning rate
`hidden_size`	256	Hidden layer size for both networks
`exploration_std`	0.1	Gaussian noise std for exploration

Tracking Cost Weights¶

Parameter	Default	Description
`dhp_w_theta`	5.0	Weight for pitch tracking error
`dhp_w_q`	0.2	Weight for pitch rate tracking error
`dhp_w_u`	0.01	Weight for control magnitude
`dhp_w_du`	0.02	Weight for control rate (smoothness)
`dhp_use_env_cost`	True	Use environment cost if available

Stabilization Options¶

Parameter	Default	Description
`dhp_use_baseline`	False	Use PD/PID baseline controller
`dhp_baseline_type`	"pd"	Baseline type: "pd" or "pid"
`dhp_baseline_kp`	0.6	Proportional gain
`dhp_baseline_kd`	0.2	Derivative gain
`dhp_residual_scale`	1.0	Scale of learned residual policy

Training Schedule¶

Parameter	Default	Description
`dhp_warmstart_actor_episodes`	0	Episodes to warmstart actor from baseline
`dhp_critic_cycle_episodes`	0	Episodes to train critic only (alternating)
`dhp_action_cycle_episodes`	0	Episodes to train actor only (alternating)

Comparison with Other ACD Designs¶

Design	Critic Output	Actor Improvement	Model Needed
HDP	\( J(R) \)	Model-based lookahead	Yes
DHP	\( \lambda = \partial J / \partial R \)	Direct gradient	Yes
GDHP	\( J(R), \lambda \)	Both J and gradients	Yes
ADHDP	\( J(R, a) \)	Critic gradients w.r.t action	No

When to Use HDP

Use HDP when you have access to a reasonably accurate linearized model of the plant. It typically converges faster than model-free methods for systems where the linear approximation holds well around the operating point.

Supported Environments¶

ImprovedB747Env — Boeing 747 longitudinal dynamics with tracking reference

Example: Step Response Tracking¶

The HDP agent can be trained to track step reference signals for pitch angle:

# Evaluate trained agent
obs, _ = env.reset()
done = False
theta_history = []

while not done:
    action = agent.select_action(obs, evaluate=True)
    obs, reward, terminated, truncated, info = env.step(action)
    done = terminated or truncated
    theta_history.append(obs[3])  # pitch angle

import matplotlib.pyplot as plt
plt.plot(theta_history, label='Actual θ')
plt.plot(env.reference_signal[0, :len(theta_history)], '--', label='Reference')
plt.xlabel('Time step')
plt.ylabel('Pitch angle (rad)')
plt.legend()
plt.title('HDP Pitch Tracking')
plt.show()

API Reference¶

HDP(env, *, gamma=0.99, actor_lr=0.0003, critic_lr=0.0003, hidden_size=256, device='cpu', seed=42, exploration_std=0.1, dhp_w_theta=5.0, dhp_w_q=0.2, dhp_w_u=0.01, dhp_w_du=0.02, dhp_use_env_cost=True, dhp_use_baseline=False, dhp_baseline_type='pd', dhp_baseline_kp=0.6, dhp_baseline_ki=0.0, dhp_baseline_kd=0.2, dhp_pid_use_normalized_theta=True, dhp_pid_mode='norm', dhp_residual_scale=1.0, dhp_warmstart_actor_episodes=0, dhp_warmstart_actor_epochs=2, dhp_warmstart_actor_disable_baseline_after=True, dhp_critic_cycle_episodes=0, dhp_action_cycle_episodes=0, log_dir=None, log_every_updates=100, **kwargs) ¶

Bases: ADP

Heuristic Dynamic Programming (HDP) agent — model-based Adaptive Critic Design.

HDP is a model-based reinforcement learning algorithm from the Adaptive Critic Designs (ACD) family. It uses a known linearized system model (matrices A, B) to perform one-step lookahead for actor improvement. The critic network learns a scalar cost-to-go function J(R), while the actor is optimized by backpropagating through the model to minimize expected future cost.

The algorithm follows the framework from Prokhorov & Wunsch (1997): - Critic learns: J(R_t) ≈ U_t + γ J(R_{t+1}) - Actor minimizes: U_t + γ J(A·R_t + B·π(R_t)) via model-based lookahead

Example

import numpy as np from tensoraerospace.agent.hdp import HDP from tensoraerospace.envs.b747 import ImprovedB747Env

def step_reference(steps, deg=5.0): ... ref = np.zeros((1, steps), dtype=np.float32) ... ref[:, steps // 5:] = np.deg2rad(deg) ... return ref

env = ImprovedB747Env( ... initial_state=np.array([0.0, 0.0, 0.0, 0.0]), ... reference_signal=step_reference(800, deg=5.0), ... number_time_steps=800, ... dt=0.02, ... ) agent = HDP(env, gamma=0.99, hidden_size=256) agent.train(num_episodes=100)

References

Prokhorov D.V., Wunsch D.C. "Adaptive Critic Designs." IEEE Trans. Neural Networks, vol. 8, no. 5, pp. 997-1007, 1997.
Werbos P.J. "Approximate dynamic programming for real-time control and neural modeling." Handbook of Intelligent Control, 1992.

Attributes:

Name	Type	Description
`actor`		Neural network that outputs control action π(R).
`critic`	`Module`	Neural network that estimates cost-to-go J(R).
`env`		The Gymnasium-compatible environment.
`gamma`		Discount factor for future costs.

Note

HDP requires an environment with: - env.model.filt_A, env.model.filt_B: linearized system matrices - env.reference_signal: pitch reference trajectory array

Initialize the HDP agent.

Parameters:

Name	Type	Description	Default
`env`	`Any`	Gymnasium-compatible environment. Must provide: - `env.model.filt_A`, `env.model.filt_B`: linearized dynamics matrices - `env.reference_signal`: reference trajectory for tracking - `env.observation_space`, `env.action_space`: Box spaces	required
`gamma`	`float`	Discount factor for future costs. Controls the trade-off between immediate and future costs. Range: [0, 1]. Default: 0.99.	`0.99`
`actor_lr`	`float`	Learning rate for the actor network optimizer (Adam). Default: 3e-4.	`0.0003`
`critic_lr`	`float`	Learning rate for the critic network optimizer (Adam). Default: 3e-4.	`0.0003`
`hidden_size`	`int`	Number of neurons in each hidden layer of both actor and critic networks. Both networks use two hidden layers with Tanh activation. Default: 256.	`256`
`device`	`Union[str, device]`	Torch device for computation ('cpu', 'cuda', 'mps', or torch.device instance). Default: 'cpu'.	`'cpu'`
`seed`	`int`	Random seed for reproducibility. Affects PyTorch, NumPy, and exploration noise. Default: 42.	`42`
`exploration_std`	`float`	Standard deviation of Gaussian noise added to actions during training for exploration. Set to 0 for deterministic training. Default: 0.1.	`0.1`
`dhp_w_theta`	`float`	Weight for pitch angle tracking error in the cost function. Higher values prioritize pitch tracking accuracy. Default: 5.0.	`5.0`
`dhp_w_q`	`float`	Weight for pitch rate tracking error in the cost function. Default: 0.2.	`0.2`
`dhp_w_u`	`float`	Weight for control magnitude penalty in the cost function. Penalizes large control inputs. Default: 0.01.	`0.01`
`dhp_w_du`	`float`	Weight for control rate (smoothness) penalty in the cost function. Penalizes rapid changes in control. Default: 0.02.	`0.02`
`dhp_use_env_cost`	`bool`	If True, use cost weights from the environment (e.g., ImprovedB747Env.w_pitch) when available. If False or unavailable, use dhp_w_* parameters. Default: True.	`True`
`dhp_use_baseline`	`bool`	If True, use a PD/PID baseline controller and train the actor as a residual policy: u = u_baseline + scale * π(R). Helps stabilize training in early stages. Default: False.	`False`
`dhp_baseline_type`	`str`	Type of baseline controller: 'pd' (proportional- derivative) or 'pid' (with integral term). Default: 'pd'.	`'pd'`
`dhp_baseline_kp`	`float`	Proportional gain for the baseline controller. Default: 0.6.	`0.6`
`dhp_baseline_ki`	`float`	Integral gain for the baseline PID controller. Only used if dhp_baseline_type='pid'. Default: 0.0.	`0.0`
`dhp_baseline_kd`	`float`	Derivative gain for the baseline controller. Default: 0.2.	`0.2`
`dhp_pid_use_normalized_theta`	`bool`	If True, normalize pitch angle by max_pitch_rad before passing to PID baseline. Default: True.	`True`
`dhp_pid_mode`	`str`	PID computation mode: 'norm' (normalized angles) or 'deg' (degrees). Default: 'norm'.	`'norm'`
`dhp_residual_scale`	`float`	Scaling factor for the learned residual policy when using baseline. Final action: u_baseline + scale * π(R). Default: 1.0.	`1.0`
`dhp_warmstart_actor_episodes`	`int`	Number of episodes to pre-train the actor by imitating the baseline controller via supervised learning. This initializes the actor as a stabilizing controller before ACD updates begin (paper recommendation). Default: 0 (disabled).	`0`
`dhp_warmstart_actor_epochs`	`int`	Number of supervised learning epochs per warm-start episode. Default: 2.	`2`
`dhp_warmstart_actor_disable_baseline_after`	`bool`	If True, disable the baseline (set dhp_use_baseline=False) after warm-start completes, so the actor takes full control. Default: True.	`True`
`dhp_critic_cycle_episodes`	`int`	Number of episodes to train only the critic (actor frozen) in each cycle. Part of the alternating training schedule from Prokhorov & Wunsch Section III. Set to 0 to disable alternating and train both networks every step. Default: 0.	`0`
`dhp_action_cycle_episodes`	`int`	Number of episodes to train only the actor (critic frozen) in each cycle. Works with dhp_critic_cycle_episodes for alternating training. Default: 0.	`0`
`log_dir`	`Union[str, Path, None]`	Directory path for TensorBoard logs. If None, logging is disabled. Default: None.	`None`
`log_every_updates`	`int`	Frequency of logging (every N gradient updates). Default: 100.	`100`
`**kwargs`	`Any`	Additional arguments passed to the base ADP class.	`{}`

Raises:

Type	Description
`ValueError`	If gamma is not in [0, 1].
`ValueError`	If exploration_std is negative.
`ValueError`	If environment lacks required attributes (filt_A, filt_B, reference_signal).

`save(path=None, *, save_gradients=False)` ¶

Save the HDP agent (actor, critic, config) to path.

Creates a timestamped subdirectory containing config.json, actor.pth, critic.pth (and optionally optimizer state dicts).

Parameters:

Name	Type	Description	Default
`path`	`Union[str, Path, None]`	Base directory. Defaults to the current working directory.	`None`
`save_gradients`	`bool`	Also persist optimizer states for resumed training.	`False`

Returns:

Name	Type	Description
`str`	`str`	Path to the created checkpoint directory.

`from_pretrained(repo_name, access_token=None, version=None, *, load_gradients=False)` `classmethod` ¶

Load a pretrained HDP agent from a local directory or Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Local folder path or Hugging Face repo id (`"namespace/repo"`).	required
`access_token`	`Optional[str]`	Hugging Face access token for private repos.	`None`
`version`	`Optional[str]`	Revision / branch / tag on Hugging Face.	`None`
`load_gradients`	`bool`	Restore optimizer states for continued training.	`False`

Returns:

Name	Type	Description
`HDP`	`'HDP'`	Fully initialized agent with restored weights.

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶

Upload a saved HDP checkpoint folder to Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Repository id (e.g. `"user/my-hdp-agent"`).	required
`folder_path`	`Union[str, Path]`	Local directory produced by :meth:`save`.	required
`access_token`	`Optional[str]`	Hugging Face access token.	`None`

`JCritic(input_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh)` ¶

Bases: Module

Critic approximating cost-to-go J(R) (HDP-style scalar critic).

`DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None)` ¶

Bases: Module

Deterministic actor with tanh output scaled to env action bounds.

References¶

Prokhorov D.V., Wunsch D.C. "Adaptive Critic Designs." IEEE Transactions on Neural Networks, vol. 8, no. 5, pp. 997-1007, 1997.
Werbos P.J. "Approximate dynamic programming for real-time control and neural modeling." Handbook of Intelligent Control, 1992.
Si J., et al. "Handbook of Learning and Approximate Dynamic Programming." Wiley-IEEE Press, 2004.

Heuristic Dynamic Programming (HDP)¶

Key Ideas¶

Architecture¶

Algorithm¶

Training Loop¶

Mathematical Formulation¶

Cost Function¶

Quick Start¶

Hyperparameters¶

Core Parameters¶

Tracking Cost Weights¶

Stabilization Options¶

Training Schedule¶

Comparison with Other ACD Designs¶

Supported Environments¶

Example: Step Response Tracking¶

API Reference¶

save(path=None, *, save_gradients=False) ¶

from_pretrained(repo_name, access_token=None, version=None, *, load_gradients=False) classmethod ¶

publish_to_hub(repo_name, folder_path, access_token=None) ¶

JCritic(input_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh) ¶

DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None) ¶

References¶

`save(path=None, *, save_gradients=False)` ¶

`from_pretrained(repo_name, access_token=None, version=None, *, load_gradients=False)` `classmethod` ¶

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶

`JCritic(input_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh)` ¶

`DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None)` ¶