Action-Dependent Heuristic Dynamic Programming (ADHDP)¶

ADHDP (Action-Dependent Heuristic Dynamic Programming) is a model-free member of the Adaptive Critic Designs (ACD) family. Unlike HDP which requires a plant model, ADHDP learns an action-dependent cost-to-go function \( J(R, a) \) that directly takes both state and action as inputs. The actor is improved by minimizing this critic output via backpropagation through the critic network.

Key Ideas¶

Action-Dependent Critic: The critic \( J(R, a) \) estimates cost-to-go as a function of both observable state \( R(t) \) and action \( a(t) \)
Model-Free: No plant model (A, B matrices) required — the critic learns directly from environment transitions
Online TD Learning: Critic trained via TD(0) on \( J(R_t, a_t) \approx U_t + \gamma J(R_{t+1}, \pi(R_{t+1})) \)
Actor Improvement: Actor minimizes \( J(R, \pi(R)) \) by backpropagating gradients through the critic

Key Difference: HDP vs ADHDP¶

Aspect	HDP (Model-Based)	ADHDP (Model-Free)
Critic Input	\( J(R) \) — state only	\( J(R, a) \) — state and action
Actor Update	Model-based lookahead	Gradient through critic
Requires Model	Yes (A, B matrices)	No
Sample Efficiency	Higher (uses model)	Lower (learns from data)

Architecture¶

Component	Role	Implementation
Actor \( \pi(R) \)	Generates control signal \( u(t) \)	`DeterministicActor` (MLP with tanh output)
Critic \( J(R, a) \)	Estimates action-dependent cost-to-go	`QCritic` (MLP: concat[R, a] → scalar)

Algorithm¶

Training Loop¶

For each episode:
    Reset environment → R(0)
    For each step t:
        1. Actor: a(t) = pi(R(t)) [+ exploration noise]
        2. Execute a(t) in environment → R(t+1), U(t)

        # Critic Update (TD Learning)
        3. a'(t+1) = pi(R(t+1))  [next action from actor]
        4. J_target = U(t) + g * J(R(t+1), a'(t+1))
        5. L_critic = MSE(J(R(t), a(t)), J_target)
        6. Update critic via gradient descent

        # Actor Update
        7. a_pi = pi(R(t))
        8. L_actor = J(R(t), a_pi)  [minimize critic output]
        9. Update actor via gradient descent through critic

Mathematical Formulation¶

Critic Loss (TD Target):

\[ \mathcal{L}_{\text{critic}} = \mathbb{E}\left[ \left( J(R_t, a_t) - \left( U_t + \gamma J(R_{t+1}, \pi(R_{t+1})) \right) \right)^2 \right] \]

Actor Loss:

\[ \mathcal{L}_{\text{actor}} = \mathbb{E}\left[ J(R_t, \pi(R_t)) \right] \]

Where: - \( U_t \) is the immediate cost (negative reward) - \( \gamma \) is the discount factor - \( \pi(R) \) is the actor policy

Quick Start¶

import numpy as np
from tensoraerospace.agent import ADHDP
from tensoraerospace.envs.b747 import ImprovedB747Env

def sine_reference(steps: int, amp_deg: float = 2.0, freq_hz: float = 0.05, dt: float = 0.1):
    """Generate sine reference signal for pitch tracking."""
    t = np.arange(steps) * dt
    ref = np.deg2rad(amp_deg) * np.sin(2 * np.pi * freq_hz * t)
    return ref.reshape(1, -1).astype(np.float32)

num_steps = 300
dt = 0.1

env = ImprovedB747Env(
    initial_state=np.array([0.0, 0.0, 0.0, 0.0], dtype=float),
    reference_signal=sine_reference(num_steps, amp_deg=2.0, freq_hz=0.05, dt=dt),
    number_time_steps=num_steps,
    dt=dt,
    include_reference_in_obs=True,
)

agent = ADHDP(
    env,
    gamma=0.99,
    actor_lr=1e-4,
    critic_lr=1e-4,
    hidden_size=128,
    exploration_std=0.02,
    device="cpu",
    # Paper-strict mode: canonical ADHDP without residual baseline
    paper_strict=True,
)

# Train the agent
agent.train(num_episodes=200, max_steps=num_steps)

# Save the trained model
agent.save("./adhdp_b747_model")

Hyperparameters¶

Core Parameters¶

Parameter	Default	Description
`gamma`	0.99	Discount factor for future costs
`actor_lr`	1e-4	Actor network learning rate
`critic_lr`	1e-4	Critic network learning rate
`hidden_size`	256	Hidden layer size for both networks
`exploration_std`	0.02	Gaussian noise std for exploration
`device`	"cpu"	Torch device ('cpu', 'cuda', 'mps')

Policy Mode¶

Parameter	Default	Description
`paper_strict`	False	If True, use canonical ADHDP without baseline
`policy_mode`	"direct"	"direct" (pure actor) or "residual" (baseline + actor)
`residual_scale`	0.2	Scale of residual policy when using baseline

Action Selection¶

Parameter	Default	Description
`action_selection`	"actor"	"actor" (use actor network) or "critic_gradient" (HDPy-style optimization)
`action_grad_steps`	0	Gradient steps for critic-based action optimization
`action_grad_lr`	0.0	Learning rate for action optimization
`action_momentum`	0.0	Momentum for action smoothing: u = mu_prev + (1-m)u_new
`action_max_abs`	1.0	Maximum action magnitude (safety envelope)

Baseline Controller¶

Parameter	Default	Description
`baseline_type`	"pid"	Baseline type: "pd" or "pid"
`baseline_kp`	-24.6295	Proportional gain (tuned for B747)
`baseline_ki`	-0.2486	Integral gain
`baseline_kd`	-7.8179	Derivative gain
`pid_i_clip`	1.0	Anti-windup integral clipping

Training Schedule (Paper Section III)¶

Parameter	Default	Description
`baseline_warmup_episodes`	0	Episodes running only baseline for critic warmup
`critic_warmup_episodes`	0	Episodes with frozen actor (critic-only training)
`critic_cycle_episodes`	0	Episodes per critic-only cycle (alternating)
`action_cycle_episodes`	0	Episodes per actor-only cycle (alternating)
`warmstart_actor_episodes`	0	Episodes to imitate baseline (supervised warmstart)

Trajectory Randomization¶

Parameter	Default	Description
`initial_state_noise_std`	0.0	Noise std for initial state randomization
`reference_roll_steps`	0	Max random roll of reference signal
`reference_noise_std`	0.0	Noise std added to reference signal

Persistent Excitation

The paper (Section III) emphasizes persistent excitation for stable learning. Instead of relying heavily on action noise, use trajectory randomization (initial_state_noise_std, reference_roll_steps) to expose the agent to diverse conditions.

Stabilization Strategies¶

ADHDP offers several strategies to stabilize training:

1. Paper-Strict Mode¶

agent = ADHDP(env, paper_strict=True)

Canonical ADHDP: pure actor policy, no baseline mixing, no BC regularizer.

2. Residual Policy¶

agent = ADHDP(env, policy_mode="residual", residual_scale=0.2)

Actor learns a residual correction on top of PID baseline: u = u_pid + 0.2 * pi(R).

3. Warm-Start Actor¶

agent = ADHDP(env, warmstart_actor_episodes=10, warmstart_actor_epochs=2)

Pre-train actor to imitate baseline via supervised learning before ACD updates.

4. Alternating Training¶

agent = ADHDP(env, critic_cycle_episodes=5, action_cycle_episodes=5)

Train critic for 5 episodes (actor frozen), then actor for 5 episodes (critic frozen).

Comparison with Other Methods¶

Method	Critic	Model	Training
ADHDP	\( J(R, a) \)	Not needed	Online TD
HDP	\( J(R) \)	Required	Model-based lookahead
DHP	\( \lambda = \partial J / \partial R \)	Required	Gradient-based
DDPG	\( Q(s, a) \)	Not needed	Replay + target networks

ADHDP vs DDPG

ADHDP is the canonical, paper-style actor-critic without modern stabilization tricks (replay buffer, target networks). For better sample efficiency and stability in practice, consider DDPG or SAC. ADHDP is valuable for research and understanding the foundations of ACD.

Supported Environments¶

ImprovedB747Env — Boeing 747 longitudinal dynamics with tracking reference

Unified training interface¶

ADHDP implements the shared unified train() signature from BaseRLModel:

agent.train(num_episodes=200, max_steps=500)

ADHDP-specific options accepted via **kwargs:

show_progress (bool, legacy alias for verbose) — controls the tqdm progress bar.
progress_desc (str) — tqdm description label.

API Reference¶

ADHDP(env, *, paper_strict=False, gamma=0.99, actor_lr=0.0001, critic_lr=0.0001, hidden_size=256, device='cpu', seed=42, policy_mode='direct', residual_scale=0.2, use_baseline_in_critic_phases=False, action_momentum=0.0, action_max_abs=1.0, action_selection='actor', action_grad_steps=0, action_grad_lr=0.0, action_grad_step_clip=0.0, action_grad_u_l2=0.0, action_grad_du_l2=0.0, actor_update_mode='minimize_critic', actor_distill_coef=1.0, actor_distill_steps=10, actor_distill_lr=0.03, distill_execute_teacher=False, teacher_rollout_noise_std=0.0, exploration_std=0.02, critic_updates_per_step=1, actor_updates_per_step=1, initial_state_noise_std=0.0, reference_roll_steps=0, reference_noise_std=0.0, baseline_warmup_episodes=0, critic_warmup_episodes=0, critic_cycle_episodes=0, action_cycle_episodes=0, use_env_cost=True, warmstart_actor_episodes=0, warmstart_actor_epochs=2, baseline_type='pid', baseline_kp=-24.6295, baseline_ki=-0.2486, baseline_kd=-7.8179, pid_i_clip=1.0, actor_bc_l2=0.0, actor_bc_decay=1.0, log_dir=None, log_every_updates=500, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None) ¶

Bases: BaseRLModel

Action-Dependent Heuristic Dynamic Programming (ADHDP) agent.

ADHDP is a model-free reinforcement learning algorithm from the Adaptive Critic Designs (ACD) family. Unlike HDP which requires a plant model, ADHDP learns an action-dependent cost-to-go function J(R, a) that directly takes both state and action as inputs. The actor is improved by minimizing this critic output via backpropagation through the critic network.

The algorithm follows the framework from Prokhorov & Wunsch (1997): - Critic learns: J(R_t, a_t) ≈ U_t + γ J(R_{t+1}, π(R_{t+1})) - Actor minimizes: J(R_t, π(R_t)) via gradient through critic

This is the canonical, paper-style actor-critic without modern stabilization tricks (replay buffer, target networks). It uses online TD(0) learning.

Example

import numpy as np from tensoraerospace.agent import ADHDP from tensoraerospace.envs.b747 import ImprovedB747Env

def sine_ref(steps, amp_deg=2.0, freq_hz=0.05, dt=0.1): ... t = np.arange(steps) * dt ... return (np.deg2rad(amp_deg) * np.sin(2np.pifreq_hz*t)).reshape(1,-1)

env = ImprovedB747Env( ... initial_state=np.zeros(4), ... reference_signal=sine_ref(300), ... number_time_steps=300, ... dt=0.1, ... include_reference_in_obs=True, ... ) agent = ADHDP(env, paper_strict=True, gamma=0.99) agent.train(num_episodes=100)

References

Prokhorov D.V., Wunsch D.C. "Adaptive Critic Designs." IEEE Trans. Neural Networks, vol. 8, no. 5, pp. 997-1007, 1997.
Werbos P.J. "A menu of designs for reinforcement learning over time." Neural Networks for Control, MIT Press, 1990.

Attributes:

Name	Type	Description
`actor`		Neural network that outputs control action π(R).
`critic`		Neural network that estimates action-dependent cost-to-go J(R, a).
`env`		The Gymnasium-compatible environment.
`gamma`		Discount factor for future costs.
`paper_strict`		If True, uses canonical ADHDP without baseline mixing.

Initialize the ADHDP agent.

Parameters:

Name	Type	Description	Default
`env`	`Any`	Gymnasium-compatible environment with Box observation and action spaces. For best results, use ImprovedB747Env with `include_reference_in_obs=True`.	required
`paper_strict`	`bool`	If True, enforce canonical ADHDP configuration: - No residual baseline in executed policy - No baseline substitution during critic-only phases - No BC regularizer This matches the original Prokhorov & Wunsch formulation. Default: False.	`False`
`gamma`	`float`	Discount factor for future costs. Range: [0, 1]. Default: 0.99.	`0.99`
`actor_lr`	`float`	Learning rate for the actor network optimizer (Adam). Default: 1e-4.	`0.0001`
`critic_lr`	`float`	Learning rate for the critic network optimizer (Adam). Default: 1e-4.	`0.0001`
`hidden_size`	`int`	Number of neurons in each hidden layer of both actor and critic networks. Both use two hidden layers with Tanh activation. Default: 256.	`256`
`device`	`Union[str, device]`	Torch device for computation ('cpu', 'cuda', 'mps', or torch.device instance). Default: 'cpu'.	`'cpu'`
`seed`	`int`	Random seed for reproducibility. Default: 42.	`42`
`policy_mode`	`str`	Policy composition mode: - "direct": Pure actor output (default for paper_strict) - "residual": u = u_baseline + residual_scale * pi(R) Default: "direct".	`'direct'`
`residual_scale`	`float`	Scale of learned residual when policy_mode="residual". Default: 0.2.	`0.2`
`use_baseline_in_critic_phases`	`bool`	If True, execute baseline controller during critic-only training phases. Default: False.	`False`
`action_momentum`	`float`	Momentum for action smoothing: u = mu_prev + (1-m)u_new. Range: [0, 1). Default: 0.0 (disabled).	`0.0`
`action_max_abs`	`float`	Maximum action magnitude (safety envelope) in normalized units. Range: (0, 1]. Default: 1.0.	`1.0`
`action_selection`	`str`	How to select actions: - "actor": Use actor network pi(R) (default) - "critic_gradient": Optimize action by minimizing J(R,a) w.r.t. a Default: "actor".	`'actor'`
`action_grad_steps`	`int`	Number of gradient steps for critic-based action optimization (only used if action_selection="critic_gradient"). Default: 0.	`0`
`action_grad_lr`	`float`	Learning rate for action optimization. Default: 0.0.	`0.0`
`action_grad_step_clip`	`float`	Maximum step size for action gradient updates. Default: 0.0 (no clipping).	`0.0`
`action_grad_u_l2`	`float`	L2 regularization on action magnitude during action optimization. Default: 0.0.	`0.0`
`action_grad_du_l2`	`float`	L2 regularization on action change (trust region) during action optimization. Default: 0.0.	`0.0`
`actor_update_mode`	`str`	Actor training mode: - "minimize_critic": Minimize J(R, pi(R)) via backprop through critic - "distill_critic_gradient": Supervised distillation of critic-gradient policy Default: "minimize_critic".	`'minimize_critic'`
`actor_distill_coef`	`float`	Loss coefficient for distillation mode. Default: 1.0.	`1.0`
`actor_distill_steps`	`int`	Gradient steps per distillation target. Default: 10.	`10`
`actor_distill_lr`	`float`	Learning rate for distillation optimization. Default: 0.03.	`0.03`
`exploration_std`	`float`	Standard deviation of Gaussian noise added to actions during training. Default: 0.02.	`0.02`
`critic_updates_per_step`	`int`	Number of critic gradient updates per environment step (MATLAB-style "epochs per step"). Default: 1.	`1`
`actor_updates_per_step`	`int`	Number of actor gradient updates per environment step. Default: 1.	`1`
`initial_state_noise_std`	`float`	Standard deviation of noise added to initial state for trajectory randomization (persistent excitation). Default: 0.0.	`0.0`
`reference_roll_steps`	`int`	Maximum random roll (shift) of reference signal at episode start. Default: 0.	`0`
`reference_noise_std`	`float`	Standard deviation of noise added to reference signal. Default: 0.0.	`0.0`
`baseline_warmup_episodes`	`int`	Episodes running only baseline controller for critic warmup (paper Section III). Default: 0.	`0`
`critic_warmup_episodes`	`int`	Episodes with frozen actor (critic-only training). Default: 0.	`0`
`critic_cycle_episodes`	`int`	Episodes per critic-only cycle in alternating training schedule. Default: 0 (no alternating).	`0`
`action_cycle_episodes`	`int`	Episodes per actor-only cycle in alternating training schedule. Default: 0.	`0`
`use_env_cost`	`bool`	If True, use environment's cost_total instead of shaped reward for TD target. Default: True.	`True`
`warmstart_actor_episodes`	`int`	Episodes to pre-train actor by imitating baseline via supervised learning. Default: 0.	`0`
`warmstart_actor_epochs`	`int`	Supervised epochs per warmstart episode. Default: 2.	`2`
`baseline_type`	`str`	Baseline controller type: "pd" or "pid". Default: "pid".	`'pid'`
`baseline_kp`	`float`	Proportional gain for baseline (tuned for B747). Default: -24.6295.	`-24.6295`
`baseline_ki`	`float`	Integral gain for PID baseline. Default: -0.2486.	`-0.2486`
`baseline_kd`	`float`	Derivative gain for baseline. Default: -7.8179.	`-7.8179`
`pid_i_clip`	`float`	Anti-windup integral clipping for PID. Default: 1.0.	`1.0`
`actor_bc_l2`	`float`	L2 regularization coefficient keeping actor close to baseline (behavioral cloning). Default: 0.0.	`0.0`
`actor_bc_decay`	`float`	Decay rate for actor_bc_l2 per episode. Default: 1.0.	`1.0`
`log_dir`	`Union[str, Path, None]`	Directory path for TensorBoard logs. If None, logging is disabled. Default: None.	`None`
`log_every_updates`	`int`	Frequency of logging (every N gradient updates). Default: 500.	`500`

Raises:

Type	Description
`ValueError`	If observation or action space is not Box-like.

`predict(*args, **kwargs)` ¶

Compatibility wrapper around select_action.

Accepts either

predict(state)
predict(state, deterministic=True/False)

`train(num_episodes=1, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

Train the ADHDP agent (unified interface).

Parameters:

Name	Type	Description	Default
`num_episodes`	`int`	Number of training episodes.	`1`
`max_steps`	`Optional[int]`	Optional per-episode step cap.	`None`
`save_best`	`bool`	Unused by ADHDP; accepted for API consistency.	`False`
`save_path`	`Optional[str]`	Unused by ADHDP; accepted for API consistency.	`None`
`verbose`	`bool`	If True, show a tqdm progress bar.	`True`
`**kwargs`	`Any`	Algorithm-specific options. Recognized keys: `show_progress` (bool, legacy): overrides `verbose`. `progress_desc` (str): tqdm description label.	`{}`

Returns:

Name	Type	Description
`dict`	`dict`	Training summary dictionary. Currently minimal.

`load(*args, **kwargs)` ¶

Load model weights from a directory created by save().

`from_dir(folder)` `classmethod` ¶

Instantiate env+agent from save() directory (config.json + actor/critic).

`from_pretrained(repo_name, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

Load pretrained model from local directory or Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Path to local folder with weights or repository name in format `namespace/repo_name` on Hugging Face Hub.	required
`access_token`	`Optional[str]`	Access token for private HF repository.	`None`
`version`	`Optional[str]`	Revision / branch / tag of HF repository.	`None`
`load_gradients`	`bool`	If `True`, also load optimizer states so training can be continued (requires `save_gradients=True` at save time).	`False`

Returns:

Name	Type	Description
`ADHDP`	`'ADHDP'`	Fully initialised agent with loaded weights.

`QCritic(obs_dim, act_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh)` ¶

Bases: Module

Critic approximating cost-to-go Q(s, a) (adaptive critic).

`DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None)` ¶

Bases: Module

Deterministic actor with tanh output scaled to env action bounds.

References¶

Prokhorov D.V., Wunsch D.C. "Adaptive Critic Designs." IEEE Transactions on Neural Networks, vol. 8, no. 5, pp. 997-1007, 1997.
Werbos P.J. "A menu of designs for reinforcement learning over time." Neural Networks for Control, MIT Press, 1990.
Si J., et al. "Handbook of Learning and Approximate Dynamic Programming." Wiley-IEEE Press, 2004.

Action-Dependent Heuristic Dynamic Programming (ADHDP)¶

Key Ideas¶

Key Difference: HDP vs ADHDP¶

Architecture¶

Algorithm¶

Training Loop¶

Mathematical Formulation¶

Quick Start¶

Hyperparameters¶

Core Parameters¶

Policy Mode¶

Action Selection¶

Baseline Controller¶

Training Schedule (Paper Section III)¶

Trajectory Randomization¶

Stabilization Strategies¶

1. Paper-Strict Mode¶

2. Residual Policy¶

3. Warm-Start Actor¶

4. Alternating Training¶

Comparison with Other Methods¶

Supported Environments¶

Unified training interface¶

API Reference¶

predict(*args, **kwargs) ¶

train(num_episodes=1, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs) ¶

load(*args, **kwargs) ¶

from_dir(folder) classmethod ¶

from_pretrained(repo_name, access_token=None, version=None, load_gradients=False) classmethod ¶

QCritic(obs_dim, act_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh) ¶

DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None) ¶

References¶

`predict(*args, **kwargs)` ¶

`train(num_episodes=1, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

`load(*args, **kwargs)` ¶

`from_dir(folder)` `classmethod` ¶

`from_pretrained(repo_name, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

`QCritic(obs_dim, act_dim, *, hidden_sizes=(256, 256), activation=nn.Tanh)` ¶

`DeterministicActor(obs_dim, act_dim, *, hidden_sizes=(256, 256), action_low=None, action_high=None)` ¶