Incremental Heuristic Dynamic Programming (IHDP)

IHDP — инкрементальный вариант Heuristic Dynamic Programming из семейства Adaptive Critic Designs (ACD) для управления нелинейными объектами при неполном знании модели. В авиационных задачах используется для синтеза продольного управления. См. также модель F‑16: LinearLongitudinalF16.

Ключевые идеи

• Инкрементальная модель линеаризует динамику локально по данным онлайн
• Actor формирует управляющее воздействие по ошибке слежения
• Critic оценивает стоимостную функцию и даёт градиенты Actor

Состав IHDP

Компонент	Роль	Реализация
Incremental model	Онлайн-идентификация и линеаризация динамики	tensoraerospace.agent.ihdp.Incremental_model.IncrementalModel
Actor	Генерация управляющего сигнала (NN)	tensoraerospace.agent.ihdp.Actor
Critic	Оценка J(x) и градиента dJ/dx (NN)	tensoraerospace.agent.ihdp.Critic
IHDPAgent	Оркестрация модулей, шаг predict и обучение	tensoraerospace.agent.ihdp.model.IHDPAgent

Быстрый старт

Пример инициализации агента и одного шага предсказания:

import numpy as np
from tensoraerospace.agent.ihdp.model import IHDPAgent

actor_settings = {
    "start_training": 100,
    "layers": (64, 32, 1),
    "activations": ("tanh", "tanh", "tanh"),
    "learning_rate": 0.01,
    "learning_rate_exponent_limit": 8,
    "type_PE": "3211",
    "amplitude_3211": 1,
    "pulse_length_3211": 15,
    "maximum_input": 25,
    "maximum_q_rate": 20,
    "WB_limits": 30,
    "NN_initial": None,
    "cascade_actor": False,
    "learning_rate_cascaded": 0.01,
}

critic_settings = {
    "Q_weights": np.eye(2),
    "start_training": 100,
    "gamma": 0.99,
    "learning_rate": 0.01,
    "learning_rate_exponent_limit": 8,
    "layers": (64, 32, 1),
    "activations": ("tanh", "tanh", "tanh"),
    "indices_tracking_states": [0, 1],
    "WB_limits": 30,
    "NN_initial": None,
}

incremental_settings = {
    "number_time_steps": 1000,
    "dt": 0.02,
    "input_magnitude_limits": 25,
    "input_rate_limits": 20,
}

tracking_states = ["alpha", "wz"]
selected_states = ["alpha", "wz"]
selected_input = ["elevator"]
number_time_steps = 1000
indices_tracking_states = [0, 1]

agent = IHDPAgent(
    actor_settings,
    critic_settings,
    incremental_settings,
    tracking_states,
    selected_states,
    selected_input,
    number_time_steps,
    indices_tracking_states,
)

# Один шаг предсказания
xt = np.zeros((len(selected_states), 1))
reference = np.zeros((len(selected_states), number_time_steps))
ut = agent.predict(xt, reference, time_step=0)

Tip

Убедитесь, что indices_tracking_states согласованы с порядком вектора состояний среды.

Гиперпараметры

Actor

Параметр	Описание
layers, activations	Архитектура NN и активации
learning_rate, learning_rate_exponent_limit	Скорость обучения и предел масштабирования
type_PE, amplitude_3211, pulse_length_3211	Персистентное возбуждение
maximum_input, maximum_q_rate, WB_limits	Ограничения и насыщения
cascade_actor, learning_rate_cascaded	Каскадный режим

Critic

Параметр	Описание
Q_weights	Матрица весов функции стоимости
gamma	Дисконт
learning_rate, learning_rate_exponent_limit	Обучение
layers, activations	Архитектура
indices_tracking_states	Индексы отслеживаемых состояний
WB_limits, NN_initial	Ограничения/инициализация

Incremental model

Параметр	Описание
number_time_steps, dt	Горизонт и шаг интегрирования
input_magnitude_limits	Ограничение по величине управления
input_rate_limits	Ограничение по скорости изменения

Поддерживаемые окружения

• LinearLongitudinalF16-v0

Примеры

• Подробный пример для F‑16: IHDP ↔ LinearLongitudinalF16

Документация API

IHDPAgent(actor_settings, critic_settings, incremental_settings, tracking_states, selected_states, selected_input, number_time_steps, indices_tracking_states)

Bases: object

IHDP Control Agent.

Parameters:

Name	Type	Description	Default
actor_settings	dict	Actor settings.	required
critic_settings	dict	Critic settings.	required
incremental_settings	dict	Incremental model settings.	required
tracking_states	list[str]	Tracked states.	required
selected_states	list[str]	Selected states.	required
selected_input	list[str]	Selected input signals.	required
number_time_steps	int	Number of time steps.	required
indices_tracking_states	list[int]	Index of tracked states.	required

Compose IHDP agent components.

Parameters:

Name	Type	Description	Default
actor_settings	dict	Configuration for Actor.	required
critic_settings	dict	Configuration for Critic.	required
incremental_settings	dict	Configuration for IncrementalModel.	required
tracking_states	list[str]	Tracked state names.	required
selected_states	list[str]	State variable names.	required
selected_input	list[str]	Control input names.	required
number_time_steps	int	Episode length.	required
indices_tracking_states	list[int]	Indices of tracked states.	required

predict(xt, reference_signals, time_step)

Make prediction and get next control signals.

Parameters:

Name	Type	Description	Default
xt	_type_	Current state of the control object at step t.	required
reference_signals	_type_	Reference control signal.	required
time_step	_type_	Current time step.	required

Returns:

Name	Type	Description
ut	_type_	Control signal at step t+1.

get_param_env()

Build a JSON-serialisable config for :meth:save.

Mirrors the structure used by other TensorAeroSpace agents: policy.name identifies the agent class, policy.params captures everything the constructor needs.

save(path=None)

Write the agent to a directory.

Files produced

• config.json — constructor kwargs (settings dicts + tracking/state metadata).
• actor.pth / critic.pth — state_dict of the actor and critic inner torch.nn networks.

Parameters:

Name	Type	Description	Default
path	Union[str, Path, None]	Base directory (None → CWD).	None

Returns:

Type	Description
str	Absolute path to the created run directory.

from_pretrained(repo_name, access_token=None, version=None) classmethod

Load an agent from a local directory or Hugging Face Hub.

Parameters:

Name	Type	Description	Default
repo_name	str	Local folder path, or namespace/repo_name on the Hugging Face Hub.	required
access_token	Optional[str]	Hub access token for private repos.	None
version	Optional[str]	Hub revision / branch / tag.	None

Returns:

Name	Type	Description
IHDPAgent	IHDPAgent	Reconstructed agent.

publish_to_hub(repo_name, folder_path, access_token=None)

Upload a :meth:save directory to the Hugging Face Hub.

Parameters:

Name	Type	Description	Default
repo_name	str	Target repository id, e.g. "me/my-ihdp".	required
folder_path	Union[str, Path]	Local folder produced by :meth:save.	required
access_token	Optional[str]	Hub access token.	None

Actor

Actor network for IHDP.

This module defines the Actor component used by the IHDP agent.

Actor(selected_inputs, selected_states, tracking_states, indices_tracking_states, number_time_steps, start_training, layers=(6, 1), activations=('sigmoid', 'sigmoid'), learning_rate=0.9, learning_rate_cascaded=0.9, learning_rate_exponent_limit=10, type_PE='3211', amplitude_3211=1, pulse_length_3211=15, WB_limits=30, maximum_input=25, maximum_q_rate=20, cascaded_actor=False, NN_initial=None, cascade_tracking_state=None, model_path=None, use_integral_correction=False, integral_gain=0.0, integral_clamp_deg=5.0, integral_warmup_steps=500)

Actor Model in IHDP.

Provides Actor class with Actor function approximator (NN). Actor creates neural network model using PyTorch and can train network online. User can choose number of layers, number of neurons, batch size, number of epochs and activation functions.

Parameters:

Name	Type	Description	Default
selected_inputs	list[str]	Selected control signals.	required
selected_states	list[str]	Selected state signals.	required
tracking_states	list[str]	Tracked states.	required
indices_tracking_states	list[int]	Indices of tracked states.	required
number_time_steps	int	Number of time steps.	required
start_training	int	Step from which training begins.	required
layers	tuple	Model layers. Defaults to (6, 1).	(6, 1)
activations	tuple	Activation layers ('sigmoid', 'sigmoid').	('sigmoid', 'sigmoid')
learning_rate	float	Learning rate. Defaults to 0.9.	0.9
learning_rate_cascaded	float	Learning rate in cascade mode. Defaults to 0.9.	0.9
learning_rate_exponent_limit	int	Learning rate exponent limit. Defaults to 10.	10
type_PE	str	PE type. Defaults to '3211'.	'3211'
amplitude_3211	int	3211 amplitude. Defaults to 1.	1
pulse_length_3211	int	3211 pulse length. Defaults to 15.	15
WB_limits	int	Weight limits. Defaults to 30.	30
maximum_input	int	Maximum value. Defaults to 25.	25
maximum_q_rate	int	Maximum rate. Defaults to 20.	20
cascaded_actor	bool	Enable cascade network mode. Defaults to False.	False
NN_initial	optional	Weight initialization. Defaults to None.	None
cascade_tracking_state	list	Tracking in cascade mode. Defaults to ['alpha', 'wz'].	None
model_path	str	Model path for loading weights. Defaults to None.	None

Initialize IHDP Actor network and hyperparameters.

Parameters:

Name	Type	Description	Default
selected_inputs	list[str]	Control input names.	required
selected_states	list[str]	State variable names.	required
tracking_states	list[str]	Tracked states for reward.	required
indices_tracking_states	list[int]	Indices of tracked states in state vector.	required
number_time_steps	int	Total time steps in episode.	required
start_training	int	Step index to start training.	required
layers	tuple[int, ...]	Hidden layer sizes.	(6, 1)
activations	tuple[str, ...]	Activations per layer.	('sigmoid', 'sigmoid')
learning_rate	float	Base learning rate.	0.9
learning_rate_cascaded	float	Learning rate for cascaded mode.	0.9
learning_rate_exponent_limit	int	Exponent limit for LR decay.	10
type_PE	str	Persistent excitation pattern.	'3211'
amplitude_3211	float	Amplitude for 3211 signal.	1
pulse_length_3211	int	Pulse length for 3211 signal.	15
WB_limits	float	Weight/bias clipping limit.	30
maximum_input	float	Max control magnitude.	25
maximum_q_rate	float	Max pitch rate.	20
cascaded_actor	bool	Whether to use cascaded network.	False
NN_initial	int | None	Optional weight initializer seed.	None
cascade_tracking_state	list[str] | None	Tracking states for cascade mode.	None
model_path	str | None	Path to load/save model weights.	None
use_integral_correction	bool	Enable an integral compensation term added to the actor's control output. IHDP minimizes a quadratic LQ-functional and therefore has no integral action — the closed loop has a small but persistent steady-state offset on setpoint-tracking. Enabling this flag adds -K_I · ∫(ref − y) dτ to u (with anti-windup clipping) which removes the offset without touching the actor architecture or the existing IHDP training loop.	False
integral_gain	float	K_I (units: deg / (rad·s) of integral state on the stab channel). Tuned empirically; values in the range 10..25 work well for the F-16 alpha-tracking benchmark. Ignored if use_integral_correction is False.	0.0
integral_clamp_deg	float	Anti-windup limit on the integrator state, in degrees. Caps the magnitude of accumulated error so a long initial transient cannot saturate the actuator after settling. Default 5°.	5.0
integral_warmup_steps	int	Number of steps after which the integrator starts accumulating. Set this to be at least as long as the persistent-excitation pulse, so the integrator does not "see" the PE injection as a real tracking error. Default 500 (==5 s at dt=0.01s).	500

build_actor_model()

Function creating Actor network. This is a fully connected network. Can define number of layers, number of neurons per layer, and activation functions.

In cascade mode the actor decomposes into two SISO sub-networks

• outer model : alpha_error (scalar) -> q_ref
• inner model_q : q_error (scalar) -> u

Both sub-networks therefore have input_dim=1 regardless of the number of tracking states declared on the agent. The cascade indices_tracking_states rewrite (which advertises [alpha_idx,wz_idx] so that :meth:run_actor_online can address both rows of the augmented observation) is applied AFTER both sub-models have been built — otherwise create_NN would size model_q for a 2-element input and matrix multiplication would fail at runtime.

save_model()

Save model.

save_dut_dWb()

Save gradient.

load_dut_dWb()

Load gradient.

load_model()

Load model weights.

create_NN(store_weights, seed)

Create NN with user input.

Parameters:

Name	Type	Description	Default
store_weights	dict	Dictionary containing weights and biases.	required
seed	int	Seed for saving random variables.	required

Returns:

Name	Type	Description
model	Sequential	Created NN model.
store_weights	dict	Dictionary containing updated weights and biases.

run_actor_online(xt, xt_ref)

Generate system input with given and real states.

Parameters:

Name	Type	Description	Default
xt	ndarray	Current state of time step.	required
xt_ref	ndarray	Reference state of current time step.	required

Returns:

Name	Type	Description
ut	ndarray	Input to system and incremental model.

train_actor_online(Jt1, dJt1_dxt1, G)

Get chain rule elements, calculate gradient and apply it to corresponding weights and biases.

Parameters:

Name	Type	Description	Default
Jt1	_type_	dEa/dJ	required
dJt1_dxt1	_type_	dJ/dx	required
G	ndarray	dx/du, obtained from incremental model.	required

train_actor_online_adaptive_alpha(Jt1, dJt1_dxt1, G, incremental_model, critic, xt_ref1)

Train Actor using adaptive alpha depending on sign and magnitude of network errors.

Parameters:

Name	Type	Description	Default
Jt1	ndarray	Critic evaluation with incremental model next time step prediction.	required
dJt1_dxt1	ndarray	Critical network gradient with respect to incremental model next time prediction.	required
G	ndarray	Input data distribution matrix.	required
incremental_model	Any	Incremental model.	required
critic	Any	Critic.	required
xt_ref1	ndarray	Reference state at next time step.	required

train_actor_online_adam(Jt1, dJt1_dxt1, G, incremental_model, critic, xt_ref1)

Train the actor online using Adam updates.

train_actor_online_alpha_decay(Jt1, dJt1_dxt1, G, incremental_model, critic, xt_ref1)

Train the actor with a learning rate that decays over time.

compute_Adam_update(count, gradient, model, learning_rate)

Compute an Adam-style weight update and apply it.

check_WB_limits(count, model)

Clamp weights/biases that exceed the configured WB_limits.

compute_persistent_excitation(*args)

Compute the persistent excitation term for the current time step.

update_actor_attributes()

Update time-dependent actor attributes after each time step.

evaluate_actor(*args)

Evaluate the actor using provided or stored state/reference.

restart_time_step()

Reset the internal time-step counter to zero.

restart_actor()

Reset actor state and optimizer-related attributes.

Critic

Critic network for IHDP.

This module defines the Critic component used by the IHDP agent.

Critic(Q_weights, selected_states, tracking_states, indices_tracking_states, number_time_steps, start_training, gamma=0.8, learning_rate=2, learning_rate_exponent_limit=10, layers=(10, 6, 1), activations=('sigmoid', 'sigmoid', 'linear'), WB_limits=30, NN_initial=None, model_path=None)

Provides Critic class with function approximator (NN) for Critic class.

Critic creates neural network model using PyTorch and can train network online. User can choose number of layers, number of neurons, batch size, number of epochs and activation functions.

Parameters:

Name	Type	Description	Default
Q_weights	list[float]	Q-function weights.	required
selected_states	list[str]	Selected states.	required
tracking_states	list[str]	Tracked states.	required
indices_tracking_states	list[int]	Index of tracked states.	required
number_time_steps	int	Number of time steps.	required
start_training	int	Training start step.	required
gamma	float	Gamma discount factor. Defaults to 0.8.	0.8
learning_rate	int	Learning rate. Defaults to 2.	2
learning_rate_exponent_limit	int	Learning rate exponent limit. Defaults to 10.	10
layers	tuple	Number of layers and neurons in layers. Defaults to (10, 6, 1).	(10, 6, 1)
activations	tuple	Activation functions in layers. Defaults to ("sigmoid", "sigmoid", "linear").	('sigmoid', 'sigmoid', 'linear')
WB_limits	int	Weight value constraints. Defaults to 30.	30
NN_initial	optional	Initial weight values. Defaults to None.	None
model_path	optional	Model path. Defaults to None.	None

Initialize IHDP Critic network and buffers.

Parameters:

Name	Type	Description	Default
Q_weights	list[float]	Diagonal weights for Q matrix.	required
selected_states	list[str]	State variable names.	required
tracking_states	list[str]	Tracked states for cost.	required
indices_tracking_states	list[int]	Indices of tracked states.	required
number_time_steps	int	Total steps in episode.	required
start_training	int	Step index to start training.	required
gamma	float	Discount factor.	0.8
learning_rate	float	Optimizer learning rate.	2
learning_rate_exponent_limit	int	Exponent limit for LR decay.	10
layers	tuple[int, ...]	Hidden layer sizes.	(10, 6, 1)
activations	tuple[str, ...]	Activation functions per layer.	('sigmoid', 'sigmoid', 'linear')
WB_limits	float	Weight/bias clipping limit.	30
NN_initial	int | None	Optional weight initializer seed.	None
model_path	str | None	Optional path to load/save model.	None

save_model()

Save model.

load_model()

Load weights.

save_Jt_ct()

Save critic state evaluation.

load_Jt_ct()

Load critic state evaluation.

build_critic_model()

Function creating neural network. Currently this is a densely connected neural network. User can define number of layers, number of neurons, and activation function.

run_train_critic_online_adaptive_alpha(xt, xt_ref)

Function that evaluates critic neural network once and returns J(xt) value. At the same time it trains function approximator with adaptive learning rate scheme.

Parameters:

Name	Type	Description	Default
xt	ndarray	Current state of time step.	required
xt_ref	ndarray	Reference state of current time step for computing one-step cost function.	required

Returns:

Name	Type	Description
Jt	ndarray	Critic evaluation at current time step.

run_train_critic_online_adam(xt, xt_ref)

Function that evaluates critic neural network once and returns J(xt) value. At the same time, it trains function approximator using Adam optimizer.

Parameters:

Name	Type	Description	Default
xt	ndarray	Current state of time step.	required
xt_ref	ndarray	Reference state of current time step for computing one-step cost function.	required

Returns:

Name	Type	Description
Jt	ndarray	Critic evaluation at current time step.

adam_iteration(dJt_dW, dE_dJ, iteration=None)

Adam updates all weights and biases considering loss function derivative with respect to NN output and derivative of neural network output with respect to weights and biases.

Parameters:

Name	Type	Description	Default
dJt_dW	_type_	Derivative of NN output with respect to weights and biases.	required
dE_dJ	_type_	Derivative of loss function with respect to NN output.	required

run_train_critic_online_alpha_decay(xt, xt_ref)

Evaluate the critic once and update it with a decaying learning rate.

Parameters:

Name	Type	Description	Default
xt	ndarray	Current state.	required
xt_ref	ndarray	Reference state used to compute one-step cost.	required

Returns:

Type	Description
ndarray	np.ndarray: Critic value estimate at the current time step.

train_critic_replay_adam(replay_size, iteration)

Train the critic using samples from the replay buffer (Adam).

compute_forward_pass(xt, xt_ref, replay=False)

compute_forward_pass(
    xt: np.ndarray,
    xt_ref: np.ndarray,
    replay: Literal[False] = False,
) -> tuple[torch.Tensor, list[np.ndarray]]

compute_forward_pass(
    xt: np.ndarray,
    xt_ref: np.ndarray,
    replay: Literal[True],
) -> tuple[torch.Tensor, list[np.ndarray], np.ndarray]

Compute critic output and gradients with respect to weights/biases.

compute_loss_derivative(*args)

Compute derivative of the critic loss with respect to critic output.

check_WB_limits(count)

Clamp weights/biases to the configured absolute limit (WB_limits).

evaluate_critic(xt, xt_ref)

Evaluate critic and compute gradient with respect to the input.

c_computation()

Compute one-step cost for the current time step.

targets_computation_online(*args)

Compute the TD target used for critic training.

update_critic_attributes()

Update time-dependent critic attributes after each step.

restart_time_step()

Reset the time step counter to zero.

restart_critic()

Reset critic internal state and buffers.

IncrementalModel(selected_states, selected_input, number_time_steps, discretisation_time=0.5, input_magnitude_limits=25, input_rate_limits=60)

Provides IncrementalModel class for system identification.

IncrementalModel computes A and x matrices needed for system identification, computes F and G matrices needed for incremental model, and evaluates identified model to provide state estimates at next time step.

Parameters:

Name	Type	Description	Default
selected_states	list[str]	Selected states.	required
selected_input	list[str]	Selected control signals.	required
number_time_steps	int	Number of time steps.	required
discretisation_time	float	Discretization time. Defaults to 0.5.	0.5
input_magnitude_limits	int	Input control signal limits. Defaults to 25.	25
input_rate_limits	int	Control signal rate constraints. Defaults to 60.	60

Initialize incremental model buffers and limits.

Parameters:

Name	Type	Description	Default
selected_states	list[str]	Names of states.	required
selected_input	list[str]	Names of control inputs.	required
number_time_steps	int	Horizon length.	required
discretisation_time	float	Sampling period.	0.5
input_magnitude_limits	float	Max control magnitude.	25
input_rate_limits	float	Max control rate change.	60

save_matrix()

Save identification matrices to disk (NumPy .npy files).

load_matrix()

Load identification matrices from disk (NumPy .npy files).

build_A_LS_matrix()

Build the least-squares A matrix used for online identification.

build_x_LS_vector()

Build the least-squares x vector used for online identification.

identify_incremental_model_LS(xt, ut_0)

Estimate F and G matrices for the incremental model (least squares).

evaluate_incremental_model(*args)

Estimate states for the next time step.

Returns:

Name	Type	Description
xt1_est	_type_	Estimated state for the next time step.

update_incremental_model_attributes()

Update attributes that change with each time step.

restart_time_step()

Reset time step to zero.

restart_incremental_model()

Restart the incremental model.

Источники

• Incremental Model Based Heuristic Dynamic Programming for Nonlinear Adaptive Flight Control
• IHDP (reference implementation)