Deep Deterministic Policy Gradient (DDPG)¶

DDPG — off‑policy актор‑критик для непрерывных действий: обучает детерминированную стратегию и Q‑функцию, используя буфер повторов и целевые сети со «мягким» обновлением.

Компоненты¶

Политика (Actor): PolicyNetwork(s) -> a, детерминированное действие через tanh
Критик (Q‑сеть): ValueNetwork(s,a) -> Q(s,a)
Целевые сети: target_policy_net, target_value_net для стабильности
Реплей‑буфер: ReplayBuffer для выборки мини‑батчей
Эксплорейшн: орнштейн–уленбековский шум OUNoise

Теория (на базе реализации)¶

Градиент политики (DPG):

\[ \nabla_\theta J(\theta) = \mathbb{E}_{s\sim \mathcal{D}}\Big[\nabla_a Q(s,a)\big|_{a=\pi_\theta(s)}\, \nabla_\theta \pi_\theta(s)\Big] \]

В коде минимизируется \(-Q(s,\pi(s))\), что эквивалентно градиентному подъёму по \(J\).

Обновление критика (таргет Беллмана с целевыми сетями):

\[ \hat{Q}(s,a) = r + \gamma\,(1-\text{done})\, Q_{\text{target}}(s', \pi_{\text{target}}(s')) \]

Лосс критика — MSE: \(\mathcal{L}_Q = (Q(s,a) - \hat{Q})^2\).

Мягкое обновление целевых сетей:

\[ \theta^- \leftarrow (1-\tau)\,\theta^- + \tau\,\theta \]

Быстрый старт¶

import gymnasium as gym
import numpy as np
from tensoraerospace.agent.ddpg.model import DDPG
from tensoraerospace.utils import generate_time_period
from tensoraerospace.signals.standard import unit_step

# Временная сетка и референс
dt = 0.01
tp = generate_time_period(tn=20, dt=dt)
number_time_steps = len(tp)
reference_signals = unit_step(degree=5, tp=tp, time_step=1000, output_rad=True).reshape(1, -1)

# Среда F‑16
env = gym.make('LinearLongitudinalF16-v0',
               number_time_steps=number_time_steps,
               initial_state=[[0],[0],[0]],
               reference_signal=reference_signals,
               use_reward=True,
               state_space=["theta","alpha","q"],
               output_space=["theta","alpha","q"],
               control_space=["ele"],
               tracking_states=["alpha"],)

agent = DDPG(env, value_lr=1e-3, policy_lr=1e-4, replay_buffer_size=1_000_000)
agent.learn(max_frames=12000, max_steps=500, batch_size=128)

Tip

Эксплорейшн обеспечивается OU‑шумом: контролируйте sigma и decay_period, чтобы плавно снижать силу шума.

Унифицированный интерфейс обучения¶

DDPG поддерживает общий унифицированный API train() из BaseRLModel:

agent.train(
    num_episodes=24,
    max_steps=500,
    batch_size=128,
    warmup_frames=2_000,
)

Под капотом train() пересчитывает num_episodes * max_steps в бюджет max_frames и вызывает устаревший метод learn(). Поддерживаемые DDPG‑специфичные именованные аргументы (передаются через **kwargs):

max_frames, batch_size, gamma, soft_tau, warmup_frames, updates_per_step, target_value_clip.

Старый вызов agent.learn(max_frames=..., max_steps=..., batch_size=...) продолжает работать без изменений.

Документация API¶

`DDPG(env, value_lr, policy_lr, replay_buffer_size, normalize_observations=True, device=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None)` ¶

Deep Deterministic Policy Gradient (DDPG) agent.

DDPG is an off-policy actor-critic algorithm for continuous control. It combines DPG with Deep Q-Learning techniques like experience replay and target networks for stability.

Key features

Deterministic policy (actor) and Q-function (critic)
Target networks with soft updates (Polyak averaging)
Experience replay buffer for sample efficiency
Ornstein-Uhlenbeck noise for exploration
Optional observation normalization for faster convergence
Automatic action scaling to environment bounds

Reference

Lillicrap et al. "Continuous control with deep reinforcement learning" (2015) https://arxiv.org/abs/1509.02971

Initialize DDPG agent.

Parameters:

Name	Type	Description	Default
`env`	`Env`	Gym environment with continuous action space. Must have .observation_space, .action_space, .reset(), and .step() methods.	required
`value_lr`	`float`	Learning rate for the critic (Q-function) network.	required
`policy_lr`	`float`	Learning rate for the actor (policy) network.	required
`replay_buffer_size`	`int`	Maximum number of transitions to store in replay buffer.	required
`normalize_observations`	`bool`	Whether to normalize observations using running mean and standard deviation. Recommended for faster convergence. Default is True.	`True`
`device`	`Optional[Union[str, device]]`	Torch device to place networks and tensors on. If None, auto-detects CUDA and falls back to CPU. Default is None.	`None`

`ddpg_update(batch_size, gamma=0.99, min_value=-np.inf, max_value=np.inf, soft_tau=0.01)` ¶

Perform one DDPG update step on both actor and critic networks.

This method implements the core DDPG algorithm: 1. Sample a minibatch from replay buffer 2. Compute target Q-values using target networks 3. Update critic by minimizing TD error 4. Update actor using deterministic policy gradient 5. Soft update target networks (Polyak averaging)

Parameters:

Name	Type	Description	Default
`batch_size`	`int`	Number of transitions to sample from replay buffer.	required
`gamma`	`float`	Discount factor for future rewards. Default is 0.99.	`0.99`
`min_value`	`float`	Minimum value for Q-value clipping. Default is -inf.	`-inf`
`max_value`	`float`	Maximum value for Q-value clipping. Default is inf.	`inf`
`soft_tau`	`float`	Soft update coefficient (Polyak averaging). Values close to 0 mean slower updates. Default is 1e-2.	`0.01`

`train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

Train DDPG (unified interface wrapper around :meth:learn).

DDPG operates in terms of environment frames, not episodes, so this wrapper translates the unified arguments into the legacy frame-based :meth:learn signature.

Parameters:

Name	Type	Description	Default
`num_episodes`	`int`	Number of episodes. Combined with `max_steps` to produce a `max_frames` budget, unless `max_frames` is supplied directly via `**kwargs`.	`100`
`max_steps`	`Optional[int]`	Max steps per episode. Defaults to 200 when not provided.	`None`
`save_best`	`bool`	Reserved for API consistency. DDPG does not currently implement best-model checkpointing inside the training loop.	`False`
`save_path`	`Optional[str]`	Reserved for API consistency.	`None`
`verbose`	`bool`	Reserved for symmetry.	`True`
`**kwargs`	`Any`	DDPG-specific keyword arguments forwarded to :meth:`learn`. Recognized keys: `max_frames` (int): override the computed frame budget. `batch_size` (int, default 64): mini-batch size for updates. `gamma` (float, default 0.995): discount factor. `soft_tau` (float, default 5e-3): Polyak factor. `warmup_frames` (int, default 10_000): exploration warmup. `updates_per_step` (int, default 1). `target_value_clip` (tuple\|None, default `(-10, 10)`).	`{}`

Returns:

Name	Type	Description
`dict`	`dict`	Summary of training with `rewards`, `frame_idx`
	`dict`	and `num_episodes`.

`learn(max_frames, max_steps, batch_size, gamma=0.995, soft_tau=0.005, warmup_frames=10000, updates_per_step=1, target_value_clip=(-10.0, 10.0))` ¶

Train the DDPG agent.

Runs the main training loop: collect experience, update networks, and log metrics. Supports warmup period for initial exploration and multiple updates per environment step for sample efficiency.

Parameters:

Name	Type	Description	Default
`max_frames`	`int`	Maximum number of environment steps to train for.	required
`max_steps`	`int`	Maximum steps per episode before truncation.	required
`batch_size`	`int`	Minibatch size for network updates.	required
`gamma`	`float`	Discount factor for future rewards. Default is 0.995.	`0.995`
`soft_tau`	`float`	Soft update coefficient for target networks (Polyak averaging). Smaller values mean slower updates. Default is 5e-3.	`0.005`
`warmup_frames`	`int`	Number of steps to collect before starting updates. Allows the replay buffer to fill with diverse experience. Default is 10_000.	`10000`
`updates_per_step`	`int`	Number of gradient updates per environment step. Higher values improve sample efficiency but slow down training. Default is 1.	`1`
`target_value_clip`	`Optional[Tuple[float, float]]`	Tuple of (min, max) for Q-value clipping. Helps prevent overestimation. Set to None to disable clipping. Default is (-10.0, 10.0).	`(-10.0, 10.0)`

`save(filepath, include_grads=False)` ¶

Save training state (checkpoint) or full model folder.

Supports two save formats:

Single file checkpoint (.pt/.pth extension):
Backward compatible format
Contains all networks, optimizers, buffers, and training state
Suitable for resuming training
Directory format (no extension or other extensions):
HuggingFace Hub compatible structure
Separate files for config and each network
Suitable for model sharing and deployment

Parameters:

Name	Type	Description	Default
`filepath`	`Union[str, Path]`	Path to save location. If ends with .pt/.pth, saves as single file. Otherwise saves as directory.	required
`include_grads`	`bool`	Whether to save optimizer states and gradients. Only applicable for single file checkpoints. Default is False.	`False`

Examples:

>>> agent.save("checkpoint.pt", include_grads=True)  # Single file
>>> agent.save("my_model")  # Directory with config.json, etc.

`load(filepath, map_location=None, load_optimizer=True, load_targets=True, load_replay=True, load_noise=True, load_grads=False, strict=True)` ¶

Load training state from a checkpoint file.

Restores networks, optimizers, replay buffer, OU noise, and observation normalization statistics. Provides granular control over which components to restore.

Parameters:

Name	Type	Description	Default
`filepath`	`Union[str, Path]`	Path to checkpoint file (.pt or .pth).	required
`map_location`	`Optional[Union[str, device]]`	Device to load tensors to. If None, uses current device. Can be 'cpu', 'cuda', 'cuda:0', etc.	`None`
`load_optimizer`	`bool`	Whether to restore optimizer states (momentum, etc.). Set to False for inference only. Default is True.	`True`
`load_targets`	`bool`	Whether to restore target network weights. Set to False for inference only. Default is True.	`True`
`load_replay`	`bool`	Whether to restore replay buffer contents. Set to False to start with fresh buffer. Default is True.	`True`
`load_noise`	`bool`	Whether to restore OU noise state. Set to False to reset exploration. Default is True.	`True`
`load_grads`	`bool`	Whether to restore parameter gradients. Useful for debugging gradient flow. Default is False.	`False`
`strict`	`bool`	Whether to strictly match state dict keys. Set to False for partial loading. Default is True.	`True`

Raises:

Type	Description
`FileNotFoundError`	If checkpoint file doesn't exist.

`get_param_env()` ¶

Collect environment and policy parameters for saving.

Creates a configuration dictionary containing all information needed to reconstruct the agent and environment. Compatible with HuggingFace Hub format for model sharing.

Returns:

Type	Description
`Dict[str, Dict[str, Any]]`	Dictionary with 'env' and 'policy' keys, each containing: - 'name': Fully qualified class name - 'params': Initialization parameters

`__load(path, load_gradients=False)` `classmethod` ¶

Load a DDPG agent from disk.

Parameters:

Name	Type	Description	Default
`path`	`Union[str, Path]`	Folder containing saved weights and config.json.	required
`load_gradients`	`bool`	Whether to restore optimizer states.	`False`

Returns:

Name	Type	Description
`DDPG`	`'DDPG'`	Reconstructed agent instance.

`from_pretrained(repo_name, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

Load a pretrained DDPG model from local directory or Hugging Face Hub.

Automatically detects whether the path is local or a Hub repository. Downloads model if necessary and reconstructs the complete agent with environment, networks, and all parameters.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Local path or HuggingFace Hub repository name. Examples: './my_model', 'username/ddpg-b747', '/abs/path/to/model'.	required
`access_token`	`Optional[str]`	HuggingFace API token for private repositories. Not needed for public repos or local paths. Default is None.	`None`
`version`	`Optional[str]`	Specific version/commit/tag to load from Hub. Default is None (latest version).	`None`
`load_gradients`	`bool`	Whether to restore optimizer states with gradients. Useful for continuing training. Default is False.	`False`

Returns:

Type	Description
`'DDPG'`	Initialized DDPG agent with loaded weights and configuration.

Raises:

Type	Description
`FileNotFoundError`	If local path doesn't exist.
`TheEnvironmentDoesNotMatch`	If config specifies wrong agent class.

Examples:

>>> # Load from local directory
>>> agent = DDPG.from_pretrained("./my_saved_model")
>>>
>>> # Load from Hugging Face Hub
>>> agent = DDPG.from_pretrained("username/ddpg-b747-v1")
>>>
>>> # Load private model with token
>>> agent = DDPG.from_pretrained(
...     "username/private-model",
...     access_token="hf_..."
... )

`push_to_hub(repo_name, access_token=None, save_path=None, include_gradients=False)` ¶

Save the model and upload to Hugging Face Hub.

Saves the model in HuggingFace-compatible format (config.json + separate network files) and uploads to the specified repository. Creates the repository if it doesn't exist.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Name of the HuggingFace Hub repository. Format: 'username/repo-name' or just 'repo-name' (uses your username).	required
`access_token`	`Optional[str]`	HuggingFace API token with write access. Required for pushing. Get from https://huggingface.co/settings/tokens	`None`
`save_path`	`Optional[Union[str, Path]]`	Local directory to save model before uploading. If None, creates a timestamped directory. Default is None.	`None`
`include_gradients`	`bool`	Whether to save optimizer states. Useful for sharing training checkpoints. Default is False.	`False`

Returns:

Type	Description
`str`	Path to the local saved folder.

Raises:

Type	Description
`ValueError`	If access_token is not provided or invalid.

Examples:

>>> agent.push_to_hub(
...     repo_name="my-awesome-ddpg",
...     access_token="hf_your_token_here"
... )
'Oct05_14-23-45_DDPG'

Note

The uploaded model can be loaded by anyone using:

agent = DDPG.from_pretrained("username/my-awesome-ddpg")

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶

Publish model to Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Repository name in Hub.	required
`folder_path`	`str`	Path to model folder.	required
`access_token`	`str`	Access token for authentication.	`None`

Источники¶

Continuous control with deep reinforcement learning

Где тестировалось¶

Unity‑среда
LinearLongitudinalF16‑v0 (пример в репозитории)

Deep Deterministic Policy Gradient (DDPG)¶

Компоненты¶

Теория (на базе реализации)¶

Быстрый старт¶

Унифицированный интерфейс обучения¶

Документация API¶

DDPG(env, value_lr, policy_lr, replay_buffer_size, normalize_observations=True, device=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None) ¶

ddpg_update(batch_size, gamma=0.99, min_value=-np.inf, max_value=np.inf, soft_tau=0.01) ¶

train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs) ¶

learn(max_frames, max_steps, batch_size, gamma=0.995, soft_tau=0.005, warmup_frames=10000, updates_per_step=1, target_value_clip=(-10.0, 10.0)) ¶

save(filepath, include_grads=False) ¶

load(filepath, map_location=None, load_optimizer=True, load_targets=True, load_replay=True, load_noise=True, load_grads=False, strict=True) ¶

get_param_env() ¶

__load(path, load_gradients=False) classmethod ¶

from_pretrained(repo_name, access_token=None, version=None, load_gradients=False) classmethod ¶

push_to_hub(repo_name, access_token=None, save_path=None, include_gradients=False) ¶

publish_to_hub(repo_name, folder_path, access_token=None) ¶

Источники¶

Где тестировалось¶

`DDPG(env, value_lr, policy_lr, replay_buffer_size, normalize_observations=True, device=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None)` ¶

`ddpg_update(batch_size, gamma=0.99, min_value=-np.inf, max_value=np.inf, soft_tau=0.01)` ¶

`train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

`learn(max_frames, max_steps, batch_size, gamma=0.995, soft_tau=0.005, warmup_frames=10000, updates_per_step=1, target_value_clip=(-10.0, 10.0))` ¶

`save(filepath, include_grads=False)` ¶

`load(filepath, map_location=None, load_optimizer=True, load_targets=True, load_replay=True, load_noise=True, load_grads=False, strict=True)` ¶

`get_param_env()` ¶

`__load(path, load_gradients=False)` `classmethod` ¶

`from_pretrained(repo_name, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

`push_to_hub(repo_name, access_token=None, save_path=None, include_gradients=False)` ¶

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶