Generative Adversarial Imitation Learning (GAIL)¶

GAIL — имитационное обучение через состязание актор–дискриминатор: политика учится генерировать траектории, неотличимые от экспертных, без явной функции вознаграждения.

Компоненты¶

Actor‑Critic: ActorCritic генерирует действие и оценивает \(V(s)\)
Discriminator: Discriminator отличает пары \((s,a)\) эксперта от агента
Оптимизатор политики: обновление актёра по PPO (клиппированный суррогат)

Теория¶

Минмакс‑задача (формально):

\[ \min_{\pi} \max_{D} \; \mathbb{E}_{(s,a)\sim \pi_E}[\log D(s,a)] + \mathbb{E}_{(s,a)\sim \pi}[\log (1 - D(s,a))] \]

Псевдо‑награда от дискриминатора (для актёра):

\[ r_D(s,a) = -\log D(s,a) \]

PPO‑обновление актёра (в нашей реализации):

\[ \mathcal{L}_\text{actor} = -\,\mathbb{E}\Big[ \min\big(r_t A_t,\ \mathrm{clip}(r_t,1-\varepsilon,1+\varepsilon) A_t\big) \Big],\quad r_t = \exp(\log \pi_\theta - \log \pi_{\theta_{\text{old}}}) \]

Advantage через GAE:

\[ \delta_t = r_D(s_t,a_t) + \gamma V(s_{t+1}) - V(s_t),\quad \hat{A}_t = \sum_{l\ge 0} (\gamma\lambda)^l\, \delta_{t+l} \]

Данные эксперта¶

Ожидается массив expert_data формы [N, obs_dim + act_dim] — конкатенация состояния и действия.

Обучение (контур по реализации)¶

Генерируем роллаут агента: \(s_t, a_t \sim \pi\), запоминаем log_prob, V(s)
Считаем псевдо‑награды r_D = -log D([s,a]), GAE‑returns/advantages
PPO‑обновление актёра/критика по мини‑батчам
Обучаем дискриминатор: D(fake)=1, D(real)=0 с BCE‑лоссом
Периодически тестируем политику и применяем early‑stopping по max_reward

Пример (LinearLongitudinalF16‑v0)¶

import gymnasium as gym
import numpy as np
from tensoraerospace.agent.gail.model import GAIL
from tensoraerospace.utils import generate_time_period
from tensoraerospace.signals.standard import unit_step

dt = 0.01
tp = generate_time_period(tn=20, dt=dt)
number_time_steps = len(tp)
reference_signals = unit_step(degree=5, tp=tp, time_step=1000, output_rad=True).reshape(1, -1)

env = gym.make('LinearLongitudinalF16-v0',
               number_time_steps=number_time_steps,
               initial_state=[[0],[0],[0]],
               reference_signal=reference_signals,
               use_reward=False,
               state_space=["theta","alpha","q"],
               output_space=["theta","alpha","q"],
               control_space=["ele"],
               tracking_states=["alpha"],)

expert_data = np.load('expert_f16.npy')
agent = GAIL(env, learning_rate=3e-3, max_steps=20, mini_batch_size=16, epochs=4, data=expert_data)
agent.learn(max_frames=5000, max_reward=-1)

# Унифицированный API (обёртка над learn)
agent.train(num_episodes=250, max_steps=20, max_reward=-1)

Унифицированный интерфейс обучения¶

GAIL предоставляет общий унифицированный API train() из BaseRLModel. Внутри он делегирует вызов устаревшему методу learn(), пересчитывая num_episodes * max_steps в бюджет max_frames. GAIL‑специфичные параметры, принимаемые через **kwargs:

max_frames (int): переопределяет вычисленный бюджет шагов.
max_reward (float): порог раннего останова по средней награде на тесте.

Tip

Качество expert_data критично: добавьте демо с разными начальными условиями и манёврами.

Gymnasium 5-tuple API

Реализация использует современный 5‑элементный API step из Gymnasium:

next_state, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated

Если вы переходите со старого кода с 4‑элементным API (next_state, reward, done, info = env.step(action)), убедитесь, что среда совместима с Gymnasium и возвращает 5‑элементный кортеж.

Документация API¶

`GAIL(env, learning_rate, max_steps, mini_batch_size, epochs, data, log_dir=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None, device=None)` ¶

Generative Adversarial Imitation Learning trainer.

Initialize the GAIL algorithm.

Parameters:

Name	Type	Description	Default
`env`	`Env`	Environment instance.	required
`learning_rate`	`float`	Learning rate.	required
`max_steps`	`int`	Maximum steps per rollout/episode.	required
`mini_batch_size`	`int`	Mini-batch size.	required
`epochs`	`int`	Number of training epochs.	required
`data`	`Array`	Expert demonstrations (states/actions).	required
`log_dir`	`Union[str, Path, None]`	Optional TensorBoard log directory. When provided, a `MetricWriter` is created and canonical metrics are emitted during :meth:`learn`. When `None` (default), no logging is performed.	`None`
`device`	`Optional[Union[str, device]]`	Torch device for the networks. Defaults to `"cuda"` when available, otherwise `"cpu"`.	`None`

`expert_reward(state, action)` ¶

Compute imitation reward using the discriminator.

`test_env()` ¶

Run one evaluation rollout and return total reward.

`ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantages, clip_param=0.2, env_step=0)` ¶

PPO update function.

Parameters:

Name	Type	Description	Default
`ppo_epochs`	`int`	Number of epochs.	required
`mini_batch_size`	`int`	Mini-batch size.	required
`states`	`Tensor`	Batch of states.	required
`actions`	`Tensor`	Batch of actions.	required
`log_probs`	`Tensor`	Batch of action log probabilities.	required
`returns`	`Tensor`	Batch of discounted rewards.	required
`advantages`	`Tensor`	Batch of advantage function values.	required
`clip_param`	`float`	Clipping constant.	`0.2`
`env_step`	`int`	Cumulative environment-step counter used as the x-axis for any TensorBoard scalars written from inside this update. Defaults to 0.	`0`

`train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

Train GAIL (unified interface wrapper around :meth:learn).

Parameters:

Name	Type	Description	Default
`num_episodes`	`int`	Number of episodes. Combined with `max_steps` to produce a `max_frames` budget unless `max_frames` is supplied explicitly via `**kwargs`.	`100`
`max_steps`	`Optional[int]`	Max steps per episode. Defaults to `self.max_steps` when not provided.	`None`
`save_best`	`bool`	Reserved for API consistency.	`False`
`save_path`	`Optional[str]`	Reserved for API consistency.	`None`
`verbose`	`bool`	Reserved for symmetry.	`True`
`**kwargs`	`Any`	GAIL-specific options: `max_frames` (int): override the computed budget. `max_reward` (float, default `inf`): early-stop threshold on mean test reward.	`{}`

Returns:

Name	Type	Description
`dict`	`dict`	Empty dict (GAIL does not yet collect metrics).

`learn(max_frames, max_reward)` ¶

Agent training function.

Parameters:

Name	Type	Description	Default
`max_frames`	`int`	Maximum number of steps in the environment.	required
`max_reward`	`int`	Reward threshold for stopping training.	required

`get_param_env()` ¶

Return serializable configuration of environment and policy.

`save(path=None, save_gradients=False)` ¶

Save GAIL model to the specified directory.

Saves actor-critic network, discriminator network, and configuration. Optionally saves optimizer states for resuming training.

Parameters:

Name	Type	Description	Default
`path`	`str \| Path \| None`	Base save directory. If None, saves to the current working directory.	`None`
`save_gradients`	`bool`	If True, also save optimizer state dicts.	`False`

Returns:

Name	Type	Description
`Path`	`Path`	The directory where the model was saved.

`from_pretrained(repo_name, env=None, data=None, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

Load pretrained model from a local directory or Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Path to a local folder or a Hugging Face repo id (e.g. `"namespace/repo_name"`).	required
`env`	`Optional[Env]`	Environment instance (required).	`None`
`data`	`Optional[ndarray]`	Expert demonstration data (optional).	`None`
`access_token`	`Optional[str]`	Hugging Face access token for private repos.	`None`
`version`	`Optional[str]`	Revision / branch / tag on Hugging Face.	`None`
`load_gradients`	`bool`	Restore optimizer states for continued training.	`False`

Returns:

Name	Type	Description
`GAIL`	`GAIL`	Initialized agent.

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶

Upload a saved model folder to Hugging Face Hub.

Parameters:

Name	Type	Description	Default
`repo_name`	`str`	Repository id on Hugging Face (e.g. `"user/my-gail"`).	required
`folder_path`	`Union[str, Path]`	Local folder produced by :meth:`save`.	required
`access_token`	`Optional[str]`	Hugging Face access token.	`None`

Источники¶

Generative Adversarial Imitation Learning

Где тестировалось¶

Unity‑среда
LinearLongitudinalF16‑v0 (пример в репозитории)

Generative Adversarial Imitation Learning (GAIL)¶

Компоненты¶

Теория¶

Данные эксперта¶

Обучение (контур по реализации)¶

Пример (LinearLongitudinalF16‑v0)¶

Унифицированный интерфейс обучения¶

Документация API¶

GAIL(env, learning_rate, max_steps, mini_batch_size, epochs, data, log_dir=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None, device=None) ¶

expert_reward(state, action) ¶

test_env() ¶

ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantages, clip_param=0.2, env_step=0) ¶

train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs) ¶

learn(max_frames, max_reward) ¶

get_param_env() ¶

save(path=None, save_gradients=False) ¶

from_pretrained(repo_name, env=None, data=None, access_token=None, version=None, load_gradients=False) classmethod ¶

publish_to_hub(repo_name, folder_path, access_token=None) ¶

Источники¶

Где тестировалось¶

`GAIL(env, learning_rate, max_steps, mini_batch_size, epochs, data, log_dir=None, wandb_project=None, wandb_entity=None, wandb_run_name=None, wandb_tags=None, wandb_config=None, device=None)` ¶

`expert_reward(state, action)` ¶

`test_env()` ¶

`ppo_update(ppo_epochs, mini_batch_size, states, actions, log_probs, returns, advantages, clip_param=0.2, env_step=0)` ¶

`train(num_episodes=100, *, max_steps=None, save_best=False, save_path=None, verbose=True, **kwargs)` ¶

`learn(max_frames, max_reward)` ¶

`get_param_env()` ¶

`save(path=None, save_gradients=False)` ¶

`from_pretrained(repo_name, env=None, data=None, access_token=None, version=None, load_gradients=False)` `classmethod` ¶

`publish_to_hub(repo_name, folder_path, access_token=None)` ¶