ding.policy.td3

ding.policy.td3

TD3Policy

Bases: DDPGPolicy

Overview

Policy class of TD3 algorithm. Since DDPG and TD3 share many common things, we can easily derive this TD3 class from DDPG class by changing _actor_update_freq, _twin_critic and noise in model wrapper. Paper link: https://arxiv.org/pdf/1802.09477.pdf

Config:

== ==================== ======== ================== ================================= ======================= ID Symbol Type Default Value Description Other(Shape) == ==================== ======== ================== ================================= ======================= 1 | type str td3 | RL policy register name, refer | this arg is optional, | | to registry POLICY_REGISTRY | a placeholder 2 | cuda bool False | Whether to use cuda for network | 3 | random_ int 25000 | Number of randomly collected | Default to 25000 for | collect_size | training samples in replay | DDPG/TD3, 10000 for | | buffer when training starts. | sac. 4 | model.twin_ bool True | Whether to use two critic | Default True for TD3, | critic | networks or only one. | Clipped Double | | | Q-learning method in | | | TD3 paper. 5 | learn.learning float 1e-3 | Learning rate for actor | | _rate_actor | network(aka. policy). | 6 | learn.learning float 1e-3 | Learning rates for critic | | _rate_critic | network (aka. Q-network). | 7 | learn.actor_ int 2 | When critic network updates | Default 2 for TD3, 1 | update_freq | once, how many times will actor | for DDPG. Delayed | | network update. | Policy Updates method | | | in TD3 paper. 8 | learn.noise bool True | Whether to add noise on target | Default True for TD3, | | network's action. | False for DDPG. | | | Target Policy Smoo- | | | thing Regularization | | | in TD3 paper. 9 | learn.noise_ dict | dict(min=-0.5, | Limit for range of target | | range | max=0.5,) | policy smoothing noise, | | | | aka. noise_clip. | 10 | learn.- bool False | Determine whether to ignore | Use ignore_done only | ignore_done | done flag. | in halfcheetah env. 11 | learn.- float 0.005 | Used for soft update of the | aka. Interpolation | target_theta | target network. | factor in polyak aver | | | -aging for target | | | networks. 12 | collect.- float 0.1 | Used for add noise during co- | Sample noise from dis | noise_sigma | llection, through controlling | -tribution, Ornstein- | | the sigma of distribution | Uhlenbeck process in | | | DDPG paper, Gaussian | | | process in ours. == ==================== ======== ================== ================================= =======================

Full Source Code

../ding/policy/td3.py

1from typing import List 2from ding.utils import POLICY_REGISTRY 3from .ddpg import DDPGPolicy 4 5 6@POLICY_REGISTRY.register('td3') 7class TD3Policy(DDPGPolicy): 8 """ 9 Overview: 10 Policy class of TD3 algorithm. Since DDPG and TD3 share many common things, we can easily derive this TD3 \ 11 class from DDPG class by changing ``_actor_update_freq``, ``_twin_critic`` and noise in model wrapper. 12 Paper link: https://arxiv.org/pdf/1802.09477.pdf 13 14 Config: 15 16 == ==================== ======== ================== ================================= ======================= 17 ID Symbol Type Default Value Description Other(Shape) 18 == ==================== ======== ================== ================================= ======================= 19 1 | ``type`` str td3 | RL policy register name, refer | this arg is optional, 20 | | to registry ``POLICY_REGISTRY`` | a placeholder 21 2 | ``cuda`` bool False | Whether to use cuda for network | 22 3 | ``random_`` int 25000 | Number of randomly collected | Default to 25000 for 23 | ``collect_size`` | training samples in replay | DDPG/TD3, 10000 for 24 | | buffer when training starts. | sac. 25 4 | ``model.twin_`` bool True | Whether to use two critic | Default True for TD3, 26 | ``critic`` | networks or only one. | Clipped Double 27 | | | Q-learning method in 28 | | | TD3 paper. 29 5 | ``learn.learning`` float 1e-3 | Learning rate for actor | 30 | ``_rate_actor`` | network(aka. policy). | 31 6 | ``learn.learning`` float 1e-3 | Learning rates for critic | 32 | ``_rate_critic`` | network (aka. Q-network). | 33 7 | ``learn.actor_`` int 2 | When critic network updates | Default 2 for TD3, 1 34 | ``update_freq`` | once, how many times will actor | for DDPG. Delayed 35 | | network update. | Policy Updates method 36 | | | in TD3 paper. 37 8 | ``learn.noise`` bool True | Whether to add noise on target | Default True for TD3, 38 | | network's action. | False for DDPG. 39 | | | Target Policy Smoo- 40 | | | thing Regularization 41 | | | in TD3 paper. 42 9 | ``learn.noise_`` dict | dict(min=-0.5, | Limit for range of target | 43 | ``range`` | max=0.5,) | policy smoothing noise, | 44 | | | aka. noise_clip. | 45 10 | ``learn.-`` bool False | Determine whether to ignore | Use ignore_done only 46 | ``ignore_done`` | done flag. | in halfcheetah env. 47 11 | ``learn.-`` float 0.005 | Used for soft update of the | aka. Interpolation 48 | ``target_theta`` | target network. | factor in polyak aver 49 | | | -aging for target 50 | | | networks. 51 12 | ``collect.-`` float 0.1 | Used for add noise during co- | Sample noise from dis 52 | ``noise_sigma`` | llection, through controlling | -tribution, Ornstein- 53 | | the sigma of distribution | Uhlenbeck process in 54 | | | DDPG paper, Gaussian 55 | | | process in ours. 56 == ==================== ======== ================== ================================= ======================= 57 """ 58 59 # You can refer to DDPG's default config for more details. 60 config = dict( 61 # (str) RL policy register name (refer to function "POLICY_REGISTRY"). 62 type='td3', 63 # (bool) Whether to use cuda for network. 64 cuda=False, 65 # (bool) on_policy: Determine whether on-policy or off-policy. Default False in TD3. 66 on_policy=False, 67 # (bool) Whether use priority(priority sample, IS weight, update priority) 68 # Default False in TD3. 69 priority=False, 70 # (bool) Whether use Importance Sampling Weight to correct biased update. If True, priority must be True. 71 priority_IS_weight=False, 72 # (int) Number of training samples(randomly collected) in replay buffer when training starts. 73 # Default 25000 in DDPG/TD3. 74 random_collect_size=25000, 75 # (bool) Whether to need policy data in process transition. 76 transition_with_policy_data=False, 77 # (str) Action space type 78 action_space='continuous', # ['continuous', 'hybrid'] 79 # (bool) Whether use batch normalization for reward 80 reward_batch_norm=False, 81 # (bool) Whether to enable multi-agent training setting 82 multi_agent=False, 83 model=dict( 84 # (bool) Whether to use two critic networks or only one. 85 # Clipped Double Q-Learning for Actor-Critic in original TD3 paper(https://arxiv.org/pdf/1802.09477.pdf). 86 # Default True for TD3, False for DDPG. 87 twin_critic=True, 88 ), 89 # learn_mode config 90 learn=dict( 91 # (int) How many updates(iterations) to train after collector's one collection. 92 # Bigger "update_per_collect" means bigger off-policy. 93 # collect data -> update policy-> collect data -> ... 94 update_per_collect=1, 95 # (int) Minibatch size for gradient descent. 96 batch_size=256, 97 # (float) Learning rates for actor network(aka. policy). 98 learning_rate_actor=1e-3, 99 # (float) Learning rates for critic network(aka. Q-network). 100 learning_rate_critic=1e-3, 101 # (bool) Whether ignore done(usually for max step termination env. e.g. pendulum) 102 # Note: Gym wraps the MuJoCo envs by default with TimeLimit environment wrappers. 103 # These limit HalfCheetah, and several other MuJoCo envs, to max length of 1000. 104 # However, interaction with HalfCheetah always gets done with False, 105 # Since we inplace done==True with done==False to keep 106 # TD-error accurate computation(``gamma * (1 - done) * next_v + reward``), 107 # when the episode step is greater than max episode step. 108 ignore_done=False, 109 # (float) target_theta: Used for soft update of the target network, 110 # aka. Interpolation factor in polyak averaging for target networks. 111 # Default to 0.005. 112 target_theta=0.005, 113 # (float) discount factor for the discounted sum of rewards, aka. gamma. 114 discount_factor=0.99, 115 # (int) When critic network updates once, how many times will actor network update. 116 # Delayed Policy Updates in original TD3 paper(https://arxiv.org/pdf/1802.09477.pdf). 117 # Default 1 for DDPG, 2 for TD3. 118 actor_update_freq=2, 119 # (bool) Whether to add noise on target network's action. 120 # Target Policy Smoothing Regularization in original TD3 paper(https://arxiv.org/pdf/1802.09477.pdf). 121 # Default True for TD3, False for DDPG. 122 noise=True, 123 # (float) Sigma for smoothing noise added to target policy. 124 noise_sigma=0.2, 125 # (dict) Limit for range of target policy smoothing noise, aka. noise_clip. 126 noise_range=dict( 127 # (int) min value of noise 128 min=-0.5, 129 # (int) max value of noise 130 max=0.5, 131 ), 132 ), 133 # collect_mode config 134 collect=dict( 135 # (int) How many training samples collected in one collection procedure. 136 # Only one of [n_sample, n_episode] shoule be set. 137 # n_sample=1, 138 # (int) Cut trajectories into pieces with length "unroll_len". 139 unroll_len=1, 140 # (float) It is a must to add noise during collection. So here omits "noise" and only set "noise_sigma". 141 noise_sigma=0.1, 142 ), 143 eval=dict(), # for compability 144 other=dict( 145 replay_buffer=dict( 146 # (int) Maximum size of replay buffer. Usually, larger buffer size is better. 147 replay_buffer_size=100000, 148 ), 149 ), 150 ) 151 152 def _monitor_vars_learn(self) -> List[str]: 153 """ 154 Overview: 155 Return the necessary keys for logging the return dict of ``self._forward_learn``. The logger module, such \ 156 as text logger, tensorboard logger, will use these keys to save the corresponding data. 157 Returns: 158 - necessary_keys (:obj:`List[str]`): The list of the necessary keys to be logged. 159 """ 160 return ["q_value", "loss", "lr", "entropy", "target_q_value", "td_error"]