h@k0SSKrSSKrSSKrSSKrSSKrSSKJr SSKJrJ r J r J r SSK r SSKrSSKJr SSKJr SSKJrJrJr SSKJr SSKJrJr SS KJr SS KJ r J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r( SS K)J*r*J+r+ SS K,J-r- SSK.J/r/ \ "SSS9r0"SS\5r1g)N)deepcopy)AnyOptionalTypeVarUnion)spaces) BaseAlgorithm)DictReplayBufferNStepReplayBuffer ReplayBuffer) BaseCallback) ActionNoiseVectorizedActionNoise) BasePolicy) load_from_pkl save_to_pkl)GymEnv MaybeCallback RolloutReturnSchedule TrainFreqTrainFrequencyUnit) safe_meanshould_collect_more_steps)VecEnv)HerReplayBufferSelfOffPolicyAlgorithmOffPolicyAlgorithm)boundc<Z^\rSrSr%Sr\R R\S'SAS\ \ \ \ 4S\ \ \ 4S\ \\4S\S \S \S \S \S \ \\\\ 44S\S\\S\\ \S\\\ \4S\S\S\\\ \4S\S\\ S\S\ \R0\ 4S\S\S\\S\S\S\S\S \\\ \R4S!448U4S"jjjrSBS$jrSBS%jrS&\ \ \R>\ RB4S#S4S'jr"SCS&\ \ \R>\ RB4S(\S#S4S)jjr#SDS*\S+\$S,\S-\ S.\S#\\\%44 U4S/jjjr&SES0\'S*\S+\$S1\S-\ S,\S.\S#\'4S2jjr(S\S \S#S4S3jr)SFS \S\\S4\S#\\*RV\*RV44S5jjr,SBS6jr-SBS7jr.S8\S9\*RVS:\ \*RV\\ \*RV44S;\*RVS<\*RVS=\/\\ \4S#S4S>jr0SGS\1S+\%S \2S8\S\\S \S1\\S#\34S?jjr4S@r5U=r6$)Hra{ The base for Off-Policy algorithms (ex: SAC/TD3) :param policy: The policy model to use (MlpPolicy, CnnPolicy, ...) :param env: The environment to learn from (if registered in Gym, can be str. Can be None for loading trained models) :param learning_rate: learning rate for the optimizer, it can be a function of the current progress remaining (from 1 to 0) :param buffer_size: size of the replay buffer :param learning_starts: how many steps of the model to collect transitions for before learning starts :param batch_size: Minibatch size for each gradient update :param tau: the soft update coefficient ("Polyak update", between 0 and 1) :param gamma: the discount factor :param train_freq: Update the model every ``train_freq`` steps. Alternatively pass a tuple of frequency and unit like ``(5, "step")`` or ``(2, "episode")``. :param gradient_steps: How many gradient steps to do after each rollout (see ``train_freq``) Set to ``-1`` means to do as many gradient steps as steps done in the environment during the rollout. :param action_noise: the action noise type (None by default), this can help for hard exploration problem. Cf common.noise for the different action noise type. :param replay_buffer_class: Replay buffer class to use (for instance ``HerReplayBuffer``). If ``None``, it will be automatically selected. :param replay_buffer_kwargs: Keyword arguments to pass to the replay buffer on creation. :param optimize_memory_usage: Enable a memory efficient variant of the replay buffer at a cost of more complexity. See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195 :param n_steps: When n_step > 1, uses n-step return (with the NStepReplayBuffer) when updating the Q-value network. :param policy_kwargs: Additional arguments to be passed to the policy on creation :param stats_window_size: Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over :param tensorboard_log: the log location for tensorboard (if None, no logging) :param verbose: Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages :param device: Device on which the code should run. By default, it will try to use a Cuda compatible device and fallback to cpu if it is not possible. :param support_multi_env: Whether the algorithm supports training with multiple environments (as in A2C) :param monitor_wrapper: When creating an environment, whether to wrap it or not in a Monitor wrapper. :param seed: Seed for the pseudo random generators :param use_sde: Whether to use State Dependent Exploration (SDE) instead of action noise exploration (default: False) :param sde_sample_freq: Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout) :param use_sde_at_warmup: Whether to use gSDE instead of uniform sampling during the warm up phase (before learning starts) :param sde_support: Whether the model support gSDE or not :param supported_action_spaces: The action spaces supported by the algorithm. actorNpolicyenv learning_rate buffer_sizelearning_starts batch_sizetaugamma train_freqgradient_steps action_noisereplay_buffer_classreplay_buffer_kwargsoptimize_memory_usagen_steps policy_kwargsstats_window_sizetensorboard_logverbosedevicesupport_multi_envmonitor_wrapperseeduse_sdesde_sample_frequse_sde_at_warmup sde_supportsupported_action_spaces.c:>[TU]UUUUUUUUUUUUUUS9 X@lX`lXPlXplXlXlXlXl SUl Xl U =(d 0Ul Xl XlU(aURUR S'UUlg)N)r#r$r%r2r3r4r5r6r7r8r9r:r;r>r:)super__init__r&r(r'r)r*r,r-r0 replay_bufferr.r/r1r+r:r2r<)selfr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r> __class__s g/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/off_policy_algorithm.pyrAOffPolicyAlgorithm.__init__Qs> ''/+/++$;  '$. ,(%:"59#6 $8$>B! % ,0LLD  y )!2returncl[UR[5(dpURn[U[5(dUS4nUS[ US54n[US[ 5(d[ SUS35e[U6Ulgg![ an[ SUSS35UeSnAff=f)zF Convert `train_freq` parameter (int or tuple) to a TrainFreq object. steprzEThe unit of the `train_freq` must be either 'step' or 'episode' not 'z'!Nz9The frequency of `train_freq` must be an integer and not ) isinstancer+rtupler ValueErrorint)rCr+es rE_convert_train_freq&OffPolicyAlgorithm._convert_train_freqs $//955Jj%00(&1  (m-? 1 -NO jmS11 #\]ghi]j\k!lmm'4DO#6  [\fgh\i[jjlm sB B3B..B3c<UR5 URUR5 URc[ UR [ R5(a#[UlURS:XdS5eOYURS:a>[UlURRURURS.5 O [UlURcURR!5n[#UR[$5(a#UR&cS5eUR&US'UR"UR(UR UR*4UR,UR.UR0S.UD6UlUR2"UR UR*UR440UR6D6UlUR8R;UR,5UlUR=5 g)NrKzAN-step returns are not supported for Dict observation spaces yet.)r1r*z9You must pass an environment when using `HerReplayBuffer`r$)r6n_envsr0)_setup_lr_scheduleset_random_seedr9r.rLobservation_spacerDictr r1r r/updater*r rBcopy issubclassrr$r& action_spacer6rTr0 policy_class lr_scheduler2r#torQ)rCr/s rE _setup_modelOffPolicyAlgorithm._setup_models ! TYY'  # # +$00&++>>+;(||q(m*mm(!+<())00T\\TXT^T^1_`+7(    %$(#<#<#A#A#C $22ODDxx+h-hh+.2hh$U+!%!9!9  &&!!"{{{{&*&@&@ "'"D ''  " "           kknnT[[1    "rGpathclURcS5e[XRUR5 g)z Save the replay buffer as a pickle file. :param path: Path to the file where the replay buffer should be saved. if path is a str or pathlib.Path, the path is automatically created if necessary. Nz The replay buffer is not defined)rBrr5)rCrbs rEsave_replay_buffer%OffPolicyAlgorithm.save_replay_buffers0!!-Q/QQ-D,,dll;rGtruncate_last_trajcv[XR5Ul[UR[5(dS5e[ URS5(dJSURl[R"URR5URl [UR[5(aZURcS5eURRUR5 U(aURR5 URURlg)aw Load a replay buffer from a pickle file. :param path: Path to the pickled replay buffer. :param truncate_last_traj: When using ``HerReplayBuffer`` with online sampling: If set to ``True``, we assume that the last trajectory in the replay buffer was finished (and truncate it). If set to ``False``, we assume that we continue the same trajectory (same episode). z6The replay buffer must inherit from ReplayBuffer classhandle_timeout_terminationFNzFYou must pass an environment at load time when using `HerReplayBuffer`)rr5rBrLr hasattrrhnp zeros_likedonestimeoutsrr$set_envtruncate_last_trajectoryr6)rCrbrfs rEload_replay_buffer%OffPolicyAlgorithm.load_replay_buffers+4>$,,l;;u=uu;t))+GHHURnUR=(a6 U=(a- USL=(a" UR=(d URS:nU(aF[R "S5 UceURS- UR -nSURU'URcS5eURbhURRS:aN[UR[5(d/[URURR5Ul [T U]9UUUUU5$)z cf `BaseAlgorithm`. NrzThe last trajectory in the replay buffer will be truncated, see https://github.com/DLR-RM/stable-baselines3/issues/46.You should use `reset_num_timesteps=False` or `optimize_memory_usage=False`to avoid that issue.rKTz:You must set the environment before calling _setup_learn())rBr0fullposwarningswarnr&rlr$r-num_envsrLrr@ _setup_learn) rCrrrsrtrurvrBrfryrDs rEr}OffPolicyAlgorithm._setup_learns.**   & & ># >T) >##<}'8'81'<   MM'  !, ,, $$q(M,E,EEC'+M   $xx#a%aa#    )!!A%t002GHH 5d6G6GIZIZ [D w#        rGrC log_intervalc URUUUUU5upUR[5[55 URcS5e[ UR [5(deURU:aURURUR URUURURUS9nUR(dOURS:abURUR:aHURS:a URO URnUS:aUR!UR"US9 URU:aMUR%5 U$)Nz3You must set the environment before calling learn())r+r-rsr'rBrr)r(r,)r}on_training_startlocalsglobalsr$rLr+r num_timestepscollect_rolloutsr-r'rBcontinue_trainingr,episode_timestepstrainr(on_training_end) rCrrrsrrurtrvrolloutr,s rElearnOffPolicyAlgorithm.learn8sL%)$5$5      % ! ""68WY7xx#Z%ZZ#$//95555  ?2++??!..! $ 4 4"00),G,,!!A%$*<*>- K/K K-!%dnnE!R O d'' 4 4 KK44_EM' " (FA N *M[[// >F $$,M"F$$/([s#D8c JURceURce[[R"5UR - S- [ RR5n[URUR- U- 5nURRSURSS9 [UR5S:a[URS5S:aURRS[!URVs/sHo3SPM sn55 URRS [!URVs/sHo3S PM sn55 URRS U5 URRS [U5SS9 URRS URSS9 UR"(aPURRSUR$R'5R)5R+55 [UR5S:a/URRS[!UR55 URR-URS9 gs snfs snf)z Write log data. NgeAz time/episodes tensorboard)excluderzrollout/ep_rew_meanrzrollout/ep_len_meanlztime/fpsztime/time_elapsedztime/total_timestepsz train/stdzrollout/success_rate)rJ)ep_info_bufferep_success_buffermaxtimetime_ns start_timesys float_infoepsilonrOr_num_timesteps_at_startloggerrecord _episode_numlenrr:r"get_stdmeanitemdump)rC time_elapsedfpsep_infos rE dump_logsOffPolicyAlgorithm.dump_logss ""...%%111DLLNT__<CS^^E[E[\ 4%%(D(DD TU ?D,=,=}U t"" #a 'C0C0CA0F,G!,K KK  4i]a]p]p@q]pRY]p@q6r s KK  4i]a]p]p@q]pRY]p@q6r s :s+ .L0A=Y 143E3E}] << KK  {TZZ-?-?-A,G,G,I,N,N,P Q t%% & * KK  5yAWAW7X Y d001Ar@qs JJ cg)z Method called after each step in the environment. It is meant to trigger DQN target network update but can be used for other purposes N)rCs rE_on_stepOffPolicyAlgorithm._on_steps rGrBrnew_obsrewardrlinfoscURb5URR5nURR5nOURX4sUlpx[ U5n [ U5HupU (dMXjRS5cM$[U [5(aQXjSn URbURRU 5n U R5H n XXU 'M MXjSX'URcMURRXSS245X'M URURU UUUU5 X0lURbXplgg)a Store transition in the replay buffer. We store the normalized action and the unnormalized observation. It also handles terminal observations (because VecEnv resets automatically). :param replay_buffer: Replay buffer object where to store the transition. :param buffer_action: normalized action :param new_obs: next observation in the current episode or first observation of the episode (when dones is True) :param reward: reward for the current transition :param dones: Termination signal :param infos: List of additional information about the transition. It may contain the terminal observations and information about timeout. Nterminal_observation) _vec_normalize_envget_original_obsget_original_rewardr_last_original_obsr enumerategetrLdictunnormalize_obskeysadd)rCrBrrrrlrnew_obs_reward_next_obsidone next_obs_keys rE_store_transition$OffPolicyAlgorithm._store_transitions_0  " " ...??AH--AACG:> 6D #XH%!'GAt %;<Hh-- %)? @I..:$($;$;$K$KI$V '}}+4> a( /#((+A"BHK..:&*&=&=&M&MhZ[W[n&] (   # #       !  " " .&. # /rGcURRS5 Sup[U[5(dS5eURS:dS5eUR S:a%UR [R:XdS5eUR(a%URRUR 5 UR5 Sn [X8U 5(GaUR(aGURS:a7XR-S:Xa%URRUR 5 URXeUR 5upUR!U 5upnnU=R"UR - slUS- nUR%['55 UR)5(d[+XR -U SS 9$UR-UU5 UR/XLXUU5 UR1UR"UR25 UR55 [7U5HunnU(dMU S- n U=R8S- slUb.UR S:a [;U/S 9O0nUR<"S 0UD6 UcM_UR8U-S:XdMtUR?5 M [X8U 5(aGMURA5 [+XR -X5$) a4 Collect experiences and store them into a ``ReplayBuffer``. :param env: The training environment :param callback: Callback that will be called at each step (and at the beginning and end of the rollout) :param train_freq: How much experience to collect by doing rollouts of current policy. Either ``TrainFreq(, TrainFrequencyUnit.STEP)`` or ``TrainFreq(, TrainFrequencyUnit.EPISODE)`` with ```` being an integer greater than 0. :param action_noise: Action noise that will be used for exploration Required for deterministic policy (e.g. TD3). This can also be used in addition to the stochastic policy for SAC. :param learning_starts: Number of steps before learning for the warm-up phase. :param replay_buffer: :param log_interval: Log data every ``log_interval`` episodes :return: F)rrzYou must pass a VecEnvrz,Should at least collect one step or episode.rKz7You must use only one env when doing episodic training.T)r)indicesr)!r#set_training_moderLr frequencyr|unitrSTEPr:r" reset_noiseon_rollout_startrr;rrJr update_localsron_stepr_update_info_bufferr"_update_current_progress_remaining_total_timestepsrrrrresetron_rollout_end)rCr$rsr+rBr-r'rnum_collected_stepsnum_collected_episodesractionsbuffer_actionsrrewardsrlridxrkwargss rEr#OffPolicyAlgorithm.collect_rolloutss< %%e,6:3#v&&@(@@&##a'W)WW' < *GeU   #,, .  1 $   " "68 ,##%%$%8<<%GI_sxyy  $ $UE 2  " "='TY[` a  3 3D4F4FH]H] ^ MMO&u- T4*a/*%%*%#/8; q8Hse!4b$**4V4$/D4E4E 4TXY4Y(.E( I_``^ !0<<?AWkkrG)rrr-r(r&r*r,r'r1r0r#rBr.r/r)r+r<)i@Bdg{Gzt?gGz?)rKrJrKNNNFrKNrNrautoFTNFrFTN)rHN)T)NTrunF)NrTF)NrK)NrN)7__name__ __module__ __qualname____firstlineno____doc__thnnModule__annotations__rstrtyperrfloatrrOrMrrr rrboolr6rSpacerArQr`pathlibPathioBufferedIOBaserdrprr r}rrrrjndarrayrrrlistrrrrr__static_attributes__ __classcell__)rDs@rErrs1f 55<<%"2=.2<@9=&+26!$)-(."' $"!"' LP;C3c4 ++,C363; C3UH_- C3  C3  C3C3C3C3#uS#X./C3C3{+C3&d<&89C3'tCH~6C3 $C3 !C3" S#X/#C3$%C3&"#'C3()C3*biin%+C3, -C3./C30sm1C323C345C36 7C389C3:"*%V\\0BC0G*H!I;C3C3J50)#Vrsi  00=^^;M8IO3C !9AUVD lD lrG