hlSrSSKrSSKrSSKrSSKrSSKJrJr SSKJ r SSK J r SSK J r JrJrJrJr SSKrSSKrSSKrSSKJr SSKJr SS KJrJrJrJr SS K J!r! SS K"J#r# SS K$J%r% SS K&J'r' SSK(J)r) SSK*J+r+J,r,J-r- SSK.J/r/J0r0J1r1J2r2 SSK3J4r4J5r5J6r6J7r7 SSK8J9r9J:r:J;r;Jr> SSK?J@r@JArAJBrBJCrCJDrDJErE SSKFJGrGJHrH \"SSS9rIS\\4\J4S\KS\44SjrL"SS\5rMg)z(Abstract base classes for RL algorithms.N)ABCabstractmethod)deque)Iterable)AnyClassVarOptionalTypeVarUnion)spaces)utils) BaseCallback CallbackListConvertCallbackProgressBarCallback) is_wrapped)Logger)Monitor) ActionNoise) BasePolicy)check_for_nested_spacesis_image_spaceis_image_space_channels_first)load_from_zip_filerecursive_getattrrecursive_setattrsave_to_zip_file)GymEnv MaybeCallbackSchedule TensorDict) FloatSchedulecheck_for_correct_spaces get_deviceget_system_infoset_random_seedupdate_learning_rate) DummyVecEnvVecEnv VecNormalizeVecTransposeImageis_vecenv_wrappedunwrap_vec_normalize)_convert_space _patch_envSelfBaseAlgorithm BaseAlgorithm)boundenvverbosereturnc[U[5(a/UnUS:a[SUS35 [R"USS9nU$U$![ a [R"U5nU$f=f)zIf env is a string, make the environment; otherwise, return env. :param env: The environment to learn from. :param verbose: Verbosity level: 0 for no output, 1 for indicating if environment is created :return A Gym (vector) environment. z*Creating environment from the given name '' rgb_array) render_mode) isinstancestrprintgymmake TypeError)r3r4env_ids ]/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/base_class.pymaybe_make_envrC0ss#s a< >vhaH I #((6{;C J3J #((6"C J #sA A+*A+c \rSrSr%Sr0r\\\\ \ 4\ S'\ \ S'\ R\ S'\ R\ S'\\ S'\\ S'\\ S 'SOS\\\ \ 4S \\\S 4S \\\4S \\\\4S\S\\S\S\\R.\4S\S\S\\S\S\S\\\ \ RS4SS 4Sjjr\SPS \S\S\S\4Sjj5r\SQSj5rS\SS 4Sjr \!S\4Sj5r"SQS jr#S!\S"\SS 4S#jr$S$\\%\RLRN\RLRN4SS 4S%jr(S\%\4S&jr)S'\S\ \ 4S(jr*S\\%\\%\44S)jr+SRS*\,S+\S\-4S,jjr.SSS"\S*\,S-\S.\S+\S\\\-44 S/jjr/STS0\%\\\4S1\\0RbSS 4S2jjr2S\\4S3jr3S\\44S4jr5SUS \S5\SS 4S6jjr6\SVS7\7S"\S*\,S8\S.\S-\S+\S\74S9jj5r8SWS:\\0Rb\\\0Rb44S;\\\0RbS4S<\\0RbS=\S\\0Rb\\\0RbS444 S>jjr9STS\\SS 4S?jjr:SXS@\\\;4SA\S\\R.\4SS 4SBjjr<\=SYSC\ \7SD\\\>R~\@R4S \\S\\R.\4SE\\\\4SF\S5\S\74SGjj5rBS\\\44SHjrCSZSD\\\>R~\@R4SI\\D\SJ\\D\SS 4SKjjrESQSLjrFSQSMjrGSNrHg )[r1CaD The base of RL algorithms :param policy: The policy model to use (MlpPolicy, CnnPolicy, ...) :param env: The environment to learn from (if registered in Gym, can be str. Can be None for loading trained models) :param learning_rate: learning rate for the optimizer, it can be a function of the current progress remaining (from 1 to 0) :param policy_kwargs: Additional arguments to be passed to the policy on creation :param stats_window_size: Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over :param tensorboard_log: the log location for tensorboard (if None, no logging) :param verbose: Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages :param device: Device on which the code should run. By default, it will try to use a Cuda compatible device and fallback to cpu if it is not possible. :param support_multi_env: Whether the algorithm supports training with multiple environments (as in A2C) :param monitor_wrapper: When creating an environment, whether to wrap it or not in a Monitor wrapper. :param seed: Seed for the pseudo random generators :param use_sde: Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False) :param sde_sample_freq: Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout) :param supported_action_spaces: The action spaces supported by the algorithm. policy_aliasespolicyobservation_space action_spacen_envs lr_schedule_loggerNr3 learning_rate policy_kwargsstats_window_sizetensorboard_logr4devicesupport_multi_envmonitor_wrapperseeduse_sdesde_sample_freqsupported_action_spaces.r5c[U[5(aURU5UlOXl[ U5UlUS:a[ SUR S35 XplUc0OUUlSUl SUl SUl Xl SUl SUlX0lX`lSUlSUlSUlSUlXlXlSUlXPlSUlSUlSUlSUlSUlSUlUGb[?X R5nURAX RU 5nURBUl!URDUl"URFUl$X l[KU5UlUb3[URDU5(dSUS URDS 35eU (dURHS:a [MS 5eUS ;a7[URB[NRP5(a[MS U35eUR*(a4[URD[NRR5(d [MS5e[URD[NRR5(au[TRV"[TRX"[TRZ"URDR\URDR^/555(dS5eggg)Nr7zUsing z devicerg?FzThe algorithm only supports z as action spaces but z was providedz]Error: the model does not support multiple envs; it requires a single vectorized environment.) MlpPolicy CnnPolicyzNYou must use `MultiInputPolicy` when working with dict observation space, not zXgeneralized State-Dependent Exploration (gSDE) can only be used with continuous actions.z@Continuous action space must have a finite lower and upper bound)0r;r<_get_policy_from_name policy_classr$rQr=r4rN num_timesteps_total_timesteps_num_timesteps_at_startrT action_noise start_timerMrP _last_obs_last_episode_starts_last_original_obs _episode_numrUrV_current_progress_remaining_stats_window_sizeep_info_bufferep_success_buffer _n_updates_custom_loggerr3_vec_normalize_envrC _wrap_envrHrInum_envsrJr- ValueErrorr DictBoxnpallisfinitearraylowhigh)selfrGr3rMrNrOrPr4rQrRrSrTrUrVrWs rB__init__BaseAlgorithm.__init__js" fc " " $ : :6 BD  &  ( a< F4;;-w/ 0 #0#8Rm !'($ 37*.$(!"& .,/("3"!%#%):> ? ll3C..llODC%(%:%:D " # 0 0D ,,DKH';3&?D #&2!$"3"35LMM23J2KL,,-]<M %q v 33 4CYCY[a[f[f8g8g #qrxqy!z{{||Jt/@/@&**$M$M !{||$++VZZ88vvKK$*;*;*?*?ARARAWAW)X YZVUV9= c^[T[5(dd[T5m[T[5(d#U(aUS:a [ S5 [ T5mUS:a [ S5 [ U4Sj/5m[TR5 [T[5(dSn[TR[R5(aWTRRR5H.nU=(d" [U5=(a [U5(+nM0 O6[TR5=(a [TR5(+nU(aUS:a [ S5 [T5mT$)a^" Wrap environment with the appropriate wrappers if needed. For instance, to have a vectorized environment or to re-order the image channels. :param env: :param verbose: Verbosity level: 0 for no output, 1 for indicating wrappers used :param monitor_wrapper: Whether to wrap the env in a ``Monitor`` when possible. :return: The wrapped environment. r7z)Wrapping the env with a `Monitor` wrapperz"Wrapping the env in a DummyVecEnv.c>T$Nr3srB)BaseAlgorithm._wrap_env..ssr|Fz(Wrapping the env in a VecTransposeImage.)r;r)r/rrr=r(rrHr,r+r rqvaluesrr)r3r4rSwrap_with_vectransposespaces` rBrnBaseAlgorithm._wrap_envs4#v&&S/Cc7++a<EFcl!|:;{m,C  5 56 &788%* "#//==!2299@@BE-C.&u-Z6STY6Z2Z+C *88M8M)N*Wt))XT&&a<DE', r|cg)z'Create networks, buffer and optimizers.Nrrys rB _setup_modelBaseAlgorithm._setup_modelr|loggercXlSUlg)z Setter for for logger object. .. warning:: When passing a custom logger object, this will overwrite ``tensorboard_log`` and ``verbose`` settings passed to the constructor. TN)rLrl)ryrs rB set_loggerBaseAlgorithm.set_loggers "r|cUR$)zGetter for the logger object.)rLrs rBrBaseAlgorithm.logger s||r|c8[UR5Ulg)z Transform to callable if needed.N)r"rMrKrs rB_setup_lr_schedule BaseAlgorithm._setup_lr_schedules(););<r|r^total_timestepscBS[U5[U5- - Ulg)z Compute current progress remaining (starts from 1 and ends to 0) :param num_timesteps: current number of timesteps :param total_timesteps: rYN)floatrg)ryr^rs rB"_update_current_progress_remaining0BaseAlgorithm._update_current_progress_remainings ,/}1EoH^1^+^(r| optimizerscURRSURUR55 [ U[ 5(dU/nUH'n[ X RUR55 M) g)z Update the optimizers learning rate using the current learning rate schedule and the current progress remaining (from 1 to 0). :param optimizers: An optimizer or a list of optimizers. ztrain/learning_rateN)rrecordrKrgr;listr')ryr optimizers rB_update_learning_rate#BaseAlgorithm._update_learning_ratesa 0$2B2B4CcCc2de*d++$J#I ,<,((r|infosdonescURceURceUc#[R"S/[ U5-5n[ U5Hqup4UR S5nUR S5nUbURRU/5 UcMKX#(dMVURRU5 Ms g)z Retrieve reward, episode length, episode success and update the buffer if using Monitor wrapper or a GoalEnv. :param infos: List of additional information about the transition. :param dones: Termination signals NFepisode is_success) rirjrsrvlen enumerategetextendappend)ryrridxinfo maybe_ep_infomaybe_is_successs rB_update_info_buffer!BaseAlgorithm._update_info_buffers""...%%111 =HHeWs5z12E"5)IC HHY/M#xx 5 (##**M?;+ &&--.>? *r|cUR$)za Returns the current environment (can be None if not defined). :return: The current environment rrs rBget_envBaseAlgorithm.get_envs xxr|cUR$)zk Return the ``VecNormalize`` wrapper of the training env if it exists. :return: The ``VecNormalize`` env. )rmrs rBget_vec_normalize_env#BaseAlgorithm.get_vec_normalize_envs&&&r| force_resetcURXR5nURUR:Xd9SURSURSURR S35e[ XRUR5 [U5Ul U(aSUl URUlXl g)a Checks the validity of the environment, and if it is coherent, set it as the current environment. Furthermore wrap any non vectorized env into a vectorized checked parameters: - observation_space - action_space :param env: The environment for learning a policy :param force_reset: Force call to ``reset()`` before training to avoid unexpected behavior. See issue https://github.com/DLR-RM/stable-baselines3/issues/597 zaThe number of environments to be set is different from the number of environments in the model: (z != zz), whereas `set_env` requires them to be the same. To load a model with a different number of environments, you must use `z.load(path, env)` insteadN) rnr4rorJ __class____name__r#rHrIr-rmrcr3)ryr3rs rBset_envBaseAlgorithm.set_envsnnS,,/||t{{*   ~T$++/AAEAXAX@YYr t * !&<&O>OP#7s"; !DNll r|ry log_intervalcg)a0 Return a trained model. :param total_timesteps: The total number of samples (env steps) to train on Note: it is a lower bound, see `issue #1150 `_ :param callback: callback(s) called at every step with state of the algorithm. :param log_interval: for on-policy algos (e.g., PPO, A2C, ...) this is the number of training iterations (i.e., log_interval * n_steps * n_envs timesteps) before logging; for off-policy algos (e.g., TD3, SAC, ...) this is the number of episodes before logging. :param tb_log_name: the name of the run for TensorBoard logging :param reset_num_timesteps: whether or not to reset the current timestep number (used in logging) :param progress_bar: Display a progress bar using tqdm and rich. :return: the trained model Nr)ryrrrrrrs rBlearnBaseAlgorithm.learnrr| observationstate episode_start deterministicc:URRXX45$)ae Get the policy action from an observation (and optional hidden state). Includes sugar-coating to handle different observations (e.g. normalizing images). :param observation: the input observation :param state: The last hidden states (can be None, used in recurrent policies) :param episode_start: The last masks (can be None, used in recurrent policies) this correspond to beginning of episodes, where the hidden states of the RNN must be reset. :param deterministic: Whether or not to return deterministic actions. :return: the model's action and the next hidden state (used in recurrent policies) )rGpredict)ryrrrrs rBrBaseAlgorithm.predicts({{"";}TTr|cUcg[XRR[R"S5R:HS9 URR U5 UR bUR R U5 gg)zh Set the seed of the pseudo-random generators (python, numpy, pytorch, gym, action_space) :param seed: Ncuda) using_cuda)r&rQtypethrIrTr3)ryrTs rBr&BaseAlgorithm.set_random_seed/se < )9)9RYYv=N=S=S)ST t$ 88  HHMM$  r|load_path_or_dict exact_matchc 0n[U[5(aUnO [XSS9upTn[UR 5S5n[5nUHqnSn [ X5n [U [RR5(aU RXH5 OU RXHUS9 URU5 Ms U(aXv:wa[SUS U35egg![ an [SUS35U eSn A ff=f) a Load parameters from a given zip-file or a nested dictionary containing parameters for different modules (see ``get_parameters``). :param load_path_or_iter: Location of the saved data (path or file-like, see ``save``), or a nested dictionary containing nn.Module parameters used by the policy. The dictionary maps object names to a state-dictionary returned by ``torch.nn.Module.state_dict()``. :param exact_match: If True, the given parameters should include parameters for each module and each of their parameters, otherwise raises an Exception. If set to False, this can be used to update only specific parameters. :param device: Device on which the code should run. F)rQ load_datarNzKey z is an invalid object name.)strictz>Names of parameters do not match agents' parameters: expected z, got ) r;dictrsetrr Exceptionrproptim Optimizerload_state_dictadd) ryrrrQparams_objects_needing_updateupdated_objectsnameattres rBset_parametersBaseAlgorithm.set_parameters>s%$ ' . .&F-.?Z_`LAq "%T%@%@%B1%E!F%DD R(4$ 2 233 $$V\2$$V\+$F    %=@ ?D236/9JL E;9 R!4v-H!IJPQQ  Rs C$$ D.C>>Dclspathcustom_objectsprint_system_infoc 8U(a[S5 [5 [UUUUS9upn UcS5eU cS5eSU;abSUS;aUSS USRS5n U (a8[ U [ 5(a#[ U S [ 5(a U S USS'SU;a#USUS:wa[S USS US35eS U;dS U;a [S5eSHn [X5X'M UbHURX(S5n[X(S US 5 U(aUbSUS'UbURUS'O SU;aUSnU"USUUSS9n U RRU5 U RRU5 U R5 U R!U SUS9 U b.U H(nU UcM [3U US"3U UR45 M* U R6(aU R,R95 U $!["aUnS[%U5;a?S[%U5;a0U R!U SUS9 [&R("SUS35 SnANUeSnAf[anU SSS Sn[+U5n[+U R,R.R0S S5nUS U-:XaBUSUU SSS S'U R!U SUS9 [&R("S!US35 SnAGNXUeSnAff=f)#a Load the model from a zip-file. Warning: ``load`` re-creates the model from scratch, it does not update it in-place! For an in-place load use ``set_parameters`` instead. :param path: path to the file (or a file-like) where to load the agent from :param env: the new environment to run the loaded model on (can be None if you only need prediction from a trained model) has priority over any saved environment :param device: Device on which the code should run. :param custom_objects: Dictionary of objects to replace upon loading. If a variable is present in this dictionary as a key, it will not be deserialized and the corresponding item will be used instead. Similar to custom_objects in ``keras.models.load_model``. Useful when you have an object in file that can not be deserialized. :param print_system_info: Whether to print system info from the saved model and the current system info (useful to debug loading issues) :param force_reset: Force call to ``reset()`` before training to avoid unexpected behavior. See https://github.com/DLR-RM/stable-baselines3/issues/597 :param kwargs: extra arguments to change the model when loading :return: new model instance with loaded parameters z== CURRENT SYSTEM INFO ==)rQrrNzNo data found in the saved filez!No params found in the saved filerNrQnet_archrzQThe specified policy kwargs do not equal the stored policy kwargs.Stored kwargs: z, specified kwargs: rHrIzTThe observation_space and action_space were not given, can't verify new environments>rIrHr4rcrJr3r]F)rGr3rQ_init_setup_modelT)rrQpi_features_extractorzMissing key(s) in state_dictzYou are probably loading a A2C/PPO model saved with SB3 < 1.7.0, we deactivated exact_match so you can save the model again to avoid issues in the future (see https://github.com/DLR-RM/stable-baselines3/issues/1233 for more info). Original error: z> Note: the model should still work fine, this only a warning.zpolicy.optimizer param_groupsrzYou are probably loading a DQN model saved with SB3 < 2.4.0, we truncated the optimizer state so you can save the model again to avoid issues in the future (see https://github.com/DLR-RM/stable-baselines3/pull/1963 for more info). Original error: z.data)r=r%rrr;rrrpKeyErrorr.rnr#ro__dict__updaterr RuntimeErrorr<warningswarnrrGrr rdatarU reset_noise)rrr3rQrrrkwargsrrpytorch_variablessaved_net_archkeymodelrsaved_optim_paramsn_params_savedn_paramsrs rBloadBaseAlgorithm.loadsF  - .  *< )/ + ''B!BB!F#FF! d "400)(3!/266zBN*^T"B"BzR`abRceiGjGj4B14E_%j1 f $)@DDY)Y""&"7!88LVTcMdLeg  d *nD.Hqr r9C&ty1DI9 ?--)_5C $S/B*CT.EY Zt/$([!!$X}5k'#   d# f% (   T& IR  () %T*2"%D69J49P9U9UV* == LL $ $ & s '#a&05SWZ[\W]5]$$Vv$N '()c*S S "((:!;N!KA!Nx!X  !34N5<<11>>qA(KLHX-J\]f^fJg)*>:1=hG$$Vf$M '()c*S S+ s2)H L"A I20I22 L?B LLLczUR5up0nUH n[X5nUR5X4'M" U$)z Return the parameters of the agent. This includes parameters from different networks, e.g. critics (value functions) and policies (pi functions). :return: Mapping of from names of the objects to PyTorch state-dicts. )rr state_dict)rystate_dicts_namesrrrrs rBget_parametersBaseAlgorithm.get_parameters$sD $::<%D$T0D??,FL& r|excludeincludecURR5nUc/n[U5RUR 55nUbUR U5nUR 5upVXV-nUH(nURS5Sn URU 5 M* UHn URU S5 M Sn Ub0n UHn [X 5n XU 'M UR5n[XXS9 g)aD Save all the attributes of the object and the model parameters in a zip-file. :param path: path to the file where the rl agent should be saved :param exclude: name of parameters that should be excluded in addition to the default ones :param include: name of parameters that might be excluded but should be included anyway N.r)rrr) rcopyrunionr differencersplitrpoprr#r)ryrr%r&rr"torch_variable_namesall_pytorch_variables torch_varvar_name param_namerrrparams_to_saves rBsaveBaseAlgorithm.save3s}}!!# ?Gg,$$T%?%?%AB  ((1G262M2M2O/ 1 H.I s+A.H KK ! /"J HHZ &"!  + " ,(4*.$'- ,,.er|c[5e)zK Write log data. (Implemented by OffPolicyAlgorithm and OnPolicyAlgorithm) )NotImplementedErrorrs rB dump_logsBaseAlgorithm.dump_logses "##r|cN[R"S5 UR"U6 g)Nz^algo._dump_logs() is deprecated in favor of algo.dump_logs(). It will be removed in SB3 v2.7.0)rrr8)ryargss rB _dump_logsBaseAlgorithm._dump_logsks vw r|)rgrlrfrdrcrerLrkr`rhr_rmrarIrQr3rirjrMrKrJr^rHr]rNrVrTrbrPrUr4) NdNrautoFTNFN)rT)r5N)F)NTrunFr)T)Nr>rATF)NNF)Tr?)Nr?NFT)NN)Ir __module__ __qualname____firstlineno____doc__rFrrr<rr__annotations__r Spaceintr rr rrr rrrQrtuplerz staticmethodr)rnrrrpropertyrrrrrrrrr\rrrrrrsndarrayrrr*rrr0rrr&r!r classmethodpathlibPathioBufferedIOBaserr#rr4r8r<__static_attributes__rr|rBr1r1CsA<=?NHT#tJ'7"789> ||#,, K O37!$)-(."' $"!LP_Vc4 ++,_V63$ %_VUH_- _V  S#X/ _V  _V"#_V_Vbiin%_V _V_Vsm_V_V_V"*%V\\0BC0G*H!I_V !_VB-v--$-RX--^66 # #D #=__VY_^b_`d288;M;M6NPRPXPXPbPb6b0c`hl`  tCy (>>j9I>"d3ic.B(C*#  8#'$( " 5)5) 5)" 5)  5)  5) sL !5)n@d38n)=@hrzzFZ@fj@*&)'x '='!6!!!F#' $("         "      837.2# U2::tCO'<<=Ubjj#o./U  + U  U rzz8E"**c/$:;; < U,  HSM  T  $!(. B j1BBbiin% B  BH!%(.37"' _ # $_Cr'8'889_f _biin% _ !c3h0 _  __ __B S$Y $,0+/ 0fCr'8'8890f(3-(0f(3-( 0f  0fd$ r|)NrErPrNrrabcrr collectionsrcollections.abcrtypingrrr r r gymnasiumr>numpyrstorchrr stable_baselines3.commonr "stable_baselines3.common.callbacksrrrr!stable_baselines3.common.env_utilrstable_baselines3.common.loggerr stable_baselines3.common.monitorrstable_baselines3.common.noiser!stable_baselines3.common.policiesr&stable_baselines3.common.preprocessingrrr"stable_baselines3.common.save_utilrrrr%stable_baselines3.common.type_aliasesrrr r!stable_baselines3.common.utilsr"r#r$r%r&r' stable_baselines3.common.vec_envr(r)r*r+r,r-*stable_baselines3.common.vec_env.patch_gymr.r/r0r<rHrCr1rr|rBrgs.  #$::*oo82468yyyy]]R/Gfck*SV&j Cj r|