h9SSKrSSKrSSKrSSKJrJrJrJr SSKr SSK r SSK J r SSKJr SSKJrJr SSKJr SSKJr SSKJrJrJr SS KJrJr SS KJr \"S S S 9r "SS \5r!g)N)AnyOptionalTypeVarUnion)spaces) BaseAlgorithm)DictRolloutBuffer RolloutBuffer) BaseCallbackActorCriticPolicy)GymEnv MaybeCallbackSchedule) obs_as_tensor safe_mean)VecEnvSelfOnPolicyAlgorithmOnPolicyAlgorithm)boundc0^\rSrSr%Sr\\S'\\S'S0S\\ \ \4S\\ \ 4S\\ \ 4S\S \ S \ S \ S \ S \ S\S\S\\ \S\\\ \4S\S\\ S\S\\\ \4S\S\\S\\R(\ 4S\S\\\ \R.S44,U4SjjjrS1SjrS2S\ SS4S jjrS\S!\S\S"\S\4 S#jrS1S$jrS3S%\SS4S&jjrS4S'\ S(\S!\!S)\S*\ S+\S,\S\ 4S-jjr"S\\#\ \#\ 44S.jr$S/r%U=r&$)5ra The base for On-Policy algorithms (ex: A2C/PPO). :param policy: The policy model to use (MlpPolicy, CnnPolicy, ...) :param env: The environment to learn from (if registered in Gym, can be str) :param learning_rate: The learning rate, it can be a function of the current progress remaining (from 1 to 0) :param n_steps: The number of steps to run for each environment per update (i.e. batch size is n_steps * n_env where n_env is number of environment copies running in parallel) :param gamma: Discount factor :param gae_lambda: Factor for trade-off of bias vs variance for Generalized Advantage Estimator. Equivalent to classic advantage when set to 1. :param ent_coef: Entropy coefficient for the loss calculation :param vf_coef: Value function coefficient for the loss calculation :param max_grad_norm: The maximum value for the gradient clipping :param use_sde: Whether to use generalized State Dependent Exploration (gSDE) instead of action noise exploration (default: False) :param sde_sample_freq: Sample a new noise matrix every n steps when using gSDE Default: -1 (only sample at the beginning of the rollout) :param rollout_buffer_class: Rollout buffer class to use. If ``None``, it will be automatically selected. :param rollout_buffer_kwargs: Keyword arguments to pass to the rollout buffer on creation. :param stats_window_size: Window size for the rollout logging, specifying the number of episodes to average the reported success rate, mean episode length, and mean reward over :param tensorboard_log: the log location for tensorboard (if None, no logging) :param monitor_wrapper: When creating an environment, whether to wrap it or not in a Monitor wrapper. :param policy_kwargs: additional arguments to be passed to the policy on creation :param verbose: Verbosity level: 0 for no output, 1 for info messages (such as device or wrappers used), 2 for debug messages :param seed: Seed for the pseudo random generators :param device: Device (cpu, cuda, ...) on which the code should be run. Setting it to auto, the code will be run on the GPU if possible. :param _init_setup_model: Whether or not to build the network at the creation of the instance :param supported_action_spaces: The action spaces supported by the algorithm. rollout_bufferpolicyNenv learning_raten_stepsgamma gae_lambdaent_coefvf_coef max_grad_normuse_sdesde_sample_freqrollout_buffer_classrollout_buffer_kwargsstats_window_sizetensorboard_logmonitor_wrapper policy_kwargsverboseseeddevice_init_setup_modelsupported_action_spaces.c>[TU]UUUUUUU U SUUUUUS9 X@lXPlX`lXplXlXlXlU =(d 0Ul U(aUR5 gg)NT)rrrr*r+r-r#r$support_multi_envr)r,r'r(r/) super__init__rrrr r!r"r%r& _setup_model)selfrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/ __class__s f/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/on_policy_algorithm.pyr3OnPolicyAlgorithm.__init__=s2 ''+"+/+$;  "  $  *$8!%:%@b"      returncUR5 URUR5 URc@[ UR [ R5(a [UlO [UlUR"URUR UR4URURURURS.UR D6UlUR$"UR URUR&4SUR(0UR*D6UlUR,R/UR5UlUR15 g)N)r-rrn_envsr#)_setup_lr_scheduleset_random_seedr,r% isinstanceobservation_spacerDictr r r action_spacer-rrr<r&r policy_class lr_scheduler#r*rto_maybe_recommend_cpur5s r7r4OnPolicyAlgorithm._setup_modelss! ! TYY'  $ $ ,$00&++>>,=),9)"77 LL  " "     ;;**;;  ((  ''  " "D$5$5t7G7G QUQ]Q] aeasas  kknnT[[1  !!#r9mlp_class_namecURRnUR[R"S5:wa=X!:Xa7[R "SUR RSUS3[5 ggg)zx Recommend to use CPU only when using A2C/PPO with MlpPolicy. :param: The name of the class for the default MlpPolicy. cpuzYou are trying to run zg on the GPU, but it is primarily intended to run on the CPU when not using a CNN policy (you are using a/ which should be a MlpPolicy). See https://github.com/DLR-RM/stable-baselines3/issues/1245 for more info. You can pass `device='cpu'` or `export CUDA_VISIBLE_DEVICES=` to force using the CPU.Note: The model will train, but the GPU utilization will be poor and the training might take longer than on CPU.N)rC__name__r-thwarningswarnr6 UserWarning)r5rIpolicy_class_names r7rF&OnPolicyAlgorithm._maybe_recommend_cpuss !--66 ;;"))E* */@/R MM()@)@(AB""3!45>> 0S *r9callbackn_rollout_stepscURcS5eURRS5 SnUR5 UR(a%URR UR 5 UR5 XT:Ga!UR(aGURS:a7XPR-S:Xa%URR UR 5 [R"5 [URUR5nURU5upxn SSS5 WR5R5nUn [UR ["R$5(avURR&(aURR)U 5n O?[*R,"XpR R.UR R05n UR3U 5uppU=R4UR - slUR7[955 UR;5(dgUR=X5 US- n[UR ["R>5(aURASS5n[CU 5HunnU(dMXRES5cM%XRESS5(dM@URRGXS5Sn[R"5 URRIU5SnSSS5 X==URJW-- ss'M URMURUU URNWW 5 XlXl'XT:aGM![R"5 URRI[W UR55nSSS5 URQWW S 9 UR7[955 URS5 g !,(df  GN =f!,(df  GN=f!,(df  Nl=f) am Collect experiences using the current policy and fill a ``RolloutBuffer``. The term rollout here refers to the model-free notion and should not be used with the concept of rollout used in model-based RL or planning. :param env: The training environment :param callback: Callback that will be called at each step (and at the beginning and end of the rollout) :param rollout_buffer: Buffer to fill with rollouts :param n_rollout_steps: Number of experiences to collect per environment :return: True if function returned with at least `n_rollout_steps` collected, False if callback terminated rollout prematurely. Nz$No previous observation was providedFrterminal_observationzTimeLimit.truncated) last_valuesdonesT)* _last_obsrset_training_moderesetr# reset_noisenum_envson_rollout_startr$rMno_gradrr-rKnumpyr?rBrBox squash_outputunscale_actionnpcliplowhighstep num_timesteps update_localslocalson_step_update_info_bufferDiscretereshape enumerateget obs_to_tensorpredict_valuesradd_last_episode_startscompute_returns_and_advantageon_rollout_end)r5rrSrrTr obs_tensoractionsvalues log_probsclipped_actionsnew_obsrewardsrZinfosidxdone terminal_obsterminal_values r7collect_rollouts"OnPolicyAlgorithm.collect_rolloutssF(~~)Q+QQ) %%e, << KK # #CLL 1!!#'|| 4 4q 8WG[G[=[_`=` '' 5*4>>4;;G -1[[-D*kkm))+G&O$++VZZ88;;,,'+kk&@&@&QO')ggg7H7H7L7LdN_N_NdNd&eO-0XXo-F *Ge   #,, .   " "68 ,##%%  $ $U 2 qLG$++V__==!//"a0'u- TD '=>J '!$)- $26"(."&LP/4 c4 12234 63; 4 UH_- 4  4  4 4 4 4 4 4 4 'tM':;4  (S#X74 4 "#!4 "#4 $ S#X/%4 &'4 (sm)4 *biin%+4 , -4 ."*%V\\0BC0G*H!I/4 4 l$63QU(j jj& j  j  jX"232t24#'.$(")#)) ) )  ) " )) )Vd3ic.B(Cr9)"rrrNtypingrrrrrbrftorchrM gymnasiumr#stable_baselines3.common.base_classr stable_baselines3.common.buffersr r "stable_baselines3.common.callbacksr !stable_baselines3.common.policiesr %stable_baselines3.common.type_aliasesrrrstable_baselines3.common.utilsrr stable_baselines3.common.vec_envrrrrr9r7rsR 00=M;?QQC3 7?RSE Er9