hSrSSKrSSKrSSKrSSKJrJr SSKJr SSK J r J r J r J r SSKrSSKrSSKJr SSKJr SSKJrJrJrJrJrJrJr SS KJrJrJ r J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r( SS K)J*r*J+r+ SS K,J-r-J.r.J/r/ \ "S SS9r0"SS\Rb5r2"SS\2\5r3"SS\35r4"SS\45r5"SS\45r6"SS\25r7g)z;Policies: abstract base class and concrete implementations.N)ABCabstractmethod)partial)AnyOptionalTypeVarUnion)spaces)nn)BernoulliDistributionCategoricalDistributionDiagGaussianDistribution DistributionMultiCategoricalDistributionStateDependentNoiseDistributionmake_proba_distribution)get_action_dimis_image_spacemaybe_transposepreprocess_obs)BaseFeaturesExtractorCombinedExtractorFlattenExtractor MlpExtractor NatureCNN create_mlp) PyTorchObsSchedule) get_deviceis_vectorized_observation obs_as_tensor SelfBaseModel BaseModel)boundcN^\rSrSr%Sr\R R\S'\ SSS\R RS4S\ RS\ RS\ \S \\\\4S \\S \S \ \R RS \\\\44U4SjjjrS%S\\\4S \\S\\\44SjjrS\4SjrS\S \S\R04SjrS\\\44Sjr\S\R84Sj5rS\SS4Sjr\S&S\ \S\S\ \R8\4S\4Sjj5r!S\"RFSS4Sjr$S\"RF4Sjr%S\SS4S jr&S!\ \"RF\\\"RF44S\4S"jr'S!\ \"RF\\\"RF44S\(\\44S#jr)S$r*U=r+$)'r#'as The base model object: makes predictions in response to observations. In the case of policies, the prediction is an action. In the case of critics, it is the estimated value of the observation. :param observation_space: The observation space of the environment :param action_space: The action space of the environment :param features_extractor_class: Features extractor to use. :param features_extractor_kwargs: Keyword arguments to pass to the features extractor. :param features_extractor: Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise) :param normalize_images: Whether to normalize images or not, dividing by 255.0 (True by default) :param optimizer_class: The optimizer to use, ``th.optim.Adam`` by default :param optimizer_kwargs: Additional keyword arguments, excluding the learning rate, to pass to the optimizer optimizerNTobservation_space action_spacefeatures_extractor_classfeatures_extractor_kwargsfeatures_extractornormalize_imagesoptimizer_classoptimizer_kwargsc ">[T U]5 Uc0nUc0nXlX lXPlX`lXplXlX0lX@l U(d?[U[[45(a#URR[SS95 ggg)NT)normalized_image)super__init__r(r)r,r-r.r/r*r+ issubclassrrupdatedict) selfr(r)r*r+r,r-r.r/ __class__s [/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/policies.pyr3BaseModel.__init__?s   #!  $ ,(* %!2("4 0. 0(@%)B&J/G)UfIg$h$h  * * 1 1$2M N%i net_kwargsreturncUR5nUcUR5nUR[X"RS95 U$)a Update the network keyword arguments and create a new features extractor object if needed. If a ``features_extractor`` object is passed, then it will be shared. :param net_kwargs: the base network keyword arguments, without the ones related to features extractor :param features_extractor: a features extractor object. If None, a new object will be created. :return: The updated keyword arguments )r, features_dim)copymake_features_extractorr5r6r?)r7r<r,s r9_update_features_extractor$BaseModel._update_features_extractor`sE __&  %!%!=!=!? $2DSrSrstr;cPUR"UR40URD6$)z-Helper method to create a features extractor.)r*r(r+r7s r9rA!BaseModel.make_features_extractorvs$,,T-C-CftGeGeffr;obscN[XRURS9nU"U5$)z Preprocess the observation if needed and extract features. :param obs: Observation :param features_extractor: The features extractor to use. :return: The extracted features )r-)rr(r-)r7rGr,preprocessed_obss r9extract_featuresBaseModel.extract_featureszs**#/E/EX\XmXmn!"233r;cT[URURURS9$)z Get data that need to be saved in order to re-create the model when loading it from disk. :return: The dictionary to pass to the as kwargs constructor when reconstruction this model. )r(r)r-)r6r(r)r-rEs r9_get_constructor_parameters%BaseModel._get_constructor_parameterss. "44**"22   r;c^UR5HnURs $ [S5$)zInfer which device this policy lives on by inspecting its parameters. If it has no parameters, the 'cpu' device is used as a fallback. :return:cpu) parametersdevicer)r7params r9rRBaseModel.devices* __&E<< '%  r;pathcp[R"UR5UR5S.U5 g)z/ Save model to a given location. :param path: ) state_dictdataN)thsaverWrM)r7rUs r9rZBaseModel.saves( t0$:Z:Z:\]_cdr;clsrRc[U5n[R"XSS9nU"S0USD6nURUS5 UR U5 U$)zj Load model from path. :param path: :param device: Device on which the policy should be loaded. :return: F) map_location weights_onlyrXrW)rrYloadload_state_dictto)r\rUrRsaved_variablesmodels r9raBaseModel.loadsXF#''$%P.of-. ol;<  r;vectorc[RRR[R"U[R UR S9UR55 g)z3 Load parameters from a 1D vector. :param vector: )dtyperRN)rYr utilsvector_to_parameters as_tensorfloatrRrQ)r7rgs r9load_from_vectorBaseModel.load_from_vectors>  ((fBHHUYU`U`)acgcrcrctur;c[RRRUR 55R 5R 5R5$)z2 Convert the parameters to a 1D vector. :return: )rYr rjparameters_to_vectorrQdetachrPnumpyrEs r9rqBaseModel.parameters_to_vectors? uu{{//0ABIIKOOQWWYYr;modec&URU5 g)z Put the policy in either training or evaluation mode. This affects certain modules, such as batch normalisation and dropout. :param mode: if true, set to training mode, else set to evaluation mode N)train)r7rus r9set_training_modeBaseModel.set_training_modes 4r; observationcSn[U[5(a[UR[R5(dSUR35eUR 5H<up4URRUnU=(d [ [XE5U5nM> U$[ [XR5UR5nU$)a6 Check whether or not the observation is vectorized, apply transposition to image (so that they are channel-first) if needed. This is used in DQN when sampling random action (epsilon-greedy policy) :param observation: the input observation to check :return: whether the given observation is vectorized or not F8The observation provided is a dict but the obs space is ) isinstancer6r(r Dictitemsr r)r7rzvectorized_envkeyrG obs_spaces r9r #BaseModel.is_vectorized_observations k4 ( (&&  cI$J`J`Iab c(--/ 2299#> !/!x3L_]`Mlnw3x07 -C-CDdF\F\Nr;cSn[U[5(a[UR[R5(dSUR35e[ R "U5nUR5Hup4URRUn[U5(a [XE5nO[R"U5nU=(d [Xe5nURS/URURQ75X'M OF[UR5(a[XR5nO[R"U5n[U[5(d>[XR5nURS/URRQ75n[XR 5nXr4$)a, Convert an input observation to a PyTorch tensor that can be fed to a model. Includes sugar-coating to handle different observations (e.g. normalizing images). :param observation: the input observation :return: The observation as PyTorch tensor and whether the observation is vectorized or not Fr|)r}r6r(r r~r@deepcopyrrrnparrayr reshapeshaper!rR)r7rzrrrGrobs_ obs_tensors r9 obs_to_tensorBaseModel.obs_to_tensorsx k4 ( (&&  cI$J`J`Iab c-- 4K'--/ 2299#> !),,*3:D88C=D!/!]3LT3]#'<<0Xd6L6LS6Q6W6W0X#Y  0D22 3 3*+7M7MNK((;/K+t,,6{DZDZ[N%--r.QD4J4J4P4P.QRK"; < ))r;)r)r,r*r+r-r(r.r/N)auto),__name__ __module__ __qualname____firstlineno____doc__rYoptim Optimizer__annotations__rAdamr Spacetyperrr6strrboolr3rBrArTensorrJrMpropertyrRrZ classmethodr"r rarndarrayrnrqrxr tupler__static_attributes__ __classcell__r8s@r9r#r#'s*xx!!! AQ>B>B!%46HHMM59O!<<OllO#''<"= O $,DcN#; O %%:; OObhh001O#4S>2OOH?CcN%%:; c3h ,g)>g 4J 4DY 4^`^g^g 4  T#s(^  ! !!eee$}%S% 3:O]j(vrzzvdvZbjjZdtU2::tCQSQ[Q[OG\;\5]bf.)*rzz4RZZ;P/P)Q)*V[\fhl\lVm)*)*r;cD^\rSrSr%Sr\\S'SS.S\4U4Sjjjr\ S\ S \ 4S j5r \ S \4S j5r \ SS \RS \ S S4Sjj5r\SS\S\S \R(4Sjj5rSS\\R0\\\R044S\\\R0S4S\\R0S\S \\R0\\\R0S444 SjjrS\R0S \R04SjrS\R0S \R04SjrSr U=r!$) BasePolicyia_The base policy object. Parameters are mostly the same as `BaseModel`; additions are documented below. :param args: positional arguments passed through to `BaseModel`. :param kwargs: keyword arguments passed through to `BaseModel`. :param squash_output: For continuous actions, whether the output is squashed or not using a ``tanh()`` function. r,F) squash_outputrc2>[TU]"U0UD6 Xlgr)r2r3_squash_output)r7rargskwargsr8s r9r3BasePolicy.__init__%s $)&)+r;progress_remainingr=cAg)z#(float) Useful for pickling policy.r`)rs r9_dummy_scheduleBasePolicy._dummy_schedule)s  r;cUR$)z (bool) Getter for squash_output.rrEs r9rBasePolicy.squash_output/s"""r;modulegainNc[U[R[R45(a\[RR UR US9 URb&URRRS5 ggg)z1 Orthogonal initialization (used in PPO and A2C) rNr) r}r LinearConv2dinit orthogonal_weightbiasrXfill_)rrs r9 init_weightsBasePolicy.init_weights4se fryy"))4 5 5 GG   D  9{{&   &&s+' 6r;rz deterministiccg)a\ Get the action according to the policy for a given observation. By default provides a dummy implementation -- not all BasePolicy classes implement this, e.g. if they are a Critic in an Actor-Critic method. :param observation: :param deterministic: Whether to use stochastic or deterministic actions :return: Taken action according to the policy Nr`r7rzrs r9_predictBasePolicy._predict>sr;state. episode_startcjURS5 [U[5(a2[U5S:Xa#[US[5(a [ S5eUR U5upV[R"5 URXTS9nSSS5 WR5R5RS/URRQ75n[UR[R 5(abUR"(aUR%U5nO?[&R("XpRR*URR,5nU(d0[U[&R.5(deUR1SS 9nXr4$!,(df  GN=f) ae Get the policy action from an observation (and optional hidden state). Includes sugar-coating to handle different observations (e.g. normalizing images). :param observation: the input observation :param state: The last hidden states (can be None, used in recurrent policies) :param episode_start: The last masks (can be None, used in recurrent policies) this correspond to beginning of episodes, where the hidden states of the RNN must be reset. :param deterministic: Whether or not to return deterministic actions. :return: the model's action and the next hidden state (used in recurrent policies) FaYou have passed a tuple to the predict() function instead of a Numpy array or a Dict. You are probably mixing Gym API with SB3 VecEnv API: `obs, info = env.reset()` (Gym) vs `obs = vec_env.reset()` (SB3 VecEnv). See related issue https://github.com/DLR-RM/stable-baselines3/issues/1694 and documentation for more information: https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecenv-api-vs-gym-apirNrr)axis)rxr}rlenr6 ValueErrorrrYno_gradrrPrsrr)rr Boxrunscale_actionrcliplowhighrsqueeze)r7rzrrrrractionss r9predictBasePolicy.predictKsR* u% k5 ) )c+.>!.C S^_`SacgHhHhW &*%7%7 %D" ZZ\mmJmLG++-%%'//0Nd6G6G6M6M0NO d'' 4 4!!--g6'''+<+<+@+@$BSBSBXBXYgrzz22 22oo1o-G~'\s F## F2actionc[UR[R5(dSUR35eURRURR p2SX- X2- - -S- $)z Rescale the action from [low, high] to [-1, 1] (no need for symmetric action space) :param action: Action to scale :return: Scaled action zETrying to scale an action using an action space that is not a Box(): g@?r}r)r rrr)r7rrrs r9 scale_actionBasePolicy.scale_actions}   vzz   g RSWSdSdRe f g %%))4+<+<+A+ATv| 34s::r; scaled_actionc[UR[R5(dSUR35eURRURR p2USUS--X2- --$)z Rescale the action from [-1, 1] to [low, high] (no need for symmetric action space) :param scaled_action: Action to un-scale zGTrying to unscale an action using an action space that is not a Box(): g?rr)r7rrrs r9rBasePolicy.unscale_actions   vzz   i TUYUfUfTg h i %%))4+<+<+A+ATc]S01TZ@AAr;rrF)NNF)"rrrrrrrrr3 staticmethodrmrrrr ModulerrrrYrrr rrr6rrrrrrrrrs@r9rrs.-49,T,,Ee #t##,RYY,e,D,,  J  t  PRPYPY    37.2# 72::tCO'<<=7bjj#o./7  + 7  7 rzz8E"**c/$:;; < 7r ;2:: ;"** ; BBJJ B2:: B Br;rc&^\rSrSrSrS\R SSSSSS\SSS\RRS4S\ RS\ RS \ S \\\\\\\\44S \\R*S \S \S\S\S\S\S\\S\\\\4S\S\S\\RR4S\\\\44"U4SjjjrS\\\44U4SjjrS-S\SS4SjjrS.SjrS \ SS4SjrS/S\R@S \S\!\R@\R@\R@44S!jjr"S0S\#S"\\S\\R@\!\R@\R@444U4S#jjjr$S$\R@S\%4S%jr&S/S&\#S \S\R@4S'jjr'S\#S(\R@S\!\R@\R@\\R@44S)jr(S\#S\%4U4S*jjr)S\#S\R@4U4S+jjr*S,r+U=r,$)1ActorCriticPolicyiaY Policy class for actor-critic algorithms (has both policy and value prediction). Used by A2C, PPO and the likes. :param observation_space: Observation space :param action_space: Action space :param lr_schedule: Learning rate schedule (could be constant) :param net_arch: The specification of the policy and value networks. :param activation_fn: Activation function :param ortho_init: Whether to use or not orthogonal initialization :param use_sde: Whether to use State Dependent Exploration or not :param log_std_init: Initial value for the log standard deviation :param full_std: Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE :param use_expln: Use ``expln()`` function instead of ``exp()`` to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, ``exp()`` is usually enough. :param squash_output: Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE. :param features_extractor_class: Features extractor to use. :param features_extractor_kwargs: Keyword arguments to pass to the features extractor. :param share_features_extractor: If True, the features extractor is shared between the policy and value networks. :param normalize_images: Whether to normalize images or not, dividing by 255.0 (True by default) :param optimizer_class: The optimizer to use, ``th.optim.Adam`` by default :param optimizer_kwargs: Additional keyword arguments, excluding the learning rate, to pass to the optimizer NTFrr(r) lr_schedulenet_arch activation_fn ortho_inituse_sde log_std_initfull_std use_explnrr*r+share_features_extractorr-r.r/c \>Uc%0nU[RR:XaSUS'[TU]UUU U UUU US9 [ U[ 5(aB[U5S:a3[ US[5(a[R"S5 USnUcU [:Xa/nO[SS/SS/S9nX@l XPl X`lXlUR!5UlUR"R$UlUR(a#UR"UlUR"UlO&UR"UlUR!5UlXlSnU (aU(dS5eU(aU U U S S .nXplUUl[1X'US 9UlUR5U5 g) Ngh㈵>eps)r.r/rr-rzAs shared layers in the mlp_extractor are removed since SB3 v1.8.0, you should now pass directly a dictionary and not a list (net_arch=dict(pi=..., vf=...) instead of net_arch=[dict(pi=..., vf=...)])@)pivfzCsquash_output=True is only available when using gSDE (use_sde=True)F)rrrlearn_features)r dist_kwargs)rYrrr2r3r}listrr6warningswarnrrrrrrAr,r?pi_features_extractorvf_features_extractorrrrr action_dist_build)r7r(r)rrrrrrrrrr*r+rr-r.r/rr8s r9r3ActorCriticPolicy.__init__s(  #! "((--/*. '    $ %+-'-   h % %#h-!*; 8TU;X\@]@] MMa  {H  '94B8R9 *$(@%"&">">"@ 33@@  ( ()-)@)@D &)-)@)@D &)-)@)@D &)-)E)E)GD &( !'y4yy2 $!.&"' K &3<^ij K r;r=c>[TU]5nUR=(d [R"S5nUR [ URURURURUSUSUSURURURURURUR S9 5 U$)Ncgrr`r`r;r9?ActorCriticPolicy._get_constructor_parameters..sRVr;rrr) rrrrrrrrrr.r/r*r+)r2rMr collections defaultdictr5r6rrrrrrr.r/r*r+)r7rXdefault_none_kwargsr8s r9rM-ActorCriticPolicy._get_constructor_parameterssw24"..W+2I2I,2W "00 !..1/B,Z8-k: 00?? $ 4 4!%!6!6)-)F)F*.*H*H  " r;n_envsc[UR[5(dS5eURRURUS9 g)z@ Sample new weights for the exploration matrix. :param n_envs: z/reset_noise() is only available when using gSDE) batch_sizeN)r}rrsample_weightslog_std)r7r s r9 reset_noiseActorCriticPolicy.reset_noise1sA $**,KLLNL '' 'Hr;cv[URURURURS9Ulg)zI Create the policy and value networks. Part of the layers can be shared. )rrrRN)rr?rrrR mlp_extractorrEs r9_build_mlp_extractor&ActorCriticPolicy._build_mlp_extractor:s1*   ]],,;;  r;c6UR5 URRn[UR[ 5(a1URR X RS9uUlUl O[UR[5(a2URR X"URS9uUlUl Ob[UR[[[45(aURR US9UlO[SURS35e[R "URR"S5UlUR&(aUR([*R,"S5UR[*R,"S5URSUR$S0nUR.(dPX0R( [*R,"S5X0R0'[*R,"S5X0R2'UR55H(upEUR7[9UR:US 95 M* UR<"UR?54S U"S50UR@D6Ul!g ) z Create the networks and the optimizer. :param lr_schedule: Learning rate schedule lr_schedule(1) is the initial learning rate ) latent_dimr)rlatent_sde_dimr)rzUnsupported distribution 'z'.rrg{Gz?rlrN)"rr latent_dim_pir}rrproba_distribution_netr action_netrrr rr NotImplementedErrorr r latent_dim_vf value_netrr,rsqrtrrrrapplyrrr.rQr/r')r7rr module_gainsrrs r9rActorCriticPolicy._buildIs !!#**88 d&&(@ A A,0,<,<,S,S(7H7H-T- )DOT\((*I J J,0,<,<,S,S(UYUfUf-T- )DOT\((+BD`bw*x y y"..EEQ^E_DO%(B4CSCSBTTV&WX X4#5#5#C#CQG ?? ''""BGGAJ L 00!!8!89;=771: 778;=771: 778 , 2 2 4  WT%6%6TBC!5--doo.?lKPQNlVZVkVklr;rGrcURU5nUR(aURU5upEO:UupgURRU5nURR U5nUR U5nUR U5n U RUS9n U RU 5n U RS/URRQ75n XU 4$)z Forward pass in all the networks (actor and critic) :param obs: Observation :param deterministic: Whether to sample or use deterministic actions :return: action, value and log probability of the action rr) rJrr forward_actorforward_criticr_get_action_dist_from_latent get_actionslog_probrr)r) r7rGrfeatures latent_pi latent_vf pi_features vf_featuresvalues distributionrr(s r9forwardActorCriticPolicy.forward|s((-  ( (#'#5#5h#? Iy'/ $K**88EI**99+FI *88C ***G((1//2"@(9(9(?(?"@A((r;r,c>UR(a"[TU] XcUR5$U5$Ub[R "S[ 5 [TU] XR5n[TU] XR5nX44$)a] Preprocess the observation if needed and extract features. :param obs: Observation :param features_extractor: The features extractor to use. If None, then ``self.features_extractor`` is used. :return: The extracted features. If features extractor is not shared, returns a tuple with the features for the actor and the features for the critic. zYProvided features_extractor will be ignored because the features extractor is not shared.) rr2rJr,rr UserWarningrr)r7rGr,r,r-r8s r9rJ"ActorCriticPolicy.extract_featuress  ( (7+CLf1H1H l~ !- o  '238R8RSK'238R8RSK+ +r;r*cURU5n[UR[5(a%URR X R 5$[UR[ 5(aURR US9$[UR[5(aURR US9$[UR[5(aURR US9$[UR[5(a&URR X R U5$[S5e)z Retrieve action distribution given the latent codes. :param latent_pi: Latent code for the actor :return: Action distribution ) action_logitszInvalid action distribution) rr}rrproba_distributionrr rr rr)r7r* mean_actionss r9r&.ActorCriticPolicy._get_action_dist_from_latents y1 d&&(@ A A##66|\\R R ((*A B B##66\6R R ((*F G G##66\6R R ((*? @ @##66\6R R ((*I J J##66|\\S\] ]:; ;r;rzc>URU5RUS9$)z Get the action according to the policy for a given observation. :param observation: :param deterministic: Whether to use stochastic or deterministic actions :return: Taken action according to the policy r)get_distributionr'rs r9rActorCriticPolicy._predicts#$$[1==M=ZZr;rcpURU5nUR(aURU5upEO:UupgURRU5nURR U5nUR U5nUR U5n URU5n UR5n XU 4$)z Evaluate actions according to the current policy, given the observations. :param obs: Observation :param actions: Actions :return: estimated value, log likelihood of taking those actions and entropy of the action distribution. ) rJrrr$r%r&r(rentropy) r7rGrr)r*r+r,r-r/r(r.r>s r9evaluate_actions"ActorCriticPolicy.evaluate_actionss((-  ( (#'#5#5h#? Iy'/ $K**88EI**99+FI88C ((1 *&&(((r;c>[TU]XR5nURR U5nUR U5$)zl Get the current policy distribution given the observations. :param obs: :return: the action distribution. )r2rJrrr$r&)r7rGr)r*r8s r9r;"ActorCriticPolicy.get_distributionsA7+C1K1KL&&44X> 00;;r;c>[TU]XR5nURR U5nUR U5$)z Get the estimated values according to the current policy given the observations. :param obs: Observation :return: the estimated values. )r2rJrrr%r)r7rGr)r+r8s r9predict_values ActorCriticPolicy.predict_valuess?7+C1K1KL&&55h? ~~i((r;)rrrrr?r,rrrrr'rrrrrrr)r=Nrr)-rrrrrr TanhrrYrrr rrrr rintr6rrrrrmrrrr3rMrrrrrr0rrJrr&rr?r;rDrrrs@r9rrsHFJ)+!#@P>B)-!%46HHMM59%W!!<<W!llW! W! 5cDd3i,@!@AB W! BII W!W!W!W!W!W!W!#''<"=W!$,DcN#;W!#'W! !W!"bhh001#W!$#4S>2%W!W!rT#s(^0I#IdI  1m(1mt1mf)299)T)eBIIWYW`W`bdbkbkLkFl)2VZ,,3;[TU]UUUUUUUUU U U U U UUUU5 grr2r3r7r(r)rrrrrrrrrr*r+rr-r.r/r8s r9r3ActorCriticCnnPolicy.__init__E(             $ % $   # r;r`)rrrrrr rFrrYrrr rrrr rrGr6rrrrrmrrrr3rrrs@r9rIrIsTHFJ)+!#@I>B)-!%46HHMM59%& !<<& ll&  & 5cDd3i,@!@AB & BII & & & & & & & #''<"=& $,DcN#;& #'& !& "bhh001#& $#4S>2%& & r;rIc&^\rSrSrSrS\R SSSSSS\SSS\RRS4S\ RS\ RS \S \\\\\\\\44S \\R,S \S \S\S\S\S\S\\S\\\\4S\S\S\\RR6S\\\\44"U4SjjjrSrU=r$)MultiInputActorCriticPolicyiGav MultiInputActorClass policy class for actor-critic algorithms (has both policy and value prediction). Used by A2C, PPO and the likes. :param observation_space: Observation space (Tuple) :param action_space: Action space :param lr_schedule: Learning rate schedule (could be constant) :param net_arch: The specification of the policy and value networks. :param activation_fn: Activation function :param ortho_init: Whether to use or not orthogonal initialization :param use_sde: Whether to use State Dependent Exploration or not :param log_std_init: Initial value for the log standard deviation :param full_std: Whether to use (n_features x n_actions) parameters for the std instead of only (n_features,) when using gSDE :param use_expln: Use ``expln()`` function instead of ``exp()`` to ensure a positive standard deviation (cf paper). It allows to keep variance above zero and prevent it from growing too fast. In practice, ``exp()`` is usually enough. :param squash_output: Whether to squash the output using a tanh function, this allows to ensure boundaries when using gSDE. :param features_extractor_class: Uses the CombinedExtractor :param features_extractor_kwargs: Keyword arguments to pass to the features extractor. :param share_features_extractor: If True, the features extractor is shared between the policy and value networks. :param normalize_images: Whether to normalize images or not, dividing by 255.0 (True by default) :param optimizer_class: The optimizer to use, ``th.optim.Adam`` by default :param optimizer_kwargs: Additional keyword arguments, excluding the learning rate, to pass to the optimizer NTFrr(r)rrrrrrrrrr*r+rr-r.r/cD>[TU]UUUUUUUUU U U U U UUUU5 grrKrLs r9r3$MultiInputActorCriticPolicy.__init__grNr;r`)rrrrrr rFrrYrrr r~rrrr rrGr6rrrrrmrrrr3rrrs@r9rPrPGsTHFJ)+!#@Q>B)-!%46HHMM59%& !;;& ll&  & 5cDd3i,@!@AB & BII & & & & & & & #''<"=& $,DcN#;& #'& !& "bhh001#& $#4S>2%& & r;rPcv^\rSrSr%Sr\\S'\RSSS4S\ RS\ RS\ \ S\S \ S \\RS \S \ S \4U4SjjjrS\R&S\R&S\\R&S44SjrS\R&S\R&S\R&4SjrSrU=r$)ContinuousCriticia{ Critic network(s) for DDPG/SAC/TD3. It represents the action-state value function (Q-value function). Compared to A2C/PPO critics, this one represents the Q-value and takes the continuous action as input. It is concatenated with the state and then fed to the network which outputs a single value: Q(s, a). For more recent algorithms like SAC/TD3, multiple networks are created to give different estimates. By default, it creates two critic networks used to reduce overestimation thanks to clipped Q-learning (cf TD3 paper). :param observation_space: Observation space :param action_space: Action space :param net_arch: Network architecture :param features_extractor: Network to extract features (a CNN when using images, a nn.Flatten() layer otherwise) :param features_dim: Number of features :param activation_fn: Activation function :param normalize_images: Whether to normalize images or not, dividing by 255.0 (True by default) :param n_critics: Number of critic networks to create. :param share_features_extractor: Whether the features extractor is shared or not between the actor and the critic (this saves computation time) r,Trr(r)rr?rr- n_criticsrc >>[TU]UUUUS9 [UR5n XlXl/Ul[U5HUn [XZ-SX65n [R"U 6n URSU 3U 5 UR RU 5 MW g)N)r,r-rqf) r2r3rr)rrU q_networksrangerr Sequential add_moduleappend)r7r(r)rr,r?rr-rUr action_dimidx q_net_listq_netr8s r9r3ContinuousCritic.__init__s   1-  $D$5$56 (@%"+-#C#L$=q(ZJMM:.E OObJ . OO " "5 ) $r;rGrr=.c$^[R"UR(+5 URXR5nSSS5 [R "WU/SS9m[ U4SjUR55$!,(df  ND=f)Nrdimc32># UH o"T5v M g7frr`).0r` qvalue_inputs r9 +ContinuousCritic.forward..sFoUU<((os)rYset_grad_enabledrrJr,catrrX)r7rGrr)rgs @r9r0ContinuousCritic.forwardsl T%B%B!B C,,S2I2IJHDvvx1q9 FdooFFFD Cs B Bc[R"5 URXR5nSSS5 URS"[R "WU/SS95$!,(df  N8=f)z Only predict the Q-value using the first network. This allows to reduce computation when all the estimates are not needed (e.g. when updating the policy in TD3). Nrrrc)rYrrJr,rXrk)r7rGrr)s r9 q1_forwardContinuousCritic.q1_forwardsT ZZ\,,S2I2IJHq!"&&(G)=RGFbiiF"))F FFr;rT)8rrr@rabcrr functoolsrtypingrrrr rsrtorchrY gymnasiumr r &stable_baselines3.common.distributionsr r rrrrr&stable_baselines3.common.preprocessingrrrr%stable_baselines3.common.torch_layersrrrrrr%stable_baselines3.common.type_aliasesrrstable_baselines3.common.utilsrr r!r"rr#rrrIrPrTr`r;r9r{sA #00srG__{; n* n*bEBCEBP[) [)| F ,F RF "3F RKFyKFr;