h_ `SSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r SSK J r Jr SSKrSSKrSSKrSSKrSSKJr SSKrSSKJr SSKJrJr SS KJ r J!r!J"r"J#r#J$r$ SMS \%S \&S S4S jjr'S\RPS\RPS \)4Sjr*S\RVRXS\)S S4Sjr-"SS5r."SS5r/"SS5r0S\\!\)4S \!4Sjr1S\)S\)S\)S \!4Sjr2S \)S \!4S!jr3SNS"\\Rh\54S \Rh4S#jjr6SOS$\5S%\5S \%4S&jjr7SPS'\%S(\ \5S)\5S*\&S \4 S+jjr8S,\ S-\RrS.\RrS S4S/jr:S0\RrS1\RrS S4S2jr;S3\RPS-\RxS \&4S4jr=S3\\%\RP4S-\R|S \&4S5jr?S3\RPS-\RS \&4S6jrAS3\RPS-\RS \&4S7jrCS3\RPS-\RS \&4S8jrES3\\%\RP4S-\RrS \&4S9jrFS:\\RP\G\4S \)4S;jrHS<\RRS=\ \5S \G\R4S>jrLS?\ S \ 4S@jrMSA\ \RSB\ \RSC\)S S4SDjrNSE\\RP\O\5\RP44S"\RhS \\R\"44SFjrPSG\#SH\%SI\%S \&4SJjrQSQSK\&S \R\O\5\54\544SLjjrSg!\a SrGNf=f)RN)deque)Iterable) zip_longest)OptionalUnion)spaces) SummaryWriter)Logger configure)GymEnvSchedule TensorDict TrainFreqTrainFrequencyUnitseed using_cudareturnc&[R"U5 [RRU5 [R"U5 U(a?S[R R lS[R R lgg)zH Seed the different random generators. :param seed: :param using_cuda: TFN) randomrnpth manual_seedbackendscudnn deterministic benchmark)rrs X/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/utils.pyset_random_seedrsX KKIINN4NN4*. '&+ #y_predy_truecURS:XaURS:Xde[R"U5nUS:Xa[R$[ S[R"X- 5U- - 5$)aZ Computes fraction of variance that ypred explains about y. Returns 1 - Var[y-ypred] / Var[y] interpretation: ev=0 => might as well have predicted zero ev=1 => perfect prediction ev<0 => worse than just predicting zero :param y_pred: the prediction :param y_true: the expected value :return: explained variance of ypred and y r)ndimrvarnanfloat)r r!var_ys rexplained_variancer)3s] ;;!  q 00 0 FF6NEaZ266OU1rvvfo/F/N+N%OOr optimizer learning_ratec2URHnXS'M g)z Update the learning rate for a given optimizer. Useful when doing linear schedule. :param optimizer: Pytorch optimizer :param learning_rate: New learning rate value lrN) param_groups)r*r+ param_groups rupdate_learning_rater0Fs!-- )D.rcP\rSrSrSrS\\\44SjrS\S\4Sjr S\ 4Sjr S r g ) FloatScheduleRz Wrapper that ensures the output of a Schedule is cast to float. Can wrap either a constant value or an existing callable Schedule. :param value_schedule: Constant value or callable schedule (e.g. LinearSchedule, ConstantSchedule) value_schedulec[U[5(aURUlg[U[[45(a[ [U55Ulg[ U5(d SU35eXlg)Nz>The learning rate schedule must be a float or a callable, not ) isinstancer2r4r'intConstantSchedulecallable)selfr4s r__init__FloatSchedule.__init__[sf nm 4 4,:,I,ID   5 5"253H"ID N++ ~/mn|m}-~ ~+"0 rprogress_remainingrc6[URU55$N)r'r4r:r=s r__call__FloatSchedule.__call__dsT(();<==rc"SURS3$)NzFloatSchedule()r4r:s r__repr__FloatSchedule.__repr__is 3 34A66rrEN) __name__ __module__ __qualname____firstlineno____doc__rr r'r;rAstrrG__static_attributes__rrr2r2Rs;1uXu_'=1>5>U> 7#7rr2cR\rSrSrSrS\S\S\SS4SjrS \S\4S jrS\4S jr S r g) LinearSchedulema LinearSchedule interpolates linearly between start and end between ``progress_remaining`` = 1 and ``progress_remaining`` = ``end_fraction``. This is used in DQN for linearly annealing the exploration fraction (epsilon for the epsilon-greedy strategy). :param start: value to start with if ``progress_remaining`` = 1 :param end: value to end with if ``progress_remaining`` = 0 :param end_fraction: fraction of ``progress_remaining`` where end is reached e.g 0.1 then end is reached after 10% of the complete training process. startend end_fractionrNc(XlX lX0lgr?rTrUrV)r:rTrUrVs rr;LinearSchedule.__init__zs (rr=cSU- UR:a UR$URSU- URUR- -UR- -$Nr#)rVrUrTr@s rrALinearSchedule.__call__sR " "d&7&7 788O::%7!7DHHtzz[T"U55$r?)r')r=r4s r!get_schedule_fn..seN;M,N&Or)warningswarnr6r'r7 constant_fnr9rEs`rget_schedule_fnrrsN MMWX.5#,//$U>%:;'''' POrrTrUrVcd^^^[R"S5 S[S[4UUU4SjjnU$)aE Create a function that interpolates linearly between start and end between ``progress_remaining`` = 1 and ``progress_remaining`` = ``end_fraction``. This is used in DQN for linearly annealing the exploration fraction (epsilon for the epsilon-greedy strategy). :params start: value to start with if ``progress_remaining`` = 1 :params end: value to end with if ``progress_remaining`` = 0 :params end_fraction: fraction of ``progress_remaining`` where end is reached e.g 0.1 then end is reached after 10% of the complete training process. :return: Linear schedule function. zBget_linear_fn() is deprecated, please use LinearSchedule() insteadr=rc<>SU- T:aT$TSU- TT- -T- -$r[rP)r=rUrVrTs rfuncget_linear_fn..funcs5 " "l 2JA 22sU{ClRR Rr)rorpr')rTrUrVrus``` r get_linear_fnrws5 MMVWSS5SS Krrbc@^[R"S5 U4SjnU$)z Create a function that returns a constant It is useful for learning rate schedule (to avoid code duplication) :param val: constant value :return: Constant schedule function. zBconstant_fn() is deprecated, please use ConstantSchedule() insteadc>T$r?rP)rfrbs rruconstant_fn..funcs r)rorp)rbrus` rrqrqs MMVW KrdevicecUS:XaSn[R"U5nUR[R"S5R:Xa9[RR 5(d[R"S5$U$)z Retrieve PyTorch device. It checks that the requested device is available first. For now, it supports only cpu and cuda. By default, it tries to use the gpu. :param device: One for 'auto', 'cuda', 'cpu' :return: Supported Pytorch device autocudacpu)rr{typer~ is_availabler{s r get_devicersb YYv F{{bii',,,RWW5I5I5K5Kyy Mrlog_pathlog_namecSn[R"[RRU[R"U5S355HnUR [R 5SnUR S5SnUSRUR S5SS5:XdMbUR5(dMy[U5U:dM[U5nM U$)aJ Returns the latest run number for the given log name and log path, by finding the greatest number in the directories. :param log_path: Path to the log folder containing several runs. :param log_name: Name of the experiment. Each run is stored in a folder named ``log_name_1``, ``log_name_2``, ... :return: latest run number rz_[0-9]*rfN) globospathjoinescapesplitsepisdigitr7)rr max_run_idr file_nameexts rget_latest_run_idrsJ "'',,xDKK4I3J'1RSTJJrvv&r* ooc"2& sxx  4Sb 9: :s{{}}QTUXQY\fQfSJ U rverbosetensorboard_log tb_log_namereset_num_timestepscSS/pTUb[c [S5eUbT[bM[X5nU(dUS-n[RR XSUS-35nUS:aSS/nO S/nO US:XaS/n[ XES 9$) a  Configure the logger's outputs. :param verbose: Verbosity level: 0 for no output, 1 for the standard output to be part of the logger outputs :param tensorboard_log: the log location for tensorboard (if None, no logging) :param tb_log_name: tensorboard log :param reset_num_timesteps: Whether the ``num_timesteps`` attribute is reset or not. It allows to continue a previous learning curve (``reset_num_timesteps=False``) or start from t=0 (``reset_num_timesteps=True``, the default). :return: The logger object NstdoutzCTrying to log data to tensorboard but tensorboard is not installed.r#rf tensorboardr)format_strings)r ImportErrorrrrrr )rrrr save_pathr latest_run_ids rconfigure_loggerr s"!%xj~"}'<_``"}'@)/G " Q MGGLLM=STCTBU2VW a<& 6N+_N A Y >>renvobservation_space action_spacecXR:wa[SUSUR35eX R:wa[SUSUR35eg)ap Checks that the environment has same spaces as provided ones. Used by BaseAlgorithm to check if spaces match after loading the model with given env. Checked parameters: - observation_space - action_space :param env: Environment to check for valid spaces :param observation_space: Observation space to check against :param action_space: Action space to check against z!Observation spaces do not match:  != zAction spaces do not match: N)r ValueErrorr)rrrs rcheck_for_correct_spacesr0sd111<=N-D-D,EQG H9@@3sTeTkTkKlAmn o  rc[U[5(dURS:Xag[UR5S:Xag[ SURS3S-5e)a For discrete observation type, detects and validates the shape, then returns whether or not the observation is vectorized. :param observation: the input observation to validate :param observation_space: the observation space :return: whether the given observation is vectorized or not rPFr#TrrzJDiscrete environment, please use () or (n_env,) for the observation shape.)r6r7rlenrrs r"is_vectorized_discrete_observationrks`+s##{'8'8B'> [   1 $2;3D3D2EU KZ [  rcfUR[UR54:Xag[UR5S:Xa'URS[UR5:Xag[SURS3S[UR5S3-S [UR5S 3-5e) a$ For multidiscrete observation type, detects and validates the shape, then returns whether or not the observation is vectorized. :param observation: the input observation to validate :param observation_space: the observation space :return: whether the given observation is vectorized or not Fr#Trz for MultiDiscrete zenvironment, please use (z,) or (n_env, ) for the observation shape.)rrnvecrrs r'is_vectorized_multidiscrete_observationrsS!2!7!78:: [   1 $):):1)=EVE[E[A\)\2;3D3D2EEX Y)#.?.D.D*E)FfM N.33455QR S  rcHURUR:Xag[UR5[UR5S-:XaURSSUR:Xag[SURS3SURS3-S URS 3-5e) a" For multibinary observation type, detects and validates the shape, then returns whether or not the observation is vectorized. :param observation: the input observation to validate :param observation_space: the observation space :return: whether the given observation is vectorized or not Fr#NTrz for MultiBinary zenvironment, please use z or rr)rrrnrs r%is_vectorized_multibinary_observationrs-333 [   3'8'>'>#?!#C C HYHYZ[Z\H]araxaxHx2;3D3D2EEV W():)@)@(AF G*,,--IJ K  rc SnURR5H%up4XRUR:wdM#Sn O U(agSnURR5H(up4XRSSUR:wdM&Sn O U(agSn[UWURU5 [ SUWRSUSURUS U35e![a nUnSnANCSnAff=f) a For dict observation type, detects and validates the shape, then returns whether or not the observation is vectorized. :param observation: the input observation to validate :param observation_space: the observation space :return: whether the given observation is vectorized or not TFr#NrzdThere seems to be a mix of vectorized and non-vectorized observations. Unexpected observation shape z for key z of type z. )ritemsris_vectorized_observationr)rrall_non_vectorizedrsubspaceall_vectorized error_msges ris_vectorized_dict_observationrs(*11779    ! !X^^ 3!&   :N*11779    ! !!" % 7"N :    %k#&68I8P8PQT8U V,,7,<,B,B+C9SER(//45R { D   #I s!C.. D8DDc R[R[[R[[R [ [R[[R[0nUR5Hup4[X5(dMU"X5s $ [SUS35e)a For every observation type, detects and validates the shape, then returns whether or not the observation is vectorized. :param observation: the input observation to validate :param observation_space: the observation space :return: whether the given observation is vectorized or not zMError: Cannot determine if the observation is vectorized with the space type .)rrrDiscreter MultiDiscreter MultiBinaryrrrrr6r)rris_vec_obs_func_dict space_typeis_vec_obs_funcs rrrs  1;EA 3 (<'A'A'C# ' 4 4";B B(D hizh{{|}~~rarrc~[U5S:Xa[R$[[R"U55$)z Compute the mean of an array if there is at least one element. For empty array, return NaN. It is used for logging only. :param arr: Numpy array or list of values :return: r)rrr&r'mean)rs r safe_meanrs*X]266;bggcl(;;rmodelincluded_namesc UR5R5VVVs/sH,up#[UVs/sHoDU;PM sn5(dM*UPM. snnn$s snfs snnnf)a= Extract parameters from the state dict of ``model`` if the name contains one of the strings in ``included_names``. :param model: the model where the parameters come from. :param included_names: substrings of names to include. :return: List of parameters values (Pytorch tensors) that matches the queried names. ) state_dictrany)rrnameparamrs rget_parameters_by_namersQ&+%5%5%7%=%=%? p%?kd3_mGn_mX[t _mGnCoE%? ppGn psA A A A A  iterablesc'l# [5n[USU06HnX;a [S5eUv M g7f)z ``zip()`` function but enforces that iterables are of equal length. Raises ``ValueError`` if iterables not of equal length. Code inspired by Stackoverflow answer for question #32954486. :param \*iterables: iterables to ``zip()`` fillvaluez Iterables have different lengthsN)objectrr)rsentinelcombos r zip_strictrs9xHi<8<  ?@ @ =s24params target_paramstauc .[R"5 [X5HWup4URR SU- 5 [R "URURX$RS9 MY SSS5 g!,(df  g=f)as Perform a Polyak average update on ``target_params`` using ``params``: target parameters are slowly updated towards the main parameters. ``tau``, the soft update coefficient controls the interpolation: ``tau=1`` corresponds to copying the parameters to the target ones whereas nothing happens when ``tau=0``. The Polyak update is done in place, with ``no_grad``, and therefore does not create intermediate tensors, or a computation graph, reducing memory cost and improving performance. We scale the target params by ``1-tau`` (in-place), add the new weights, scaled by ``tau`` and store the result of the sum in the target params (in place). See https://github.com/DLR-RM/stable-baselines3/issues/93 :param params: parameters to use to update the target params :param target_params: parameters to update :param tau: the soft update coefficient ("Polyak update", between 0 and 1) r#)alphaoutN)rno_gradrdatamul_add)rrrr target_params r polyak_updatersb( #-f#D E    " "1s7 + FF<$$ejjARAR S$E s A'B Bobsc <[U[R5(a[R"XS9$[U[ 5(a8UR 5VVs0sHup#U[R"X1S9_M snn$[S[U535es snnf)z Moves the observation to the given device. :param obs: :param device: PyTorch device :return: PyTorch tensor of the observation on a desired device. rz!Unrecognized type of observation ) r6rndarrayr as_tensordictr Exceptionr)rr{r_obss r obs_as_tensorr4sy#rzz""||C// C  JM))+V+;CR\\$66+VV;DI;GHHWs!B train_freqnum_collected_stepsnum_collected_episodescUR[R:XaXR:$UR[R:XaX R:$[ SURS35e)a Helper used in ``collect_rollouts()`` of off-policy algorithms to determine the termination condition. :param train_freq: How much experience should be collected before updating the policy. :param num_collected_steps: The number of already collected steps. :param num_collected_episodes: The number of already collected episodes. :return: Whether to continue or not collecting experience by doing rollouts of the current policy. zgThe unit of the `train_freq` must be either TrainFrequencyUnit.STEP or TrainFrequencyUnit.EPISODE not 'z'!)unitrSTEP frequencyEPISODEr)rrrs rshould_collect_more_stepsrDsn,111"%9%999 .66 6%(<(<<< 22*?qAQAQAS@T(UV))+ __>>277//12"..__ H  z'='=>?Lnn& "SEE7"-- ' l  !!    s !D)) D65D6)F)r})rr)rNrT)T)Trrrrrro collectionsrcollections.abcr itertoolsrtypingrrr gymnasiumrnumpyrtorchrrstable_baselines3rtorch.utils.tensorboardr rstable_baselines3.common.loggerr r %stable_baselines3.common.type_aliasesr r rrrr7boolrrr'r)optim Optimizerr0r2rRr8rrrwrqr{rNrrrSpacerrrrrrrrrrrrrrrnnModuleTensorrrrrrrtupler rPrrr4sS  $!"5>mm,#,4,D,*PrzzP2::P%P& *BHH$6$6 *u *QU *776gg833.PE(E/$:PxP*U%H2 U x &uRYY^,")).C(%) $ "? "?c]"?"? "?  "?J^&^V\\^Y_YeYe^jn^$rfllrFLLrTr( rzz fjj ]a * E#rzz/4J _e_n_n sw (  X^XlXl qu * rzz V\VhVh mq *)  ) v{{) _c) X5bjj+AV\VbVbgk4<5T501>? I IW\]_]f]fhr]rWs I        8 " "d38nc6I0J "SMsL!!L-,L-