hBSSKrSSKJrJr SSKJr SSKJrJrJ r SSK r SSK r SSKJr SSKJrJr SSKJrJrJrJr SSKJr SS KJr SSKr"S S \5r"S S \5r"SS\5r "SS\5r!"SS\ 5r""SS\5r#g!\a SrNMf=f)N)ABCabstractmethod) Generator)AnyOptionalUnion)spaces)get_action_dim get_obs_shape)DictReplayBufferSamplesDictRolloutBufferSamplesReplayBufferSamplesRolloutBufferSamples) get_device) VecNormalizec ^\rSrSr%Sr\R \S'\\ S4\S'SS\ S\R S\R S\ \ R\ 4S \ 4 U4S jjjr\S \R"S \R"4S j5rS \ 4SjrS SjrS SjrS SjrS!S\ S\\4Sjjr\S!S\R"S\\S \ \\44Sjj5rS"S\R"S\S \ R>4Sjjr \S!S\ \R"\!\ \R"44S\\S \ \R"\!\ \R"444Sjj5r"\S!S\R"S\\S \R"4Sjj5r#Sr$U=r%$)# BaseBuffera6 Base class that represent a buffer (rollout or replay) :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param device: PyTorch device to which the values will be converted :param n_envs: Number of parallel environments observation_space. obs_shape buffer_size action_spacedevicen_envsc>[TU]5 XlX lX0l[ U5Ul[U5UlSUl SUl [U5Ul XPl g)NrF)super__init__rrrr rr action_dimposfullrrr)selfrrrrr __class__s Z/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/common/buffers.pyrBaseBuffer.__init__*sZ &!2(&'89(6  (  arrreturncURn[U5S:a/UQSP7nURSS5R"USUS-/USSQ76$)z Swap and then flatten axes 0 (buffer_size) and 1 (n_envs) to convert shape from [n_steps, n_envs, ...] (when ... is the shape of the features) to [n_steps * n_envs, ...] (which maintain the order) :param arr: :return: rN)shapelenswapaxesreshape)r&r,s r#swap_and_flattenBaseBuffer.swap_and_flatten>sY  u:>eKQKE||Aq!))%(U1X*=Jab JJr%cTUR(a UR$UR$)z) :return: The current size of the buffer )r rrr!s r#sizeBaseBuffer.sizeMs  99## #xxr%c[5e)z Add elements to the buffer. NotImplementedError)r!argskwargss r#addBaseBuffer.addUs "##r%c@[U6HnUR"U6 M g)z. Add a new batch of transitions to the buffer N)zipr;)r!r9r:datas r#extendBaseBuffer.extend[s JD HHdOr%c SUlSUlg)z Reset the buffer. rFN)rr r3s r#resetBaseBuffer.resetcs r% batch_sizeenvcUR(a URO URn[RR SX1S9nUR XBS9$)z :param batch_size: Number of element to sample :param env: associated gym VecEnv to normalize the observations/rewards when sampling :return: rr4rF)r rrnprandomrandint _get_samples)r!rErF upper_bound batch_indss r#sampleBaseBuffer.samplejsH+/))d&& YY&&q+&G    55r%rOc[5e)z) :param batch_inds: :param env: :return: r7)r!rOrFs r#rMBaseBuffer._get_samplesus "##r%arraycopycU(a[R"XRS9$[R"XRS9$)a  Convert a numpy array to a PyTorch tensor. Note: it copies the data by default :param array: :param copy: Whether to copy or not the data (may be useful to avoid changing things by reference). This argument is inoperative if the device is not the CPU. :return: )r)thtensorr as_tensor)r!rTrUs r#to_torchBaseBuffer.to_torchs- 99U;;7 7||E++66r%obsc.UbURU5$U$N) normalize_obs)r\rFs r#_normalize_obsBaseBuffer._normalize_obss ?$$S) ) r%rewardchUb.URU5R[R5$U$r^)normalize_rewardastyperJfloat32)rbrFs r#_normalize_rewardBaseBuffer._normalize_rewards, ?''/66rzzB B r%) rrrrr rrrr)autor*r'Nr^)T)&__name__ __module__ __qualname____firstlineno____doc__r Space__annotations__tupleintrrWrstrr staticmethodrJndarrayr0r4r;r@rCrrrPrrrrMboolTensorrZdictr`rg__static_attributes__ __classcell__r"s@r#rrs ||#S#X)/ "<<ll  biin%   ( Kbjj KRZZ K Kc$  6 68L+A 6DH$**$+3L+A$ "$88 9$$ 7bjj 7 7 7'+ 2::tCO44 5 l # rzz4RZZ00 1"**8L3IUWU_U_r%rc^\rSrSr%Sr\R \S'\R \S'\R \S'\R \S'\R \S'\R \S'S"S \S \ RS \ RS \ \ R\4S \S\S\4U4SjjjrS\R S\R S\R S\R S\R S\\\\4SS4SjrS#S\S\\S\4U4SjjjrS#S\R S\\S\4Sjjr\S\R6R8S\R6R84S j5rS!rU=r$)$ ReplayBufferas Replay buffer used in off-policy algorithms like SAC/TD3. :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param device: PyTorch device :param n_envs: Number of parallel environments :param optimize_memory_usage: Enable a memory efficient variant of the replay buffer which reduces by almost a factor two the memory used, at a cost of more complexity. See https://github.com/DLR-RM/stable-baselines3/issues/37#issuecomment-637501195 and https://github.com/DLR-RM/stable-baselines3/pull/28#issuecomment-637559274 Cannot be used in combination with handle_timeout_termination. :param handle_timeout_termination: Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284 observationsnext_observationsactionsrewardsdonestimeoutsrrrrroptimize_memory_usagehandle_timeout_terminationc>[T U]XX4US9 [X-S5Ul[b[R "5R nU(aU(a [S5eX`l[R"URUR/URQ7URS9UlU(dG[R"URUR/URQ7URS9Ul[R"URURUR 4UR#UR5S9Ul[R"URUR4[R&S9Ul[R"URUR4[R&S9UlXpl[R"URUR4[R&S9Ul[bURR0UR$R0-UR(R0-UR*R0-n U(dXRR0- n U W:a*U S-n US-n[2R4"SU SSUSS 35 ggg) Nrr*zpReplayBuffer does not support optimize_memory_usage = True and handle_timeout_termination = True simultaneously.dtypeeAWThis system does not have apparently enough memory to store the complete replay buffer .2fGB > GB)rrmaxrpsutilvirtual_memory available ValueErrorrrJzerosrrrrrr_maybe_cast_dtyperrfrrrrnbyteswarningswarn) r!rrrrrrr mem_availabletotal_memory_usager"s r#rReplayBuffer.__init__sA V\]{4a8  "113==M !%?H &;"HHd&6&6 %Udnn%U]n]t]tu$%'XXt/?/?.^t~~.^fwf}f}%~D "xx   t{{DOO +>>ATATTW[WaWaWhWhh )"&<&<&C&CC"!M1"c)"$  %%7$[R"U5URURS-UR-'O-[R"U5URUR'[R"U5URUR'[R"U5UR UR'[R"U5UR"UR'UR$(aM[R"UVs/sHowR'SS5PM sn5UR(UR'U=RS- sl URUR:XaSUlSUl ggs snf)Nr*TimeLimit.truncatedFTr) isinstancerr Discreter/rrrrJrTrrrrrrrrrgetrr )r!r\rrrbrrinfos r#r;ReplayBuffer.adds d,,foo > >++t{{?')hhsm$((#  % %CE88HCUD  txx!|t/?/?? @/1xx/AD " "488 ,!#&!1 TXX!#&!1 TXX!xx~ 488  * *&(hhch/ich[_9NPU0Vch/i&jDMM$(( # A  88t'' 'DIDH (0js8I0rErFcZ>UR(d [TU] XS9$UR(aD[R R SURUS9UR-UR-nO)[R R SURUS9nURX2S9$)az Sample elements from the replay buffer. Custom sampling when using memory efficient variant, as we should not sample the element with index `self.pos` See https://github.com/DLR-RM/stable-baselines3/pull/28#issuecomment-637559274 :param batch_size: Number of element to sample :param env: associated gym VecEnv to normalize the observations/rewards when sampling :return: rErFr*rHrrI) rrrPr rJrKrLrrrM)r!rErFrOr"s r#rPReplayBuffer.samples))7>Z>A A 99))++At/?/?j+QTXT\T\\`d`p`ppJ**1dhhZ*HJ   55r%rOc [RRSUR[ U54S9nUR (a5UR URUS-UR-USS24U5nO#UR URXSS24U5nUR URXSS24U5URXSS24UURX4SURX4- -RSS5URURX4RSS5U54n[![#[%UR&U556$)Nrhighr4r*)rJrKrLrr-rr`rrrrrrr/rgrrrrmaprZ)r!rOrF env_indicesrr?s r#rMReplayBuffer._get_samples3sHii'' 3z?BT'U  % %**4+<+[TU]XX4US9 XPlX`lSUlUR 5 g)NrF)rrrrgenerator_readyrC r!rrrrrrrr"s r#rRolloutBuffer.__init__ws5 V\]$ $ r%r'Nc@>[R"URUR/URQ7[R S9Ul[R"URURUR4[R S9Ul[R"URUR4[R S9Ul [R"URUR4[R S9Ul [R"URUR4[R S9Ul [R"URUR4[R S9Ul [R"URUR4[R S9Ul [R"URUR4[R S9UlSUl[ TU]E5 gNrF)rJrrrrrfrrrrrrrrrrrrC)r!r"s r#rCRolloutBuffer.resetsSHHd&6&6 %Udnn%U]_]g]ghxx!1!14;; PXZXbXbc xx!1!14;; ?rzzR xx!1!14;; ?rzzR  hh(8(8$++'FbjjYhh 0 0$++>bjjQ 4#3#3T[["AT((D$4$4dkk#B"**U$  r% last_valuesrcUR5R5R5R5nSn[ [ UR 55HnX@R S- :Xa%SUR[R5- nUnO'SURUS-- nURUS-nURUURU-U--URU- nXpRUR-U-U--nX0RU'M URUR-Ulg)aR Post-processing step: compute the lambda-return (TD(lambda) estimate) and GAE(lambda) advantage. Uses Generalized Advantage Estimation (https://arxiv.org/abs/1506.02438) to compute the advantage. To obtain Monte-Carlo advantage estimate (A(s) = R - V(S)) where R is the sum of discounted reward with value bootstrap (because we don't always have full episode), set ``gae_lambda=1.0`` during initialization. The TD(lambda) estimator has also two special cases: - TD(1) is Monte-Carlo estimate (sum of discounted rewards) - TD(0) is one-step estimate with bootstrapping (r_t + gamma * v(s_{t+1})) For more information, see discussion in https://github.com/DLR-RM/stable-baselines3/pull/375. :param last_values: state value estimation for the last step (one for each env) :param dones: if the last step was a terminal step (one bool for each env). rr*?N)clonecpunumpyflattenreversedrangerrerJrfrrrrrrr)r!rr last_gae_lamstepnext_non_terminal next_valuesdeltas r#compute_returns_and_advantage+RolloutBuffer.compute_returns_and_advantages("'')--/557??A  U4#3#345D''!++$'%,,rzz*B$B!) $'$*=*=dQh*G$G!"kk$(3 LL&k)ADU)UUX\XcXcdhXiiE ::#?BS#SVb#bbL$0OOD !64 r%r\rrb episode_startvaluelog_probcJ[UR5S:XaURSS5n[UR[ R 5(a)URUR/URQ75nURURUR45n[R"U5URUR'[R"U5URUR'[R"U5URUR'[R"U5UR UR'UR#5R%5R'5R)5UR*UR'UR#5R%5R'5UR,UR'U=RS- sl URUR.:XaSUlgga :param obs: Observation :param action: Action :param reward: :param episode_start: Start of episode signal. :param value: estimated value of the current state following the current policy. :param log_prob: log probability of the action following the current policy. rrr*TN)r-r,r/rrr rrrrrJrTrrrrrrrrrrrrr )r!r\rrbrrrs r#r;RolloutBuffer.addss& x~~ ! #''A.H d,,foo > >++t{{?&(hhsm$((#!#&!1 TXX!#&!1 TXX(*(?DHH% % 1 1 3 9 9 ; C C E DHH#+>>#3#7#7#9#?#?#Atxx  A  88t'' 'DI (r%rEc#0# UR(dS5e[RRURUR -5nUR (d?/SQnUH.nURURU5URU'M0 SUlUcURUR -nSnXPRUR -:a:URX%XQ-5v XQ- nXPRUR -:aM9gg7f)N)rrrrrrTr) r rJrK permutationrrrr0__dict__rM)r!rEindices _tensor_namesrX start_idxs r#rRolloutBuffer.getsyy"y))''(8(84;;(FG##M((,(=(=dmmF>S(T f%(#'D   ))DKK7J **T[[88##G 8N$OP P  #I**T[[88s DDDrOrFchURUURUURUR5URUR5UR UR5UR UR54n[[[URU556$r^) rrrrrrrrrrrrZ)r!rOrFr?s r#rMRolloutBuffer._get_sampless   j ) LL $ KK # + + - NN: & . . 0 OOJ ' / / 1 LL $ , , .  $U3t}}d+C%DEEr% rrrr rrrrrrrrrir*Gz?r*rjr^)rkrlrmrnrorJrvrqrsr rprrWrrtfloatrrCrxrr;rrrrrrMrzr{r|s@r#rrWs,** ZZ ZZ  ZZJJzz JJ)/"<<ll  biin%    #5#52::#5Z^#5J' ZZ' ' ' zz ' yy '))' 'R$hsm$yAUW[]aAa7b$<'+ FJJ Fl # F  F Fr%rc8^\rSrSr%Sr\R \S'\\ \ \ S44\S'\\ \ R4\S'\\ \ R4\S'SS\ S\R S \RS \\R"\ 4S \ S \S \4U4SjjjrS\\ \ R4S\\ \ R4S\ RS\ RS\ RS\\\ \4SS4SjrSS\ S\\S\4U4SjjjrSS\ RS\\S\4SjjrSrU=r$) DictReplayBufferi a Dict Replay buffer used in off-policy algorithms like SAC/TD3. Extends the ReplayBuffer to use dictionary observations :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param device: PyTorch device :param n_envs: Number of parallel environments :param optimize_memory_usage: Enable a memory efficient variant Disabled for now (see https://github.com/DLR-RM/stable-baselines3/pull/243#discussion_r531535702) :param handle_timeout_termination: Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284 r.rrrrrrrrrc >[[U] XX4US9 [UR[ 5(dS5e[ X-S5Ul[b[R"5RnU(aS5eX`l URR5V V s0sH@upU [R"URUR/U Q7X)R S9_MB sn n UlURR5V V s0sH@upU [R"URUR/U Q7X)R S9_MB sn n Ul[R"URURUR&4UR)UR 5S9Ul[R"URUR4[R,S9Ul[R"URUR4[R,S9UlXpl[R"URUR4[R,S9Ul[bSn UR"R5HupXR6- n M XR*R6-UR.R6-UR0R6-nU(d7SnUR"R5HupXR6- nM X- nUW:a*US-nUS-n[8R:"SUS S US S 35 gggs sn n fs sn n f) Nrz6DictReplayBuffer must be used with Dict obs space onlyr*z7DictReplayBuffer does not support optimize_memory_usagerrrrrrr)rr~rrrryrrrrrritemsrJrrrrrrrrrfrrrrrrr)r!rrrrrrrrkey _obs_shape obs_nbytes_r\rnext_obs_nbytesr"s r#rDictReplayBuffer.__init__"s lD*;>#7#7#9 #9 4++T[[F:FN_NdNjNjk k#9  $(>>#7#7#9" #9 4++T[[F:FN_NdNjNjk k#9"  xx   t{{DOO ?!#&!1 TXX!#&!1 TXX!xx~ 488  * *&(hhch/ich[_9NPU0Vch/i&jDMM$(( # A  88t'' 'DIDH (0jsJ>rErFc&>[[U] XS9$)z Sample elements from the replay buffer. :param batch_size: Number of element to sample :param env: associated gym VecEnv to normalize the observations/rewards when sampling :return: r)rr~rP)r!rErFr"s r#rPDictReplayBuffer.samples\4/:/OOr%rOc([RRSUR[ U54S9nUR UR R5VVs0sHupEXEXSS24_M snnU5nUR URR5VVs0sHupEXEXSS24_M snnU5n[U[5(de[U[5(deUR5VVs0sHupEX@RU5_M nnnUR5VVs0sHupEX@RU5_M n nn[UURURX45U URURX4SURX4- -5R!SS5URUR#UR$X4R!SS5U55S9$s snnfs snnfs snnfs snnf)Nrrr*r)rrrrr)rJrKrLrr-r`rrrrryrZr rrrr/rgr) r!rOrFrrr\obs_ next_obs_rrs r#rMDictReplayBuffer._get_sampless ii'' 3z?BT'U ""Y]YjYjYpYpYr#sYrXSCZa-G)H$HYr#suxy''BFBXBXB^B^B` aB`hcSjq01 1B` acf $%%%%)T****@D M HC]]3// MEN__EVWEVS--"44EVW&%MM$,,z/F"GH/-- :+B Cq4==YcYpKqGq rs{{AMM$"8"8jF]9^9f9fgikl9mor"st  $t a NWsG< "H H4Hrrr^)rkrlrmrnror DictrqryrtrrrsrJrvrprrWrrwrrrr;rrr rPrMrzr{r|s@r#rr s {{"CsCx())sBJJ''CO,,)/&++/??";;?ll ? biin% ?  ? $?%)??B# #rzz/ "#sBJJ'# #  # jj #DcN## #P'+ P Pl # P ! P P$'+ JJ l #  !   r%rc^\rSrSr%Sr\R \S'\\ \ \ S44\S'\\ \ R4\S'SS\ S\R S\RS \\R"\ 4S \S \S \ 4U4S jjjrSU4SjjrS\\ \ R4S\ RS\ RS\ RS\R*S\R*SS4SjrS S\\ S\\SS44SjjrS S\ RS\\S\4SjjrSrU=r$)!DictRolloutBufferia Dict Rollout buffer used in on-policy algorithms like A2C/PPO. Extends the RolloutBuffer to use dictionary observations It corresponds to ``buffer_size`` transitions collected using the current policy. This experience will be discarded after the policy update. In order to use PPO objective, we also store the current value of each state and the log probability of each taken action. The term rollout here refers to the model-free notion and should not be used with the concept of rollout used in model-based RL or planning. Hence, it is only involved in policy and value function training but not action selection. :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param device: PyTorch device :param gae_lambda: Factor for trade-off of bias vs variance for Generalized Advantage Estimator Equivalent to Monte-Carlo advantage estimate when set to 1. :param gamma: Discount factor :param n_envs: Number of parallel environments r.rrrrrrrrc>[[U] XX4US9 [UR[ 5(dS5eXPlX`lSUlUR5 g)Nrz7DictRolloutBuffer must be used with Dict obs space onlyF) rrrrrryrrrrCrs r#rDictRolloutBuffer.__init__sT mT+KLio+p$..$//j1jj/$ $ r%r'Nc>0UlURR5HNup[R"UR UR /UQ7[RS9URU'MP [R"UR UR UR4[RS9Ul [R"UR UR 4[RS9Ul [R"UR UR 4[RS9Ul [R"UR UR 4[RS9Ul [R"UR UR 4[RS9Ul [R"UR UR 4[RS9Ul[R"UR UR 4[RS9UlSUl["[$U]O5 gr)rrrrJrrrrfrrrrrrrrrrrrC)r!robs_input_shaper"s r#rCDictRolloutBuffer.resets}$(NN$8$8$: C%'XXt/?/?._._gigqgq%rD  c "%;xx!1!14;; PXZXbXbc xx!1!14;; ?rzzR xx!1!14;; ?rzzR  hh(8(8$++'FbjjYhh 0 0$++>bjjQ 4#3#3T[["AT((D$4$4dkk#B"**U$ mT(*r%r\rrbrrrc[UR5S:XaURSS5nURR 5Hn[ R "X5n[URRU[R5(a,URUR4URU-5nXRUUR'M URURUR45n[ R "U5URUR'[ R "U5UR UR'[ R "U5UR"UR'UR%5R'5R)5R+5UR,UR'UR%5R'5R)5UR.UR'U=RS- sl URUR0:XaSUlggr)r-r,r/rrrJrTrrr rrrrrrrrrrrrrrrr ) r!r\rrbrrrrrs r#r;DictRolloutBuffer.adds& x~~ ! #''A.H$$))+C88CH%D$0077?!#&!1 TXX!#&!1 TXX(*(?DHH% % 1 1 3 9 9 ; C C E DHH#+>>#3#7#7#9#?#?#Atxx  A  88t'' 'DI (r%rEc## UR(dS5e[RRURUR -5nUR (dURR5H#up4URU5URU'M% /SQnUH.nURURU5URU'M0 SUlUcURUR -nSnXpRUR -:a:URX'Xq-5v Xq- nXpRUR -:aM9gg7f)Nr)rrrrrTr) r rJrKrrrrrrr0rrM)r!rErrr\rrXrs r#rDictRolloutBuffer.get"s$yy"y))''(8(84;;(FG## --335)-)>)>s)C!!#&6XM'(,(=(=dmmF>S(T f%(#'D   ))DKK7J **T[[88##G 8N$OP P  #I**T[[88s EEErOrFc .[URR5VVs0sHup4X0RXA5_M snnURURU5URUR UR 55URURUR 55URURUR 55URURUR 55S9$s snnf)N)rr old_values old_log_probrr) r rrrZrrrrrr)r!rOrFrr\s r#rMDictRolloutBuffer._get_samples<s (PTPaPaPgPgPijPi*3#}}S_==PijMM$,,z":;}}T[[%<%D%D%FGt~~j'A'I'I'KL}}T__Z%@%H%H%JKMM$,,z":"B"B"DE   jsD rrrjr^)rkrlrmrnror rrqryrtrrrsrJrvrprrWrrrrCrxr;rrr rrrMrzr{r|s@r#rrs0{{"CsCx())sBJJ'')/";;ll  biin%   ( +) #rzz/ ") ) ) zz ) yy )))) )Z%)$SM$ +T47 8$:'+  JJ  l #   "   r%rcv^\rSrSrSrSSS.S\S\4U4SjjjrSS \RS \ \ S \ 4S jjr S rU=r$)NStepReplayBufferiKa Replay buffer used for computing n-step returns in off-policy algorithms like SAC/DQN. The n-step return combines multiple steps of future rewards, discounted by the discount factor gamma. This can help improve sample efficiency and credit assignment. This implementation uses the same storage space as a normal replay buffer, and NumPy vectorized operations at sampling time to efficiently compute the n-step return, without requiring extra memory. This implementation is inspired by: - https://github.com/younggyoseo/FastTD3 - https://github.com/DLR-RM/stable-baselines3/pull/81 It avoids potential issues such as: - https://github.com/younggyoseo/FastTD3/issues/6 :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param device: PyTorch device :param n_envs: Number of parallel environments :param optimize_memory_usage: Not supported :param handle_timeout_termination: Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284 :param n_steps: Number of steps to accumulate rewards for n-step returns :param gamma: Discount factor for future rewards r)r)n_stepsrrrcv>[TU]"U0UD6 XlX lUR(a [ S5eg)Nz./tfsfuffD rzzD 8ND ZmD D r%r)$rabcrrcollections.abcrrrrrrrJtorchrW gymnasiumr &stable_baselines3.common.preprocessingr r %stable_baselines3.common.type_aliasesr r rrstable_baselines3.common.utilsr stable_baselines3.common.vec_envrr ImportErrorrr~rrrrr%r#r>s#%''P 69 @@Fv:vrrFJrFji |i XP P fk k i Fs BBB