h(JSSKrSSKrSSKJrJrJr SSKrSSKr SSK J r SSK J r SSKJr SSKJrJr SSKJrJr "SS \ 5rg) N)AnyOptionalUnion)spaces)DictReplayBuffer)DictReplayBufferSamples)VecEnv VecNormalize)KEY_TO_GOAL_STRATEGYGoalSelectionStrategyc^\rSrSr%Sr\\\S'S'S\S\ RS\ RS\S\ \ R\4S\S \S \S \S \ \\4S \4U4SjjjrS\\\44SjrS\\\4SS4SjrS\SS4SjrS\\\R24S\\\R24S\R2S\R2S\R2S\\\\4SS4U4SjjrS\SS4SjrS(S\S\\S\4SjjrS(S \R2S!\R2S\\S\4S"jjr S(S \R2S!\R2S\\S\4S#jjr!S \R2S!\R2S\R24S$jr"S)S%jr#S&r$U=r%$)*HerReplayBuffera7 Hindsight Experience Replay (HER) buffer. Paper: https://arxiv.org/abs/1707.01495 Replay buffer for sampling HER (Hindsight Experience Replay) transitions. .. note:: Compared to other implementations, the ``future`` goal sampling strategy is inclusive: the current transition can be used when re-sampling. :param buffer_size: Max number of element in the buffer :param observation_space: Observation space :param action_space: Action space :param env: The training environment :param device: PyTorch device :param n_envs: Number of parallel environments :param optimize_memory_usage: Enable a memory efficient variant Disabled for now (see https://github.com/DLR-RM/stable-baselines3/pull/243#discussion_r531535702) :param handle_timeout_termination: Handle timeout termination (due to timelimit) separately and treat the task as infinite horizon task. https://github.com/DLR-RM/stable-baselines3/issues/284 :param n_sampled_goal: Number of virtual transitions to create per real transition, by sampling new goals. :param goal_selection_strategy: Strategy for sampling goals for replay. One of ['episode', 'final', 'future'] :param copy_info_dict: Whether to copy the info dictionary and pass it to ``compute_reward()`` method. Please note that the copy may cause a slowdown. False by default. env buffer_sizeobservation_space action_spacedevicen_envsoptimize_memory_usagehandle_timeout_terminationn_sampled_goalgoal_selection_strategycopy_info_dictc >[T U]UUUUUUUS9 X@lXl[ U [ 5(a[ U R5UlOXl[ UR[5(dS[[535eXl SSURS-- - Ul [R"[UR 5V s/sH)n [UR"5V s/sHn 0PM sn PM+ sn 5Ul[R&"UR UR"4[R(S9Ul[R&"UR UR"4[R(S9Ul[R&"UR"[R(S9Ulgs sn fs sn f)N)rrrrz3Invalid goal selection strategy, please use one of g?)dtype)super__init__rr isinstancestrr lowerrr listr her_rationparrayrangerrinfoszerosint64ep_start ep_length_current_ep_start)selfrrrrrrrrrrr_ __class__s a/home/james-whalen/.local/lib/python3.13/site-packages/stable_baselines3/her/her_replay_buffer.pyrHerReplayBuffer.__init__2s    "7'A  , -s 3 3+?@W@]@]@_+`D (+B (  ( (*?   _ @F[A\@] ^ _ -cT%8%81%<=>XXtO_O_I`aI`AE$++,>?,>q,>?I`ab $"2"2DKK!@Q 4#3#3T[["AR!#$++RXX!F @asG3 G?GGreturnc@URR5nUS U$)zY Gets state for pickling. Excludes self.env, as in general Env's may not be pickleable. r)__dict__copyr.states r1 __getstate__HerReplayBuffer.__getstate__es"  ""$ %L r8NcXURRU5 SU;deSUlg)ze Restores pickled state. User must call ``set_env()`` after unpickling before using. :param state: rN)r5updaterr7s r1 __setstate__HerReplayBuffer.__setstate__ps+ U#E!!!r;c@URb [S5eXlg)z$ Sets the environment. :param env: Nz5Trying to set env of already initialized environment.)r ValueError)r.rs r1set_envHerReplayBuffer.set_env|s 88 TU Ur;obsnext_obsactionrewarddoner(c>[UR5HnURURU4nURURU4n U S:dMAX-n [ R "URU 5UR-n SURX4'M URR5URUR'UR(aX`RUR'[T U]5XX4XV5 [UR5HnXW(dMURU5 M! g)Nr)r'rr+posr,r%arangerr-r6rr(radd_compute_episode_length) r.rDrErFrGrHr(env_idx episode_startepisode_length episode_endepisode_indicesr0s r1rLHerReplayBuffer.addsT[[)G MM$((G*;N!+< "$))DHHk"BTEUEU"U;<78 *#'"8"8"="="? dhh   #(JJtxx   C64?T[[)G}},,W5*r;rNcURUnURnX2:aX0R- n[R"X#5UR-nX2- UR XA4'URURU'g)z Compute and store the episode length for environment with index env_idx :param env_idx: index of the environment for which the episode length should be computed N)r-rJrr%rKr,)r.rNrOrQrRs r1rM'HerReplayBuffer._compute_episode_lengthst ..w7 hh  & ++ +K))M?$BRBRR3>3N/0*.((w'r; batch_sizec URS:n[R"U5(d [S5e[R"U5n[R R XASS9n[R"XSR5upg[URU-5n[R"Xh/5up[R"Xx/5upURXU5n URXU5nURR5Vs0sH6nU[ R""U RUURU45_M8 nn[ R""U R$UR$45nUR&R5Vs0sH6nU[ R""U R&UUR&U45_M8 nn[ R""U R(UR(45n[ R""U R*UR*45n[-UUUUUS9$s snfs snf)z Sample elements from the replay buffer. :param batch_size: Number of element to sample :param env: Associated VecEnv to normalize the observations/rewards when sampling :return: Samples rzUnable to sample before the end of the first episode. We recommend choosing a value for learning_starts that is greater than the maximum number of timesteps in the environment.T)sizereplace observationsactionsnext_observationsdonesrewards)r,r%any RuntimeError flatnonzerorandomchoice unravel_indexshapeintr$split_get_real_samples_get_virtual_samplesr[keysthcatr\r]r^r_r)r.rVris_valid valid_indicessampled_indices batch_indices env_indices nb_virtualvirtual_batch_indicesreal_batch_indicesvirtual_env_indicesreal_env_indices real_data virtual_datakeyr[r\r]r^r_s r1sampleHerReplayBuffer.samples>>A%vvho x0 ))**=SW*X&(%5%5o~~%V" *45 46HH]L4Y102l0S-**+=QTU 001F]`a $00557 7 //4l6O6OPS6TUV V7  &&)++\-A-ABC$55::< < 44S9<;Y;YZ];^_` `<   ););<=&&)++\-A-ABC&%/     s =I=IrqrrcURURR5VVs0sHupEXEXSS24_M snnU5nURURR5VVs0sHupEXEXSS24_M snnU5n[ U[ 5(de[ U[ 5(deUR5VVs0sHupEX@R U5_M nnnUR5VVs0sHupEX@R U5_M n nn[UUR URX45U UR URX4SURX4- -5RSS5UR URURX4RSS5U55S9$s snnfs snnfs snnfs snnf)a$ Get the samples corresponding to the batch and environment indices. :param batch_indices: Indices of the transitions :param env_indices: Indices of the environments :param env: associated gym VecEnv to normalize the observations/rewards when sampling, defaults to None :return: Samples NrrZ)_normalize_obsr[itemsr]r dictto_torchrr\r^timeoutsreshape_normalize_rewardr_) r.rqrrrrzrDobs_ next_obs_r[r]s r1ri!HerReplayBuffer._get_real_sampless ""\`\m\m\s\s\u#v\uPXPSC]-J)K$K\u#vx{|''EIE[E[EaEaEc dEcSm!34 4Ec dfi $%%%%)T****@D M HC]]3// MEN__EVWEVS--"44EVW&%MM$,,}/I"JK/-- =56!dmmMLf>g:ghgb!nMM$"8"8mF`9a9i9ijlno9pru"vw  $w d NWsG /G GGcURR5VVs0sHupEXEXSS24_M snnnURR5VVs0sHupEXEXSS24_M nnnUR(a%[R "UR X45nO$[[U55Vs/sHn0PM nnURX5n U WS'XS'URcS5eURRSUSUSUS/S9n U SR[R5n URXS5nURXc5nUR5VVs0sHupEX@R!U5_M n nnUR5VVs0sHupEX@R!U5_M n nn[#U UR!UR$X45U UR!UR&X4SUR(X4- -5R+S S5UR!UR-U R+S S5U55S 9$s snnfs snnfs snfs snnfs snnf) aK Get the samples, sample new desired goals and compute new rewards. :param batch_indices: Indices of the transitions :param env_indices: Indices of the environments :param env: associated gym VecEnv to normalize the observations/rewards when sampling, defaults to None :return: Samples, with new desired goals and new rewards N desired_goalzcYou must initialize HerReplayBuffer with a VecEnv so it can compute rewards for virtual transitionscompute_reward achieved_goalr)indicesrr~rZ)r[rr]rr6deepcopyr(r'len _sample_goalsr env_methodastyper%float32rrrr\r^rrr) r.rqrrrrzrDrEr(r/ new_goalsr_r[r]s r1rj$HerReplayBuffer._get_virtual_samplessa HLGXGXG^G^G`aG`83s A566G`aLPLbLbLhLhLjkLjC]:;;Ljk   MM$**]-G"HIE!&s='9!:;!:AR!:E;&&}B 'N#,  HH  q p q ((%%  _ %   C& !*##BJJ/!!#+&&x5AD L HC]]3// LEM^^EUVEUS--"44EUV&%MM$,,}/I"JK/-- =56!dmmMLf>g:ghgb!nMM$"8"8Q9OQT"UV  Qbk <>MVsII"8 I(/I-"I3c URX4nURX4nUR[R:XaUS- nOUR[R :Xa1X- UR -n[RRXd5nOXUR[R:Xa![RRSU5nO[SURS35eXS-UR -nURSXr4$)z Sample goals based on goal_selection_strategy. :param batch_indices: Indices of the transitions :param env_indices: Indices of the environments :return: Sampled goals rrz Strategy z" for sampling goals not supported!r) r+r,rr FINALFUTURErr%rcrandintEPISODErAr])r.rqrrbatch_ep_startbatch_ep_lengthtransition_indices_in_episodecurrent_indices_in_episodetransition_indicess r1rHerReplayBuffer._sample_goalscs}'AB..)CD  ' '+@+F+F F,;a,? )  ) )-B-I-I I+8*HDL\L\)\ &,.II,=,=>X,j )  ) )-B-J-J J,.II,=,=a,Q )y)E)E(FFhij j;LPTP`P``%%o67I7VWWr;cURUR:gR5(a[R"S5 [ R "URUR:g5SHlnSURURS- U4'UR[U55 UR(dMNSURURS- U4'Mn gg)z If called, we assume that the last trajectory in the replay buffer was finished (and truncate it). If not called, we assume that we continue the same trajectory (same episode). zThe last trajectory in the replay buffer will be truncated. If you are in the same episode as when the replay buffer was saved, you should use `truncate_last_trajectory=False` to avoid that issue.rTrN) r-rJr`warningswarnr%wherer^rMrgrr)r.rNs r1truncate_last_trajectory(HerReplayBuffer.truncate_last_trajectorys  " "dhh . 3 3 5 5 MMW  88D$:$:dhh$FGJ48 488a<01,,S\:222;?DMM$((Q,"78K 6r;) r-rrr,r+rr$r(r)autorFTfutureF)N)r3N)&__name__ __module__ __qualname____firstlineno____doc__rr __annotations__rgrDictSpacerrlrr!boolr rrrr9r>rBr%ndarrayr#rLrMr rr{rirjrr__static_attributes__ __classcell__)r0s@r1rrs~@ & )/&++/EM$1G1G";;1Gll 1G  1G biin% 1G1G $1G%)1G1G"''@@r;r)r6rtypingrrrnumpyr%torchrl gymnasiumr stable_baselines3.common.buffersr%stable_baselines3.common.type_aliasesr stable_baselines3.common.vec_envr r -stable_baselines3.her.goal_selection_strategyr r rr;r1rs7 ''=IAeI@&I@r;