ni`YSrSSKJr SSKJr SSKJr SSKrSSKJs Jr SSK J r SSK Jr SS K Jr "S S \5rg!\ a SSK r N*f=f) zProximal Policy Optimization algorithm. Based on John Schulman's implementation in Python and Theano: https://github.com/joschu/modular_rl/blob/master/modular_rl/ppo.py )absolute_import)division)print_functionN)memory) normalize)utilitycl\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrg) PPOAlgorithm$zBA vectorized implementation of the PPO algorithm by John Schulman.c  ^TUlX lX0lX@lXPl[ R "URRSSSSSS9Ul[ R "URRSSSSSS9Ul URRSURRSURRSURRSURRS4n[R"XeRURS 5Ul["R$"SS5UlURR(=(a [*R,"5n["R."U(aS OS 5 URRR0S R2n["R4"S [6R8"UR:XX55UlUR=["R>"URR5SS2S4["R@"[CUR555n ["RD"S5 [R"U[CT5URS5Ul#U RHcSUl%O["RLRNRPRSU4SjU RH5 ["RLRNRPRSSU RH5Ul%["R$"["R>"URR5SSS9Ul*["R$"["R>"URR5SSS9Ul+["R$"["R>"URR5SSS9Ul,SSS5 SSS5 ["R$"URRZS["R\S9Ul/URRaURRb5Ul2g!,(df  N=f!,(df  N=f)acCreate an instance of the PPO algorithm. Args: batch_env: In-graph batch environment. step: Integer tensor holding the current training step. is_training: Boolean tensor for whether the algorithm should train. should_log: Boolean tensor for whether summaries should be returned. config: Object containing the agent configuration as attributes. rTnormalize_observ)centerscaleclipnameF normalize_rewardrz/gpu:0z/cpu:0rnetworkN ppo_temporaryepisodescv>UR[T5/URR5SS-5$)Nr) set_shapelenshapeas_list)x batch_envs \/home/james-whalen/.local/lib/python3.13/site-packages/pybullet_envs/agents/ppo/algorithm.py'PPOAlgorithm.__init__..Us, S^$4qww7H7L$LMc:^[R"U4SjS5$)Nc0>[R"T5$N)tf zeros_likersr r!9PPOAlgorithm.__init__....XsBMM!,B=A<=C=A<> ?{{2 @ ? ) ( @ ? ? ) ( (s$A7CB7$ C7 C C C c b^^^^[R"S5 URRU5nURcSnOB[R R RRU4SjUR5nURUSS2S4[R"URS5U5m[R"URTRRU4Sj5mTRR!T5SS2S4m[R"UR"UUU4Sj[$5nURc[R&"5nO,[(R*"URTR,T5n[R."U[R0"UR2TTSS2S45[R0"UR4TTR6SS2S45[R0"UR8TTR:SS2S45/5 [R<"TSS2S4S5[R>"U54sSSS5 sSSS5 $!,(df  O=fSSS5 g!,(df  g=f)zCompute batch of actions and a summary for a batch of observation. Args: agent_indices: Tensor containing current batch indices. observ: Tensor of a batch of observations for all agents. Returns: Tuple of action batch tensor and summary tensor. zperform/Nc2>[R"UT5$r&)r'gather)rrls r r!&PPOAlgorithm.perform..s"))A}B]r#rc>TR$r&)mean)rasr r!rts  r#c >[RR[RRSTRSS2S45[RRS[R "TR SS2S455[RRSTSS2S45[RRST5/5$)Nrvrstdr:logprob)r'summarymerge histogramrvexplogstd)r:ryrasr r!rtsBJJ$4$4jj""66;;q!t+<=jj""5"&&q!t1D*EFjj""8VAqD\:jj""9g6 6 % r#r:) r'rfr7 transformrLrMrNrOrPrGrHrcondr2policysamplelog_probr3strrgr assign_nested_varsrKrjscatter_updaterQrRrvrSr~check_numericsidentity) rZrlr6rKrz assign_stater:ryras ` @@@r performPPOAlgorithm.performws z """,,V4f   ! $$))778]8<8H8HJ}}VAtG_bggfll1o.FNfwwt((&--*>*>@STf &&v.q!t4g     g    !xxz 11$2B2BFLLR_` " "   D--}fQTl K  DOO]FKK1Q R $   1x8"++g:NN   / # ".   / # " "s$H#J >4J2 J  J J  J.c ^^^^^[R"S5 [R"TRUUUUU4Sj[5sSSS5 $!,(df  g=f)aProcess the transition tuple of the current step. When training, add the current transition tuple to the memory and update the streaming statistics for observations and rewards. A summary string is returned if requested at this step. Args: agent_indices: Tensor containing current batch indices. observ: Batch tensor of observations. action: Batch tensor of actions. reward: Batch tensor of rewards. unused_done: Batch tensor of done flags. unused_nextob: Batch tensor of successor observations. Returns: Summary tensor. z experience/c,>TRTTTT5$r&)_define_experience)r:rlr6r8rZsr r!)PPOAlgorithm.experience..s$))-Pr#Nr'rfrr2r)rZrlr6r:r8 unused_done unused_nextobs````` r experiencePPOAlgorithm.experiences;$ } % WW    P P  & % %s .A A"ct^^^ ^ [RRTRR U5TR R U5/5m [R "T /5 TRR(a TRnX#[R"TRU5[R"TRU5U4nTRRXQ5nSSS5 [R "W/5 TRRU5m[R "TR RU55m [R""TR$UU UU 4Sj[&5nUsSSS5 $!,(df  N=f!,(df  g=f)z=Implement the branch of experience() entered during training.Nc >[RRTTRR5TRR5[RR STR 5[RRST5[RRSTR5[RR ST5/5$)N memory_sizenormalized_observr:normalized_reward) r'rzr{r7r9scalarr?r|rQ) norm_observ norm_rewardrZupdate_filterssr r!1PPOAlgorithm._define_experience..sBJJ$4$4!!))+!!))+jj t/A/ABjj""#6 Djj""8T->->?jj 3[A6 % r#)r'rzr{r7updater9rjr4train_on_agent_actionrQrsrRrSrJappendr reduce_meanrr3r) rZrlr6r:r8batchrrzrrrs ` @@@r rPPOAlgorithm._define_experiencesKZZ%%   # #F +   # #F + -.N .!1 2  + +""ryy)6 89;4CTCTCP:RSY[e~~$$U:f 3 & *''11&9kNN4#6#6#@#@#HIk     g + * 3 2 + *s/BFA9F) F&) F7c^^[R"S5 [R"TRUU4Sj[5sSSS5 $!,(df  g=f)aAdd episodes to the memory and perform update steps if memory is full. During training, add the collected episodes of the batch indices that finished their episode to the memory. If the memory is full, train on it, and then clear the memory. A summary string is returned if requested at this step. Args: agent_indices: Tensor containing current batch indices. Returns: Summary tensor. z end_episode/c&>TRT5$r&)_define_end_episode)rlrZsr r!*PPOAlgorithm.end_episode..s0H0H0Wr#Nr)rZrls``r end_episodePPOAlgorithm.end_episodes5 ~ & WWT&&(WY\ ] ' & &s +A AcURRU5up#URRUR- n[ R "[ R"[ R"U5SU55nUVs/sHn[ R"Xe5PM nnURRU[ R"X55XPR-5n[ R"U/5 URR[ R"U5S5nSSS5 [ R"W/5 URURR:n [ R"XR[ 5sSSS5 $s snf!,(df  N|=f!,(df  g=f)z>Implement the branch of end_episode() entered during training.rN)rJdatar4r<r?r'rangeminimumrrsr>replacerj assign_addr _trainingr) rZrlrlength space_left use_episodeselemr inc_index memory_fulls r r PPOAlgorithm._define_end_episodes8~~**=9H**T-?-??J88BJJrxx '>q'A:NOL:BC($ $-(HC \\ ! !(BIIf,K".1C1C"CEF & *$$//0Fq0IJi + ) -&&$,,*C*CCk WW[..# 6 . - D + * - -s F!=3F&AF7& F47 Gc t[R"S5 [R"URURR 5n[R "U/5 URR5nSSS5 Wuup4pVpx[R "[R"US5/5 [R"U5nSSS5 URRU5nURRU5nURX4XVUU5n [R "U /5 URX5Xh5n SSS5 [R "W /5 [R "URR#5URR%S55n SSS5 [R "W /5 [&R("[R*"5URR,5n [R.R1XU /5sSSS5 sSSS5 $!,(df  GN=f!,(df  GN=f!,(df  GN$=f!,(df  N=f!,(df  O=fSSS5 g!,(df  g=f)zPerform multiple training iterations of both policy and value baseline. Training on the episodes collected in the memory. Reset the memory afterwards. Always returns a summary string. Returns: Summary tensor. trainingNr)r'rf assert_equalr?r4r<rjr>rassert_greaterrr7rr9_perform_update_steps_adjust_penaltygroupriassignr variable_summariestrainable_variablesweight_summariesrzr{) rZ assert_fullrr6r:old_mean old_logstdr8rupdate_summarypenalty_summary clear_memoryweight_summarys r rPPOAlgorithm._trainings z "OOD$6$6 8Q8QRk " "K= 1||  " 2?C<4vxV " "B$5$5fa$@#A BV$ C"",,V4f"",,V4f11&(X^28:n " "N#3 4..vT 5 " "O#4 5xx 2 2 4d6H6H6O6OPQ6RS  6 " "L> 2 33B4J4J4L48LL4Q4QSzz. QR 3 2 # " 1 1 C B 5 4 5 5 2 2 2 # " "sA J)$I?zreturn and value: znormalized advantage: c 2>T RTTTTTTT5$r&) _update_step) _1_2r: advantagerr6rrr8rZs r r!4PPOAlgorithm._perform_update_steps..1s d>O>O*fi?Ir#)rre)parallel_iterationsz value loss: z policy loss: N)r discounted_returnr4discountrGrC gae_lambda lambda_returnr'nnmomentssqrtPrintrscanr update_epochsrrj)rZr6r:rrr8rreturn_rCrvvariance value_loss policy_lossrz print_lossesrs``````` @r r"PPOAlgorithm._perform_update_stepss$'' 8M8MNG MM&& ) / /E ||''vt||?T?T(, (?(?Ai/iUU]]9Aq6T]JNDT!bggh&7$&>?I..12>>%3HIK_aIR^^I%>$?AYZI')ww0I0I/1xx 8R8R/SUaCD(F$JW88BHHQ )C(DnUHHQ )D(EWYL * #[K(+T__-N-Nz-Z([$O*-t/P/PQ\/]*^&&#6M#&66M..s=-/PQHjj /1PQ 0"..AQ2RS""3#H$UZJ[\""3'79I#JDX]L^_  G ( ,'1&H I&Hbkk!n&H I - , I - ,s0G(7 G#G(#G(( G6c h[R"S5 URX5Rn[R "X#UR R5nXT- nSURUS-U5-n[RR[RRSU5[RRS[R"U55/5n[R"U5n[R"US5U4sSSS5 $!,(df  g=f)anCompute the loss function for the value baseline. The value loss is the difference between empirical and approximated returns over the collected episodes. Returns the loss tensor and a summary strin. Args: observ: Sequences of observations. reward: Sequences of reward. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. rg?ravg_value_lossN)r'rfrGrCr rr4r_maskrzr{r|rrr) rZr6r8rrCrrrrzs r rPPOAlgorithm._value_loss\s | $mmF+11e))&$,,:O:OPg/iIqL&99j   **  |Z 8 **  ,bnnZ.H I" g>>*-j   z< 8' A % $ $s DD## D1c^[R"S5 [R"X5n[R"UR [R "X4X5U5S5n [R"[R"XU5[R"X4U5- 5n [R"UR U [R"U5-U5S5*n URU -n URRURR-n [R"[R"X:[R 55m[R""[R$"TS:U4Sj[&5/5 URR([R"X:[R*5-X- S--nSSS5 X-W-n[R,R/[R,R1SU5[R,R1SU 5[R,R1S U 5[R,R1S U 5[R,R1S U5[R,R1S X-5[R,R1SU5[R,R3S [R"U 55[R,R3S[R"U 55[R,R3S[R"U55/ 5n[R"US5n[R4"US5U4sSSS5 $!,(df  GN=f!,(df  g=f)aTCompute the policy loss composed of multiple components. 1. The policy gradient loss is importance sampled from the data-collecting policy at the beginning of training. 2. The second term is a KL penalty between the policy at the beginning of training and the current policy. 3. Additionally, if this KL already changed more than twice the target amount, we activate a strong penalty discouraging further divergence. Args: mean: Sequences of action means of the current policy. logstd: Sequences of action log stddevs of the current policy. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. action: Sequences of actions. advantage: Sequences of advantages. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. rrrc6>[R"ST/S5$)Nrz kl cutoff! )r'r) cutoff_countsr r!+PPOAlgorithm._policy_loss..sRXXa,-Wr#rNentropyklsurrogate_loss kl_penalty kl_cutoffkl_penalty_combined avg_surr_lossavg_kl_penaltyavg_policy_loss)r'rfr diag_normal_entropyrrdiag_normal_klr}diag_normal_logpdf stop_gradientrVr4 kl_targetkl_cutoff_factor reduce_sumcastint32rjrintkl_cutoff_coefrUrzr{r|rr)rZrvr~rrr:rrrrpolicy_gradientrrcutoff_thresholdr rrzrs @r rPPOAlgorithm._policy_lossvs, } %++D9g >> **W++H$OQW XZ[ ]b  $ $T6 :  $ $X6 B CDo **_r'7'7 'BBF KQPPn==2%j//$,,2O2OO]]2772+@"((#KLl " " 77#GO> O O Oc ^[R"S5 TRX5n[R"[R"[R "UR U55SSS9n[R"STR/S5n[R"Xg/5 [R"TR[R"X#UR UR5U55n[R"X/S5n[R"USTR R"-:U4S j[$5n [R"US TR R"-:U4S j[$5n S S S 5 [R"W W /5 [R&R)[R&R+S W5[R&R+STR5/5sS S S 5 sS S S 5 $!,(df  N=f!,(df  O=fS S S 5 g !,(df  g =f)aAdjust the KL policy between the behavioral and current policy. Compute how much the policy actually changed during the multiple update steps. Adjust the penalty strength for the next training phase if we overshot or undershot the target divergence too much. Args: observ: Sequences of observations. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. length: Batch of sequence lengths. Returns: Summary tensor. adjust_penaltyFzpolicy should change)messagerzcurrent penalty: z kl change: g?c>[R"TRRTRS-5S/S5$)N?rzincrease penalty r'rrVrrZsr r!.PPOAlgorithm._adjust_penalty..-BHHT]]11$--#2EFM`ar#gffffff?c>[R"TRRTRS- 5S/S5$)Nr rzdecrease penalty r!r"sr r!r#r$r#N kl_changepenalty)r'rfrGr reduce_allequalrvrrVrjrrr rr~rr4rfloatrzr{r) rZr6rrrr assert_change print_penaltyr&maybe_increasemaybe_decreases ` r rPPOAlgorithm._adjust_penaltys ' ( f-goobmmBHHW\\84T&U&+.DFmhhq4==/3FGm " "M#A BNN JJw--hGLLRYR`R`a  HHY ]C  dll444 4 a    dll444 4 a   C  " "NN#C Dzz JJ  k9 5 JJ  i 7!   E D+ ) ( C B E D D+ ) ( (s>BI-CH$ I*A'H5 I$ H2 .I5 I ?I Ic`[R"S5 [R"URSR5n[R "USSS24USS2S4:[R 5nX-n[R"US5sSSS5 $!,(df  g=f)zSet padding elements of a batch of sequences to zero. Useful to then safely sum along the time dimension. Args: tensor: Tensor of sequences. length: Batch of sequence lengths. Returns: Masked sequences. maskrNmasked)r'rfrrrCrrUr)rZtensorrrange_r1r2s r rPPOAlgorithm._masksx v xx Q--.f WWVD!G_vag6 Cd}f   vx 0   s A>B B-)r0r4rJr2rQrSrRrLr>r?rGr7rYrVr9r3r1N)__name__ __module__ __qualname____firstlineno____doc__rbrorrrrrrrrrrrr__static_attributes__r#r r r $sXJ<I|$'OR2:^" 7S:&6P JDB44Dl) V1r#r )r: __future__rrrrEtensorflow.compat.v1compatv1r' Exception tensorflowrerrr objectr r<r#r rDsP '%##C16C1s A A A