niJ`SrSSKJr SSKJr SSKJr SSKrSSKJs Jr SSK J r SSK J r SS K J r \R"S S 5r"S S \5rg)zProximal Policy Optimization algorithm. Based on John Schulman's implementation in Python and Theano: https://github.com/joschu/modular_rl/blob/master/modular_rl/ppo.py )absolute_import)division)print_functionN)memory) normalize)utility NetworkOutputz"policy, mean, logstd, value, statec\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSSjrSrg) PPOAlgorithm#zBA vectorized implementation of the PPO algorithm by John Schulman.c XlX lX0lX@lXPl[ R "URRSSSSSS9Ul[ R "URRSSSSSS9Ul URRSURRSURRSURRSURRS4n[R"XeRURS 5Ul["R$"SS5UlURR(=(a [*R,"5n["R."U(aS OS 5 UR1["R2"URR5S S 2S 4["R4"[7UR55S S 9 URR9URRR:SR<5n["R>"S5 [R"U[7U5URS5Ul [*RB"URE[7U5["RF55Ul$["R$"["R2"URR5SSS9Ul%["R$"["R2"URR5SSS9Ul&["R$"["R2"URR5SSS9Ul'S S S 5 S S S 5 ["R$"URRPS["RFS9Ul)URRUURRVSS9Ul,URR[URR\SS9Ul/g !,(df  N=f!,(df  N=f)acCreate an instance of the PPO algorithm. Args: batch_env: In-graph batch environment. step: Integer tensor holding the current training step. is_training: Boolean tensor for whether the algorithm should train. should_log: Boolean tensor for whether summaries should be returned. config: Object containing the agent configuration as attributes. rTnormalize_observ)centerscaleclipnameF normalize_rewardr/gpu:0/cpu:0Nreuser ppo_temporaryepisodes last_action)r last_mean last_logstd)dtypepolicy_optimizervalue_optimizer)0 _batch_env_step _is_training _should_log_configrStreamingNormalizeobserv_observ_filterreward_reward_filteractionr EpisodeMemory update_every max_length_memorytfVariable _memory_indexuse_gpur available_gpusdevice_network zeros_likeoneslennetworkshapevaluevariable_scope _episodescreate_nested_vars zero_statefloat32 _last_state _last_action _last_mean _last_logstdkl_init_penalty_penaltyr! policy_lr_policy_optimizerr"value_lr_value_optimizer) self batch_envstep is_training should_logconfigtemplater5cells e/home/james-whalen/.local/lib/python3.13/site-packages/pybullet_envs/minitaur/agents/ppo/algorithm.py__init__PPOAlgorithm.__init__&s; OJ#!L#66t7M7Ma7P>B=A<=C=A<>">q"A"G"G Hd  _ ---hIHYHY.8:"55dooc)nVXV`V`6abKK doo6L6L(M(--:<++bmmDOO4J4J&K&++68KK doo6L6L(M(--:< . 6&KK < O9QD! . - 6 5s&CQ%#D=Q Q% Q" Q%% Q3c[R"S5 [R"URU5nUR R U5n[R"X#/5 [R"S5sSSS5 sSSS5 $!,(df  O=fSSS5 g!,(df  g=f)zReset the recurrent states and stored episode. Args: agent_indices: 1D tensor of batch indices for agents starting an episode. Returns: Summary tensor. zbegin_episode/N) r2 name_scoper reinit_nested_varsrDr@clearcontrol_dependenciesconstant)rN agent_indices reset_state reset_buffers rV begin_episodePPOAlgorithm.begin_episode^s ' (..t/?/?Ok^^))-8l " "K#> ?{{2 @ ? ) ( @ ? ? ) ( (s$AB/+B B/ B" B// B=c Z^^^[R"S5 URRU5nUR USS2S4[R "UR S5UR5m[R"URTRRU4Sj5mTRRT5SS2S4m[R"URUUU4Sj[5n[R"[ R""URTR$5UR&R)TSS2S45UR*R)TR,SS2S45UR.R)TR0SS2S45/5 [R2"TSS2S4S5[R4"U54sSSS5 sSSS5 $!,(df  O=fSSS5 g!,(df  g=f)zCompute batch of actions and a summary for a batch of observation. Args: observ: Tensor of a batch of observations for all agents. Returns: Tuple of action batch tensor and summary tensor. zperform/Nrc>TR$N)mean)r<srV&PPOAlgorithm.perform..ys c >[RR[RRSTRSS2S45[RRS[R "TR SS2S455[RRSTSS2S45[RRST5/5$)Nrhrstdr-logprob)r2summarymerge histogramrhexplogstd)r-rnr<srVrirj}sBJJ$4$4jj""67<<1+=>jj""5"&&11E*FGjj""8VAqD\:jj""9g6 6 % rkr-)r2r[r* transformr8r:r=rDcondr%policysamplelog_probr&strr^r assign_nested_varsstaterEassignrFrhrGrscheck_numericsidentity)rNr)ror-rnr<s @@@rVperformPPOAlgorithm.performms z """,,V4f fQWorwwv||A/GIYIYZgwwt(('..*?*?AUVf''/15g     g  " "  $ $T%5%5w}} E    " "6!Q$< 0 // ad!3 4    " "7>>!Q$#7 8 $   1x8"++g:NN    # "    # " "s$F H:4H. H H H H*c^^^^[R"S5 [R"TRUUUU4Sj[5sSSS5 $!,(df  g=f)aProcess the transition tuple of the current step. When training, add the current transition tuple to the memory and update the streaming statistics for observations and rewards. A summary string is returned if requested at this step. Args: observ: Batch tensor of observations. action: Batch tensor of actions. reward: Batch tensor of rewards. unused_done: Batch tensor of done flags. unused_nextob: Batch tensor of successor observations. Returns: Summary tensor. z experience/c*>TRTTT5$rg)_define_experience)r-r)r+rNsrVri)PPOAlgorithm.experience..s0G0GPVX^0_rkNr2r[rur%ry)rNr)r-r+ unused_done unused_nextobs```` rV experiencePPOAlgorithm.experiences7" } % WWT&&(_ & % %s -A A c p^^^^ [RRTRR U5TR R U5/5m [R "T /5 TRR(a TRnXTRTRU4nTRRU[R"[TR 555nSSS5 [R "W/5 TRR#U5m[R$"TR R#U55m[R&"TR(UUUU 4Sj[*5nUsSSS5 $!,(df  N=f!,(df  g=f)z=Implement the branch of experience() entered during training.Nc >[RRTTRR5TRR5[RR STR 5[RRST5[RRSTR5[RR ST5/5$)N memory_sizenormalized_observr-normalized_reward) r2rorpr*r,scalarr4rqrE) norm_observ norm_rewardrNupdate_filterssrVri1PPOAlgorithm._define_experience..sBJJ$4$4!!))+!!))+jj t/A/ABjj""#6 Djj""8T->->?jj 3[A6 % rk)r2rorpr*updater,r^r'train_on_agent_actionrErFrGr@appendranger;r#rt reduce_meanrur&ry) rNr)r-r+batchrrorrrs ` @@@rVrPPOAlgorithm._define_experiences;ZZ%%   # #F +   # #F + -.N .!1 2  + +""doot/@/@&He~~$$UBHHS5I,JKf 3 & *''11&9kNN4#6#6#@#@#HIk     g + * 3 2 + *s/BFA9F' F$' F5c^^[R"S5 [R"TRUU4Sj[5sSSS5 $!,(df  g=f)aAdd episodes to the memory and perform update steps if memory is full. During training, add the collected episodes of the batch indices that finished their episode to the memory. If the memory is full, train on it, and then clear the memory. A summary string is returned if requested at this step. Args: agent_indices: 1D tensor of batch indices for agents starting an episode. Returns: Summary tensor. z end_episode/c&>TRT5$rg)_define_end_episode)r`rNsrVri*PPOAlgorithm.end_episode..s0H0H0WrkNr)rNr`s``rV end_episodePPOAlgorithm.end_episodes5 ~ & WWT&&(WY\ ] ' & &s +A AcURRU5up#URRUR- n[ R "[ R"[ R"U5SU55nUVs/sHn[ R"Xe5PM nnURRU[ R"X55XPR-5n[ R"U/5 URR[ R"U5S5nSSS5 [ R"W/5 URURR:n [ R"XR[ 5sSSS5 $s snf!,(df  N|=f!,(df  g=f)z>Implement the branch of end_episode() entered during training.rN)r@datar'r/r4r2rminimumr=gatherr1replacer^ assign_addru _trainingry) rNr`rlength space_left use_episodeselemr inc_index memory_fulls rVr PPOAlgorithm._define_end_episodes8~~**=9H**T-?-??J88BJJrxx '>q'A:NOL:BC($ $-(HC \\ ! !(BIIf,K".1C1C"CEF & *$$//0Fq0IJi + ) -&&$,,*C*CCk WW[..# 6 . - D + * - -s F!=3F&AF7& F47 Gc [R"S5 [R"URURR 5n[R "U/5 URR5nSSS5 Wuup4pVpx[R "[R"US5/5 [R"U5nSSS5 URRU5nURRU5nURX4XVXx5n [R "U /5 URX7U5n SSS5 [R "W /5 UR!X5Xh5n SSS5 [R "W /5 [R""URR%5URR'S55n SSS5 [R "W /5 [(R*"[R,"5URR.5n [R0R3XX/5sSSS5 sSSS5 $!,(df  GN=f!,(df  GN=f!,(df  GNV=f!,(df  GN6=f!,(df  N=f!,(df  O=fSSS5 g!,(df  g=f)zPerform multiple training iterations of both policy and value baseline. Training on the episodes collected in the memory. Reset the memory afterwards. Always returns a summary string. Returns: Summary tensor. trainingNr)r2r[ assert_equalr4r'r/r^r1rassert_greaterr~r*rtr,_update_policy _update_value_adjust_penaltygroupr]r|r variable_summariestrainable_variablesweight_summariesrorp)rN assert_fullrr)r-old_mean old_logstdr+rpolicy_summary value_summarypenalty_summary clear_memoryweight_summarys rVrPPOAlgorithm._trainings  z "OOD$6$6 8Q8QRk " "K= 1||  " 2?C<4vxV " "B$5$5fa$@#A BV$ C"",,V4f"",,V4f**68QW`n " "N#3 4**66B  5 " "M? 3..vT 4 " "O#4 5xx 2 2 4d6H6H6O6OPQ6RS  6 " "L> 2 33B4J4J4L48LL4Q4QSzz `a 3 2! # " 1 1 C B 5 4 3 3 5 5 2 2 2! # " "sA K,$I8?TRTTT5$rg)_update_value_step)_1_2rr)r+rNs rVri,PPOAlgorithm._update_value..sT-D-DVVU[-\rkrZrparallel_iterationsrz value loss: N) r2r[scanrr'update_epochs_valuePrintrr^)rNr)r+rlossro print_losss```` rVrPPOAlgorithm._update_values ~ &gg\ hht||'G'GH2r(235md88At 45~Fj " "D#5 6t||771<= 7 6 ' & 7 6 6 ' & &s$BC4*C C4 C' #C44 Dc fURXU5upE[URRU56upgURR [Xg55n[ R RU[ R RS[ R"U55[R"[Xg5[SS95/5n[ R"U/5 [ R"U5[ R"U5/sSSS5 $!,(df  g=f)zCompute the current value loss and perform a gradient update step. Args: observ: Sequences of observations. reward: Sequences of reward. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. gradient_norm.*)r>N) _value_lossziprMcompute_gradientsapply_gradientsr2rorpr global_normr gradient_summariesdictr^r~) rNr)r+rrro gradients variablesoptimizes rVrPPOAlgorithm._update_value_steps$$VV>)+DE""3y#OP G ( ,kk$W!5 6 - , ,s ,,D"" D0c h[R"S5 URX5Rn[R "X#UR R5nXT- nSURUS-U5-n[RR[RRSU5[RRS[R"U55/5n[R"U5n[R"US5U4sSSS5 $!,(df  g=f)anCompute the loss function for the value baseline. The value loss is the difference between empirical and approximated returns over the collected episodes. Returns the loss tensor and a summary strin. Args: observ: Sequences of observations. reward: Sequences of reward. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. value_lossg?ravg_value_lossN)r2r[r8r>r discounted_returnr'discount_maskrorprqrrr}) rNr)r+rr>return_ advantagerros rVrPPOAlgorithm._value_loss%s | $mmF+11e))&$,,:O:OPg/iIqL&99j   **  |Z 8 **  ,bnnZ.H I" g>>*-j   z< 8' A % $ $s DD## D1c ^^^^^^^[R"S5 [R"UTTRR 5nTR TT5RnTRR(aB[R"XXTTRR TRR5mOXx- m[RRTSS/SS9upTU - [R"U 5S-- m[R"T[R"U5[R"U5/S5m[R"T[R"T5/S5m[R"UUUUUUU4S j[R "TRR"5S S /SS 9up[R"S[R"U 5/S 5n [R$"X/5 U TRR"S-sSSS5 sSSS5 $!,(df  O=fSSS5 g!,(df  g=f)aPerform multiple update steps of the policy. The advantage is computed once at the beginning and shared across iterations. We need to decide for the summary of one iteration, and thus choose the one after half of the iterations. Args: observ: Sequences of observations. action: Sequences of actions. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. reward: Sequences of rewards. length: Batch of sequence lengths. Returns: Summary tensor. update_policyrrT)axes keep_dimsg:0yE>zreturn and value: znormalized advantage: c0>TRTTTTTT5$rg)_update_policy_step) rrr-rrr)rrrNs rVri-PPOAlgorithm._update_policy.._sT-E-E &(J 6.CrkrrZrz policy loss: rN)r2r[r rr'rr8r> gae_lambda lambda_returnnnmomentssqrtrrrrupdate_epochs_policyr^)rNr)r-rrr+rrr>rhvariancerrorrs````` ` @rVrPPOAlgorithm._update_policy?s$  '))&&$,,:O:OPgmmFF+11e  ))&AVAV*.,,*A*AC O uu}}YaVt}Lndt#(9D(@Ai(( bnnW-r~~e/DEG[]i((9r~~i'@&AC[\iggCC hht||'H'HIBPR8235md88At 45Gj " "D#5 6t||88A=> 7 6' ( '& 7 6 6' ( ' 's$G,I H9& I9 I I I"c URX5nURURURX4UXV5up[ UR R U56upUR R[ X55n [RRU [RRS[R"U 55[R"[ X5[SS95/5n [R "U /5 [R""U5[R""U 5/sSSS5 $!,(df  g=f)aCompute the current policy loss and perform a gradient update step. Args: observ: Sequences of observations. action: Sequences of actions. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. advantage: Sequences of advantages. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. rr)rvN)r8 _policy_lossrhrsrrKrrr2rorprrr rrr^r~) rNr)r-rrrrr<rrorrrs rVr PPOAlgorithm._update_policy_stepgsmmF+G%%gllGNNHZ`&/9MD!7!7!I!I$!OPI%%55c)6OPHjj /2>>)+DE""3y#PQ G ( ,kk$W!5 6 - , ,s ,E  Ec^[R"S5 [R"X5n[R"UR [R "X4X5U5S5n [R"[R"XU5[R"X4U5- 5n [R"UR U [R"U5-U5S5*n URU -n URRURR-n [R"[R"X:[R 55m[R""[R$"TS:U4Sj[&5/5 URR([R"X:[R*5-X- S--nSSS5 X-W-n[R,R/[R,R1SU5[R,R1SU 5[R,R1S U 5[R,R1S U 5[R,R1S U5[R,R1S X-5[R,R1SU5[R,R3S [R"U 55[R,R3S[R"U 55[R,R3S[R"U55/ 5n[R"US5n[R4"US5U4sSSS5 $!,(df  GN=f!,(df  g=f)aTCompute the policy loss composed of multiple components. 1. The policy gradient loss is importance sampled from the data-collecting policy at the beginning of training. 2. The second term is a KL penalty between the policy at the beginning of training and the current policy. 3. Additionally, if this KL already changed more than twice the target amount, we activate a strong penalty discouraging further divergence. Args: mean: Sequences of action means of the current policy. logstd: Sequences of action log stddevs of the current policy. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. action: Sequences of actions. advantage: Sequences of advantages. length: Batch of sequence lengths. Returns: Tuple of loss tensor and summary tensor. policy_lossrrc6>[R"ST/S5$)Nrz kl cutoff! )r2r) cutoff_countsrVri+PPOAlgorithm._policy_loss..sRXXa,-WrkrNentropyklsurrogate_loss kl_penalty kl_cutoffkl_penalty_combined avg_surr_lossavg_kl_penaltyavg_policy_loss)r2r[r diag_normal_entropyrrdiag_normal_klrrdiag_normal_logpdf stop_gradientrIr' kl_targetkl_cutoff_factor reduce_sumcastint32r^ruintkl_cutoff_coefrCrorprqrr})rNrhrsrrr-rrrrpolicy_gradientrrcutoff_thresholdrrrors @rVrPPOAlgorithm._policy_losss, } %++D9g >> **W++H$OQW XZ[ ]b  $ $T6 :  $ $X6 B CDo **_r'7'7 'BBF KQPPn==2%j//$,,2O2OO]]2772+@"((#KLl " " 77#GO> O O Oc ^[R"S5 TRX5n[R"[R"[R "UR U55SSS9n[R"STR/S5n[R"Xg/5 [R"TR[R"X#UR UR5U55n[R"X/S5n[R"USTR R"-:U4S j[$5n [R"US TR R"-:U4S j[$5n S S S 5 [R"W W /5 [R&R)[R&R+S W5[R&R+STR5/5sS S S 5 sS S S 5 $!,(df  N=f!,(df  O=fS S S 5 g !,(df  g =f)aAdjust the KL policy between the behavioral and current policy. Compute how much the policy actually changed during the multiple update steps. Adjust the penalty strength for the next training phase if we overshot or undershot the target divergence too much. Args: observ: Sequences of observations. old_mean: Sequences of action means of the behavioral policy. old_logstd: Sequences of action log stddevs of the behavioral policy. length: Batch of sequence lengths. Returns: Summary tensor. adjust_penaltyFzpolicy should change)messagerzcurrent penalty: z kl change: g?c>[R"TRRTRS-5S/S5$)N?rzincrease penalty r2rrIr|rNsrVri.PPOAlgorithm._adjust_penalty..-BHHT]]11$--#2EFM`arkgffffff?c>[R"TRRTRS- 5S/S5$)Nrrzdecrease penalty rrsrVrir r!rkN kl_changepenalty)r2r[r8r reduce_allequalrhrrIr^rrr r rsrur'rfloatrorpr) rNr)rrrr< assert_change print_penaltyr#maybe_increasemaybe_decreases ` rVrPPOAlgorithm._adjust_penaltys ' ( f-goobmmBHHW\\84T&U&+.DFmhhq4==/3FGm " "M#A BNN JJw--hGLLRYR`R`a  HHY ]C  dll444 4 a    dll444 4 a   C  " "NN#C Dzz JJ  k9 5 JJ  i 7!   E D+ ) ( C B E D D+ ) ( (s>BI-CH$ I*A'H5 I$ H2 .I5 I ?I Ic`[R"S5 [R"URSR5n[R "USSS24USS2S4:[R 5nX-n[R"US5sSSS5 $!,(df  g=f)zSet padding elements of a batch of sequences to zero. Useful to then safely sum along the time dimension. Args: tensor: Tensor of sequences. length: Batch of sequence lengths. Returns: Masked sequences. maskrNmasked)r2r[rr=r>rrCr})rNtensorrrange_r.r/s rVrPPOAlgorithm._masksx v xx Q--.f WWVD!G_vag6 Cd}f   vx 0   s A>B B-Nc [R"SUS9 [R"U5nURR=(a [ R "5n[R"U(aSOS5 [R"US5nURRURRRSR5n[RRUUUU[R SS9uupxpS S S 5 [R"WS 5n[R"WS 5n[R"W S 5n [R"R$R'U[R("U55n [+XXU5sS S S 5 $!,(df  N=f!,(df  g =f) aHCompute the network output for a batched sequence of observations. Optionally, the initial state can be specified. The weights should be reused for all calls, except for the first one. Output is a named tuple containing the policy as a TensorFlow distribution, the policy mean and log standard deviation, the approximated state value, and the new recurrent state. Args: observ: Sequences of observations. length: Batch of sequence lengths. state: Batch of initial recurrent states. reuse: Python boolean whether to reuse previous variables. Returns: NetworkOutput tuple. r<rrrr)rT) swap_memoryNrhrsr>)r2r?convert_to_tensorr'r5r r6r7r}r<r#r-r=r>r dynamic_rnnrCcontrib distributionsMultivariateNormalDiagrr_NetworkOutput) rNr)rr{rr5rUrhrsr>rvs rVr8PPOAlgorithm._networksP$  9E 2##F+f $$A)?)?)Ag 99Xh 7""684||##DOO$:$:$@$@$C$I$IJ')uu'8'89?9?9>9;EI (9(K$u 8  tV ,d  2fw/ezz''>>tRVVF^Tf F& ?! 3 2 8 7 3 2s&A(G>BF2BG2 G <G G)r#r'r@r%rErGrFrDr1r4r*rIrKr,r&r$rM)NNT)__name__ __module__ __qualname____firstlineno____doc__rWrcrrrrrrrrrrrrrrr8__static_attributes__rkrVr r #seJ6Qp O>*6^" 7b<>,7,B4&?P764Dl) V1$"@rkr )r@ __future__rrr collections tf.compat.v1compatv1r2rZrrr namedtupler:objectr rBrkrVrJsJ '%''9]^t@6t@rk