h SrSSKJr SSKJrJr SSKrSSKr SSK J r J r SSK Jr S/r"SS\ R \ \ \ \ 4\ R"R$5rg) zA collection of wrappers for modifying the reward with an internal state. * ``NormalizeReward`` - Normalizes the rewards to a mean and standard deviation ) annotations)Any SupportsFloatN)ActTypeObsType)RunningMeanStdNormalizeRewardc^\rSrSrSrSS Sjjr\S Sj5r\RS Sj5rS U4Sjjr Sr U=r $) r aNormalizes immediate rewards such that their exponential moving average has an approximately fixed variance. The property `_update_running_mean` allows to freeze/continue the running mean calculation of the reward statistics. If `True` (default), the `RunningMeanStd` will get updated every time `self.normalize()` is called. If False, the calculated statistics are used but not updated anymore; this may be used during evaluation. A vector version of the wrapper exists :class:`gymnasium.wrappers.vector.NormalizeReward`. Note: In v0.27, NormalizeReward was updated as the forward discounted reward estimate was incorrectly computed in Gym v0.25+. For more detail, read [#3154](https://github.com/openai/gym/pull/3152). Note: The scaling depends on past trajectories and rewards will not be scaled correctly if the wrapper was newly instantiated or the policy was changed recently. Example without the normalize reward wrapper: >>> import numpy as np >>> import gymnasium as gym >>> env = gym.make("MountainCarContinuous-v0") >>> _ = env.reset(seed=123) >>> _ = env.action_space.seed(123) >>> episode_rewards = [] >>> terminated, truncated = False, False >>> while not (terminated or truncated): ... observation, reward, terminated, truncated, info = env.step(env.action_space.sample()) ... episode_rewards.append(reward) ... >>> env.close() >>> np.var(episode_rewards) np.float64(0.0008876301247721108) Example with the normalize reward wrapper: >>> import numpy as np >>> import gymnasium as gym >>> env = gym.make("MountainCarContinuous-v0") >>> env = NormalizeReward(env, gamma=0.99, epsilon=1e-8) >>> _ = env.reset(seed=123) >>> _ = env.action_space.seed(123) >>> episode_rewards = [] >>> terminated, truncated = False, False >>> while not (terminated or truncated): ... observation, reward, terminated, truncated, info = env.step(env.action_space.sample()) ... episode_rewards.append(reward) ... >>> env.close() >>> np.var(episode_rewards) np.float64(0.010162116476634746) Change logs: * v0.21.0 - Initially added * v1.0.0 - Add `update_running_mean` attribute to allow disabling of updating the running mean / standard c [RRRXUS9 [RRX5 [ SS9Ul[R"S/5Ul X l X0l SUl g)a6This wrapper will normalize immediate rewards s.t. their exponential moving average has an approximately fixed variance. Args: env (env): The environment to apply the wrapper epsilon (float): A stability parameter gamma (float): The discount factor that is used in the exponential moving average. )gammaepsilon)shapegTN) gymutilsRecordConstructorArgs__init__Wrapperr return_rmsnparraydiscounted_rewardr r_update_running_mean)selfenvr rs \/home/james-whalen/.local/lib/python3.13/site-packages/gymnasium/wrappers/stateful_reward.pyrNormalizeReward.__init__Msc ''00G0T T'(r2!#3%  $(!cUR$)zRProperty to freeze/continue the running mean calculation of the reward statistics.r)rs rupdate_running_mean#NormalizeReward.update_running_meancs(((rcXlg)z[Sets the property to freeze/continue the running mean calculation of the reward statistics.Nr!)rsettings rr"r#hs %,!rct>[TU]U5up#pEnURUR-SU- -[ U5-UlUR (a%UR RUR5 U[R"UR RUR-5- nX'XEU4$)z?Steps through the environment, normalizing the reward returned.) supersteprr floatrrupdatersqrtvarr) ractionobsreward terminated truncatedinfonormalized_reward __class__s rr)NormalizeReward.stepms497<3G0ZD"&!7!7$**!D N" &M"  $ $ OO " "4#9#9 :#RWWT__-@-@4<<-O%PPzdBBr)rrrr r)gGz?g:0yE>)rzgym.Env[ObsType, ActType]r r*rr*)returnbool)r%r8)r.rr7z9tuple[ObsType, SupportsFloat, bool, bool, dict[str, Any]]) __name__ __module__ __qualname____firstlineno____doc__rpropertyr"setterr)__static_attributes__ __classcell__)r5s@rr r s~4r ) &)) ),)), ,CC BCCr)r= __future__rtypingrrnumpyr gymnasiumrgymnasium.corerrgymnasium.wrappers.utilsr__all__rrrr rrrrIsW #%+3  hCKK'723SYY5T5ThCr