hT5 ^ % S r SSKJr SSKJr SSKJrJrJr SSK r SSK
J r SSK r
SSKJr SSKJr SSKJr SS KJrJr SS
KJr SS KJr SS KJ r \( a SSK r " S
S\5 r " S S\5 r \ R@ r!S\"S' S r# " S S\\ \ R@ \$\%\&\ S4 5 r' " S S\\5 r(\)S:X at \ " \(" SS95 r*\*RW 5 u r,r-\." \,\-5 Sr/\/( d: \$" \0" S 5 5 r1\*Re \15 u r,r3r/r4r-\." \,\3\/\4\-5 \/( d M: \5" 5 gg) zpThis module provides a CliffWalking functional environment and Gymnasium environment wrapper CliffWalkingJaxEnv. ) annotations)path)
TYPE_CHECKING
NamedTuple TypeAliasN)spaces)FunctionalJaxEnv)DependencyNotInstalled)ActTypeFuncEnv)EzPickle)
AutoresetMode)HumanRenderingc \ rS rSr% SrS\S' S\S' S\S' S\S ' S
\S ' S \S
' S\S' S\S' S\S' S\S' S\S' S\S' S\S' Srg)RenderStateType znA named tuple which contains the full render state of the Cliffwalking Env. This is static during the episode.zpygame.Surfacescreenztuple[int, int]shapeintnS cell_sizez
np.ndarraycliffzEtuple[pygame.Surface, pygame.Surface, pygame.Surface, pygame.Surface]
elf_images start_imggoal_imgztuple[str, str]bg_imgsz%tuple[pygame.Surface, pygame.Surface]mountain_bg_imgnear_cliff_imgsnear_cliff_img cliff_img N__name__
__module__ __qualname____firstlineno____doc____annotations____static_attributes__r! ]/home/james-whalen/.local/lib/python3.13/site-packages/gymnasium/envs/tabular/cliffwalking.pyr r sR x G UU
::$$99 r* r c 8 \ rS rSr% SrS\S' S\S' S\S' S rg
) EnvState. zEA named tuple which contains the full state of the Cliffwalking game. jax.Arrayplayer_positionr last_actionboolfallenr! Nr" r! r* r+ r- r- . s O Lr* r- r PRNGKeyTypec 6 U S S:H U S S: - U S S:* - $ )zNChecks to see if the player_position means the player has fallen of the cliff.r
r! )r0 s r+ fell_offr9 9 s:
q 1 " $ 1 # %r* c \ \ rS rSrSr\R " SS\R S9r \R " SSS\R S9r
S /S
\ R S .r
S SS
jjrSSS jjrSSS jjrSSS jjr S SS jjr S S S jjr S S S jjrS S jrSrg ) CliffWalkingFunctionalB u Cliff walking involves crossing a gridworld from start to goal while avoiding falling off a cliff.
## Description
The game starts with the player at location [3, 0] of the 4x12 grid world with the
goal located at [3, 11]. If the player reaches the goal the episode ends.
A cliff runs along [3, 1..10]. If the player moves to a cliff location it
returns to the start location.
The player makes moves until they reach the goal.
Adapted from Example 6.6 (page 132) from Reinforcement Learning: An Introduction
by Sutton and Barto [1].
With inspiration from:
[https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py](https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py)
## Action Space
The action shape is `(1,)` in the range `{0, 3}` indicating
which direction to move the player.
- 0: Move up
- 1: Move right
- 2: Move down
- 3: Move left
## Observation Space
There are 3 x 12 + 1 possible states. The player cannot be at the cliff, nor at
the goal as the latter results in the end of the episode. What remains are all
the positions of the first 3 rows plus the bottom-left cell.
The observation is a value representing the player's current position as
current_row * ncols + current_col (where both the row and col start at 0).
For example, the starting position can be calculated as follows: 3 * 12 + 0 = 36.
The observation is returned as an `numpy.ndarray` with shape `(1,)` and dtype `numpy.int32` .
## Starting State
The episode starts with the player in state `[36]` (location [3, 0]).
## Reward
Each time step incurs -1 reward, unless the player stepped into the cliff,
which incurs -100 reward.
## Episode End
The episode terminates when the player enters state `[47]` (location [3, 11]).
## Arguments
```python
import gymnasium as gym
gym.make('tablular/CliffWalking-v0')
```
## References
[1] R. Sutton and A. Barto, “Reinforcement Learning:
An Introduction” 2020. [Online]. Available: [http://www.incompleteideas.net/book/RLbook2020.pdf](http://www.incompleteideas.net/book/RLbook2020.pdf)
## Version History
- v0: Initial version release
r r6 )lowhighdtype/ r7 )r= r> r r? rgb_array ) render_modes
render_fpsautoreset_modeNc 6 UR n[ R " US SUS:H - - SUS:H - - US SUS:H - - SUS:H - - /5 n[ R " [ R " [ R " US S5 S5 [ R " [ R " US S5 S5 /5 n[ U5 n[ R " US SU-
- SU- - US SU-
- /5 n[
UR S5 US US9nU$ ) z9The Cliffwalking environment's state transition function.r r7 r6 )rH r0 r1 r3 )r0 jnparraymaximumminimumr9 r- reshape)selfstateactionkeyparams new_positionr3 new_states r+
transition!CliffWalkingFunctional.transition s- ,, yy Q1! #45v{9KL Q1! #45v{9KL
yy CKK Q ;Q? CKK Q #>?383D3D3Je / /PQ ##J{$;WE# & R\\ HHV%%..~> ?i
O (R
s F' 'F=c SSK nUR R 5 UR " 5 g! [ a n[ S5 UeSnAff = f)zCloses the render state.r Nz@pygame is not installed, run `pip install "gymnasium[toy-text]"`)rr rs r
displayquit)rQ r rr es r+ render_close#CliffWalkingFunctional.render_close] sJ
(R
s 2
A
AA
r! N)
rR r- rS zint | jax.ArrayrT r4 rU Nonereturnr- )r[ r4 rU r r r- )rR r- rU r r r/ )
rR r- rS r rl r- rU r r r/ )iX i )r r r r r r )rR r- r r rU r r z"tuple[RenderStateType, np.ndarray])r r r r )r# r$ r% r& r' r Boxrv int32 action_spaceobservation_spacer NEXT_STEPmetadatarX r\ ra re rm r r r r) r! r* r+ r; r; B s; ?B ::!!288 [ R " U 4SU0UD6 [ S0 UD6nUR [ R
5 [ TU ] UU R US9 g)z8Initializes Gym wrapper for cliffwalking functional env. render_mode)r r Nr! )r
__init__r; r{ rh jitsuperr )rQ r kwargsenv __class__s r+ r CliffWalkingJaxEnv.__init__n sW $BKB6B$.v.
cgg
]]#
r* r! r )r z
str | None) r# r$ r% r& r' r r r)
__classcell__)r s @r+ r r i s F!,
RMH
r* r __main__rB )r FzPlease input an action
)6r'
__future__r osr typingr r r rh jax.numpynumpyrL rv gymnasiumr !gymnasium.envs.functional_jax_envr gymnasium.errorr
!gymnasium.experimental.functionalr r gymnasium.utilsr
gymnasium.vectorr gymnasium.wrappersr rr r r- Arrayr4 r( r9 r floatr2 r; r r# r resetobsinfoprintre inputrS steprm truncatedexitr! r* r+ r s, v " 7 7
> 2 > $ * - j *z Y "d HciieT?D HIdN
)8
$ z + D
EC IC #tHU56714&1A.VXy$
c68Y5 h
F! r*