h#4 S SK Jr S SKJr S SKJr S SKJr S SKr S SK
r S SK
J r J
r
S SKJr S SKJr S rS rS
rS r\S S /\S S /\S S /\S S /0r " S
S\ 5 rg) )closing)StringIO)path)AnyN)Envspaces)categorical_sample)DependencyNotInstalled c ^ \ rS rSrSr/ SQSS.rSS\S- S\4S jjrS
\ R S \ R 4S jr S
\ \
\ R - S\
S \ \\\\
\4 4S jrS rSSS.S\
S- S\S- 4U 4S jjjrS rS rS rS rSrU =r$ )CliffWalkingEnv u
Cliff walking involves crossing a gridworld from start to goal while avoiding falling off a cliff.
## Description
The game starts with the player at location [3, 0] of the 4x12 grid world with the
goal located at [3, 11]. If the player reaches the goal the episode ends.
A cliff runs along [3, 1..10]. If the player moves to a cliff location it
returns to the start location.
The player makes moves until they reach the goal.
Adapted from Example 6.6 (page 132) from Reinforcement Learning: An Introduction
by Sutton and Barto [1].
The cliff can be chosen to be slippery (disabled by default) so the player may move perpendicular
to the intended direction sometimes (see `is_slippery`).
With inspiration from:
[https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py](https://github.com/dennybritz/reinforcement-learning/blob/master/lib/envs/cliff_walking.py)
## Action Space
The action shape is `(1,)` in the range `{0, 3}` indicating
which direction to move the player.
- 0: Move up
- 1: Move right
- 2: Move down
- 3: Move left
## Observation Space
There are 3 x 12 + 1 possible states. The player cannot be at the cliff, nor at
the goal as the latter results in the end of the episode. What remains are all
the positions of the first 3 rows plus the bottom-left cell.
The observation is a value representing the player's current position as
current_row * ncols + current_col (where both the row and col start at 0).
For example, the starting position can be calculated as follows: 3 * 12 + 0 = 36.
The observation is returned as an `int()`.
## Starting State
The episode starts with the player in state `[36]` (location [3, 0]).
## Reward
Each time step incurs -1 reward, unless the player stepped into the cliff,
which incurs -100 reward.
## Episode End
The episode terminates when the player enters state `[47]` (location [3, 11]).
## Information
`step()` and `reset()` return a dict with the following keys:
- "p" - transition proability for the state.
As cliff walking is not stochastic, the transition probability returned always 1.0.
## Arguments
```python
import gymnasium as gym
gym.make('CliffWalking-v1')
```
## References
[1] R. Sutton and A. Barto, “Reinforcement Learning:
An Introduction” 2020. [Online]. Available: [http://www.incompleteideas.net/book/RLbook2020.pdf](http://www.incompleteideas.net/book/RLbook2020.pdf)
## Version History
- v1: Add slippery version of cliffwalking
- v0: Initial version release
)human rgb_arrayansi ) render_modes
render_fpsN render_mode is_slipperyc SU l [ R " SU R 5 U l [ R " U R 5 U l SU l X l [ R " U R [ S9U l
SU R SSS24' 0 U l [ U R
5 H n[ R " X0R 5 n[ U R 5 Vs0 s H oU/ _M snU R U' U R U[ 5 U R U [ ' U R U[ 5 U R U [ ' U R U[" 5 U R U [" ' U R U[$ 5 U R U [$ ' GM [ R " U R
5 U l S U R& U R ' [( R* " U R
5 U l [( R* " U R 5 U l Xl S
U l U R S U R2 S - U R S U R2 S - 4U l S U l S U l S U l S U l S U l S U l S U l! S U l" S U l# g s snf ) N)r )r
r r )dtypeTr
r r g ?)< r r )$shapenpravel_multi_indexstart_state_indexprodnSnAr zerosbool_cliffPrange
unravel_index_calculate_transition_probUPRIGHTDOWNLEFTinitial_state_distribr Discreteobservation_space action_spacer cell_size window_sizewindow_surfaceclock
elf_images start_imggoal_img cliff_imgmountain_bg_imgnear_cliff_imgtree_img)selfr r spositionas ^/home/james-whalen/.local/lib/python3.13/site-packages/gymnasium/envs/toy_text/cliffwalking.py__init__CliffWalkingEnv.__init__h s*
!#!5!5fdjj!I ''$**%& hhtzz6 # AqtG tww A''::6H(-dgg71B7DFF1I ; ;Hb IDFF1IbM # > >x ODFF1Ie " = =h MDFF1IdO " = =h MDFF1IdO
&(XXdgg%6"=@""4#9#9:!' !9 "OODGG 4& " JJqMDNN1 - - JJqMDNN1 - -
#
# "
= 8s 1 Kcoordreturnc [ US U R S S-
5 US' [ US S5 US' [ US U R S S-
5 US' [ US S5 US' U$ )z5Prevent the agent from falling out of the grid world.r r )minr max)r? rF s rC _limit_coordinates"CliffWalkingEnv._limit_coordinates sq uQxA !23auQx#auQxA !23auQx#a currentmovec U R ( d [ U /nO'US-
S- X"S- S- 4 Vs/ s H n[ U PM nn/ nU GH n[ R " U5 [ R " U5 - nU R U5 R [ 5 n[ R " [ U5 U R 5 nU R [ U5 ( a- UR S[ U5 - U R SS45 M U R S S-
U R S S-
4n [ U5 U :H n
UR S[ U5 - USU
45 GM U$ s snf )a Determine the outcome for an action. Transition Prob is always 1.0.
Args:
current: Current position on the grid as (row, col)
delta: Change in position for transition
Returns:
Tuple of ``(transition_probability, new_state, reward, terminated)``
where `transition_probability` is 1 if the environment is not slippery, otherwise 1/3 for `move`
and the perpendicular moves.
r r iFr r )r POSITION_MAPPINGr arrayrK astypeintr tupler r' appendlenr! ) r? rN rO deltasactoutcomesdelta new_position new_stateterminal_state
is_terminateds rC r+ *CliffWalkingEnv._calculate_transition_prob sI &t,-F 48!8q.$PQUV1W1W# %1W
E 88G,rxx>L 22<@GGLL,,U<-@$**MI{{5 ./S[ $2H2H$PU VW"&**Q-!"3TZZ]Q5F!G %l 3~ E
S[ )R OP s E+c U R U R U n[ U Vs/ s H o3S PM snU R 5 nX$ u pVpsX`l Xl U R
S:X a U R
5 [ U5 XsSSU04$ s snf )Nr r Fprob)r( r@ r np_random
lastactionr renderrT )r? rB transitionstipr@ rs rC stepCliffWalkingEnv.step s ffTVVnQ' k :k!k :DNN K ^
a w & KKM1vqUVQK// ;s B
)seedoptionsrm rn c > [ TU ] US9 [ U R U R 5 U l S U l U R S:X a U R 5 [ U R
5 SS04$ )N)rm r rb r )
superresetr r0 rc r@ rd r re rT )r? rm rn __class__s rC rq CliffWalkingEnv.reset s^
4
#D$>$>O w & KKM466{VQK''rM c U R cG U R c e[ R R SU R R
S35 g U R S:X a U R
5 $ U R U R 5 $ )NzYou are calling render method without specifying any render mode. You can specify the render_mode at initialization, e.g. gym.make("z", render_mode="rgb_array")r )r specgymloggerwarnid _render_text _render_gui)r? s rC re CliffWalkingEnv.render s #99( (( JJOO""&)),, /JL
v %$$& &##D$4$45 5rM c SS K nU R c UR 5 US:X a` UR
R 5 UR
R
S5 UR
R U R 5 U l O UR U R 5 U l U R c UR R 5 U l
U R Gc [ R " [ R " [" 5 S5 [ R " [ R " [" 5 S5 [ R " [ R " [" 5 S5 [ R " [ R " [" 5 S5 /nU Vs/ s HB nUR$ R' UR( R+ U5 U R, 5 PMD snU l
U R. cs [ R " [ R " [" 5 S 5 nUR$ R' UR( R+ U5 U R, 5 U l U R0 cs [ R " [ R " [" 5 S
5 nUR$ R' UR( R+ U5 U R, 5 U l U R2 c [ R " [ R " [" 5 S 5 [ R " [ R " [" 5 S 5 /nU Vs/ s HB nUR$ R' UR( R+ U5 U R, 5 PMD snU l U R4 c [ R " [ R " [" 5 S
5 [ R " [ R " [" 5 S5 /nU Vs/ s HB nUR$ R' UR( R+ U5 U R, 5 PMD snU l U R6 cs [ R " [ R " [" 5 S5 nUR$ R' UR( R+ U5 U R, 5 U l [9 U R: 5 GH n [< R> " XR@ 5 u pXR, S - XR, S - 4n U
S- U S- - n
U R RC U R2 U
U 5 U RD X4 ( a& U R RC U R6 U 5 XR@ S S-
: aB U RD U
S- U 4 ( a) U R RC U R4 U
U 5 XRF :X a& U R RC U R. U 5 XR: S-
:X a& U R RC U R0 U 5 XRH :X d GM U S U S SU R, S - -
4nU RJ b U RJ OSnU R RC U R U U5 GM US:X a] URL RO 5 UR
RQ 5 U R RS U RT S 5 g [< RV " [< RX " URZ R] U R 5 5 SS9$ ! [ a n[ S5 UeS nAff = fs snf s snf s snf )Nr z@pygame is not installed, run `pip install "gymnasium[toy-text]"`r CliffWalkingzimg/elf_up.pngzimg/elf_right.pngzimg/elf_down.pngzimg/elf_left.pngz
img/stool.pngzimg/cookie.pngzimg/mountain_bg1.pngzimg/mountain_bg2.pngz img/mountain_near-cliff1.pngz img/mountain_near-cliff2.pngzimg/mountain_cliff.pngr r g?r )r r r )axes)/pygame ImportErrorr
r6 initdisplay set_captionset_moder5 Surfacer7 timeClockr8 r joindirname__file__ transformscaleimageloadr4 r9 r: r< r= r; r) r# r r* r blitr' r! r@ rd eventpumpupdatetickmetadata transposerR surfarraypixels3d)r? moder ehikersf_name file_namebg_imgsnear_cliff_imgsr@ rowcolposcheck_board_maskelf_pos last_actions rC r{ CliffWalkingEnv._render_gui s
& KKMw##%**>:&,nn&=&=d>N>N&O#&,nnT5E5E&F# :: **,DJ ?? " $,,x02BC $,,x02EF $,,x02DE $,,x02DE F % $F &&v||'8'8'@$..Q $ DO >> ! $,,x"8/JI # - - 3 3 !!),dnn DN == $,,x"8:JKI " , , 2 2 !!),dnn DM ' $,,x02HI $,,x02HIG &$ %F &&v||'8'8'@$..Q %$D & $,,x02PQ $,,x02PQ O .# -F &&v||'8'8'@$..Q -#D >> ! $,,x"8:RSI # - - 3 3 !!),dnn DN tww A''::6HC **C..2C,CDC "Qwq 0 $ $T%9%9:J%KS Q{{38$##(( =ZZ]Q&&4;;sQw|+D##(()<)<=M)NPST***##(( =GGaK ##(( <FF{ q63q6C$..2C,C#CD15 1LdooRS ##(( )EwO# & 7? LL NN ! ! # JJOODMM, 7 8<<))2243F3FGHy
i (R
, &$#s, ^) !A _!A _ !A _)
_3 ^??_c 6 [ 5 n[ U R 5 H n[ R " X R
5 nU R U:X a SnO"US:X a SnOU R U ( a SnOSnUS S:X a UR 5 nUS U R
S S-
:X a UR 5 nUS-
nUR U5 M UR S5 [ U5 UR 5 sS S S 5 $ ! , ( d f g = f) Nz x )r
z T z C z o r r
)
r r) r# r r* r r@ r' lstriprstripwriter getvalue)r? outfiler@ rA outputs rC rz CliffWalkingEnv._render_textA s *tww A''::6Hvv{ W$ X& {a {djjma // $ MM& !% &
d
W
##%
s 0D
Dc | U R b/ SS KnUR R 5 UR 5 g g )Nr )r6 r r quit)r? r s rC closeCliffWalkingEnv.close\ s/ * NN ! KKM +rM )r( r' r3 r4 r; r7 r8 r: r0 r rd r< r$ r# r= r2 r r@ r r9 r! r> r5 r6 )NF)__name__
__module__ __qualname____firstlineno____doc__r strr& rD r ndarrayrK listrT rU floatr r+ rk dictrq re r{ rz r __static_attributes__
__classcell__)rr s @rC r r s JZ 7H
/ C$J / D / b
rzz Cy2::- 58
eE3T)* + B
0 +/t (S4Z ( ( (
6 Yv&6 rM r )
contextlibr ior osr typingr numpyr gymnasiumrv r r gymnasium.envs.toy_text.utilsr gymnasium.errorr
r, r- r. r/ rQ r rM rC r sk ! < 2 QAq!fdQGL Kc KrM