ȅiHSSKrSSKrSSKJr SSKJrJr SSKrSSKJ s J r SSK J s J s Jr SSKJr SSKJr SSKJr SSKJr SSKJr SS KJr SS KJr SS KJrJ r J!r!J"r"J#r# \RH"\%5r&S \S \'4Sjr("SS5r)S,S\S\S\\!S\*S \+\+\,S4\+\,S444 Sjjr-\S\,S\,S\,S\#\-S\,S\*S \+\,\R\S-44Sj5r/\S\R\S \,4Sj5r0S,S\S\S \1\,S-S\\!S\*S \+\+\,S4\+\,S444 S!jjr2\RfRhr5S"\R\S\S\\!S \+\1\,\1\,44S#jr6S$\RnS\S\\!S \Rn4S%jr8S&\RrRtS'\\;S \4S(jrg)-N)Sequence)Anycast) LazyString) ShapeType)maybe_run_for_local_tensor) DeviceMesh)redistribute_cost) DTensorSpec)OpSchema) _StridedShardPartial Placement ReplicateShardschemareturncSUS3$)Nz%Implicit redistribution occurred for z/ while ExplicitRedistributionContext was active)rs Y/home/james-whalen/.local/lib/python3.13/site-packages/torch/distributed/tensor/_utils.py#_format_implicit_redistribution_msgrs 26(:i jjc|\rSrSrSr\R "5rSS\S\4Sjjr \ S\ S\ S\ 4S j5r S rS rS rg )ExplicitRedistributionContext a  Within this context manager, DTensor will refuse to perform implicit redistribution, instead raising an error. Manual calls to ``redistribute()`` are required wherever a redistribution must occur to avoid erroring. This can be used to ensure that the user is aware of all redistribution. Note: it is easier to use this mode on just the forward pass of a typical DTensor program, as the backwards pass may contain implicit redistribution calls that are not visible to the user and difficult to replace with manual calls. Redistribution during backward can be made explicit by writing `autograd.Function`s that are no-op during forward and perform a manual redistribution during backwards. enable (bool) if False, disables the context manager. Can be used nested inside an enabled region. strict (bool) if True, triggers on any redistribution. If False, only triggers on redistributions that perform communication. mode (str) Determines what happens when ExplicitRedistributionContext triggers: "raise": raises an exception, "warn" issues a warning enablestrictcXXlX lUS;a[SU35eUS:HUlg)N)raisewarnz Invalid mode r)_enable_strict RuntimeError_raise_on_redistribution)selfrrmodes r__init__&ExplicitRedistributionContext.__init__6s3  ( (tf56 6(,%rsrc_specdst_specredistribution_msgc[URSS5=n(aoSnUR(a"UR(aSnO[ X5S:*nU(d2UR (a [ U5e[RU5 ggg)N_activeTFr) getattr_localr!r"r r$r#loggerwarning)clsr)r*r+instancealloweds robserve_redistribution4ExplicitRedistributionContext.observe_redistribution=svszz9d; ;8 ;G###G/CqHG44&'9::NN#56  I.) 5*[!2%%C#l++M%@%F%F%H! #4^#D  &I  u]3 33r global_tensorc [UR55n[UR55n[U5GHupVURU5nUR 5(a[ [ U5nURS:a[SU35eURn U [U5:dSU S[U5SUS35eX9n X-S:Xd SU SU35eX-X9'[[U55H"n X:wdM XKXIU-:dMXKU-XK'M$ M[U[[45(aGM[S [U5S 35e X44$) a' Compute the local size and stride of a DTensor from the given global tensor info. For example, if we have a global tensor with size (4, 8, 4) and stride (32, 1, 8). If the DTensor placements are [Shard(2)] and world_size is 2; then the local size is (4, 8, 2) and stride is (16, 1, 8). Args: tensor (:class:`torch.Tensor`): Global tensor which DTensor will distribute mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]): The attribute of the DTensor that describes its layout on the mesh topology. Returns: local_shape: A List of int which specifies the size of the local tensor. local_stride: A List of int which specifies the stride of the local tensor. rzOShard placements should have negative dims normalized in the user-facing APIs: rsrtz for placement number .z Global dim z not divisible by mesh size zplacement type z not supported!)resizestrideruis_shardrrrvAssertionErrorrwranger`rrr#type) rrMrNr{ local_strideidxrZrXshard_placementr~global_dim_sizeis rcompute_local_tensor_infors4}))+,K ,,./L#J/ #     "5)4O""Q&$--<,=?(++Is;//  {*DSEUDVW((+uA/ / *4O"2a7 o..J=/Z 7&5%EK "3|,-N$<+B]+RR&2o&FLO .I 7';<<i0AQR R=0@  $$rrSc^^^[T5S:wa [S5e[T5TR:wa%[S[T5STRS35e[ TS[ 5(aT$[ TS[ 5(a[U4Sj5nU"T5n[TR55Vs/sH!n[R"XDRS9PM# nn[R"XdT5 [UU4S j5nU"XF5n[T5n XTSR '[R""U 5$[S [%TS5S 35es snf) a Compute the global size of a DTensor from the given local tensor shape, the mesh and placements. Different from `compute_global_tensor_info`, which assumes sharding is even, this util allgathers local shards' shapes from all ranks and thus can support uneven sharding. NOTE: Currently this function only supports 1D mesh. Args: shape (:class:`torch.Size`): Shape of the local tensor mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]]): The attribute of the DTensor that describes its layout on the mesh topology. Return: tensor_shape: Shape of the global DTensor. r_z>compute_global_tensor_shape only supports 1 placement for now.z/Expected one placement per mesh dim, but found z placements and z mesh dims.rcT>[R"[U5TRS9$)Ndevice)rbrfre device_type)rSrMs r_create_local_shape_tensor?compute_global_tensor_shape.._create_local_shape_tensorws<<U D4D4DE Errc >SnTSRn[[T 55Vs/sH oDU:wdM UPM nnUHEn[R"XXe5(d [ S5eUR 5nX'U- nMG U$s snf)Nrz?Non-sharded dimensions should have identical size across ranks.)rvrrwrbequalr#tolist) r{gathered_shaped_tensorssharded_dim_sumr~d other_dims shape_tensorshape_tensor_listrNrSs r"_validate_and_compute_global_shapeGcompute_global_tensor_shape.._validate_and_compute_global_shapesO"1 ))I%*3u:%6I%6y.!%6JI 7 {{;#:L# UHnTUTU-v M g7fr<r).0rrstride_divisorss r 'compute_local_stride..s$8Q1 aOA..8Qs) rwrurrrrvrrry)rrMrNmesh_idxprjrs` @rcompute_local_striderscC ..O ,  ::<<UA""A3}-. #mA&66#A&$))H*==&/ - 8=c->P8Q rc$[U[R5(aU$[U[5(aU/nOA[ U5S:Xa'[US[ 5(a[ US5nO [ U5n[R"U5$)z Unify variable types of size argument to torch.Size Acceptable types include: int, Sequence[int], Tuple[int], Tuple[Sequence[int]], or torch.Size r_r)r`rbrrdrwrre)r torch_sizes rnormalize_to_torch_sizersn$ ## $V TaJtAw99$q'] $Z ::j !!r)F)?loggingrGcollections.abcrtypingrrrb)torch.distributed._functional_collectives distributed_functional_collectivesrtorch.distributed.tensor._apirf_apirtorch._loggingrtorch._prims_commonrtorch.distributed._local_tensorrtorch.distributed.device_meshr *torch.distributed.tensor._collective_utilsr &torch.distributed.tensor._dtensor_specr #torch.distributed.tensor._op_schemar (torch.distributed.tensor.placement_typesr rrrr getLoggerrBr0strrrrIryrdrVTensorrlrorerR_C#_DTensor_compute_global_tensor_infocompute_global_tensor_inforrr_ops OpOverloadobjectrrrrrrrs$ :://%)F4H>8   8 $kkSk7B7B| 77 7#7 7  5c?E#s(O +, 7t333 3}$ 3  33 3 t# #$33Bu|| O4O4O49t#O4# O4  O4  5c?E#s(O +, O4d#XXII=%<<=% =%#=% 49d3i  =%@H ::H 'H 5=i5HH  ZZH VO ZZ " "O*26*:OO*$.