ȅiSSKrSSKrSSKrSSKJrJrJrJr SSKrSSK J r SSK J s J r SSKJr SSKJr SSKJr SSKJr SSKJr SS KJr \\!\"\!\!\"\ RF\\$S\"4\RJ4r&Sr'SS\RPS\"S\&S\)4Sjjr*SS\RPS\)S\&S\)4Sjjr+SS\RPS\"S\&S\)S\RP4 Sjjr,SS\RPS\"S\&S\)4Sjjr-SS\RPS\)S\"S\&S\)4 Sjjr.SS\RPS\)S\"S\&S\)4 Sjjr/SS\!\RPS\)S\&S\)S\!\RP4 Sjjr0SS\!\RPS\&S\)S\!\RP4Sjjr1SS \!\RPS\)S\!\"S\&S\)S\!\RP4 S!jjr2S"r3SS\RPS#\!\"S-S$\!\"S-S\&S\)S\RP4 S%jjr4SS\RPS#\!\"S-S$\!\"S-S\&S\)S\RP4 S&jjr5S'\RP4S(jr6S)r7\RpRsS*\6\7S+9 S'\RP4S,jr:S-r;\RpRsS.\:\;S+9 S'\RP4S/jrS3r?\RpRsS4\>\?S+9 S'\RP4S5jr@S6rA\RpRsS7\@\AS+9 S8\!\RP4S9jrBS:rC\RpRsS;\B\CS+9 S8\!\RP4S<jrDS=rE\RpRsS>\D\ES+9 S8\!\RP4S?jrFS@rG\RpRsSA\F\GS+9 SS\RPSB\!\"S\&S\)S\RP4 SCjjrH"SDSE\RP5rISS\&S\)S\$\)\!\"\"44SFjjrJSS\&S\)S\RJ4SGjjrK"SHSI\RR5rN\RpRSJSKSLSM9SN\RPS\RP4SOj5rP\PRSN\RPS\RP4SPj5rRS'\RP4SQjrSSRrT\PRs\S\TS+9 S\U4SSjrVS\RP4STjrW\RSSU\U4SVjj5rYSWrZSXr[SYr\SZr]S[r^S\r_S]r`S^raS_rbS`rcSardSbreScrfSdrgSerhSfriSgrjShrkSirl\RpRSjSk5rn\nRSl\]Sm5 \nRSn\bSm5 \nRSo\aSm5 \nRSp\dSm5 \nRSq\^Sm5 \nRSr\gSm5 \nRSs\hSm5 \nRSt\iSm5 \nRSu\jSm5 \nRSv\kSm5 \nRSw\lSm5 \nRSx\fSm5 \nRSy\\Sm5 \nRSz\cSm5 \RRR\RRRNR5 \RRR\RRRN5 \RpRS{S|5rv\RpRS{Sk5rw/S}Qrx\R\zr{\xH]r|\|S\|RS~5r~\"\S\~35r\vGR\|\GRGRS9 \wR\~\S5 M_ SS\RPS\RPS\US\)S\"4 SjjrSS\RPSN\RPS\)S\US\"S\)4 Sjjr\ GR GRS\ GR GRS\ GR GRS\ GR GRS\ GR GRS\ GR GRS\ GR GRS\ GR GRS0rSS\RPS\)S\US\)4SjjrSS\RPSN\RPS\)4SjjrSS\!\RPS\RPS\)4SjjrSSK JrJrJrJrJ+rJ4rJ.r \\\\\\\\\\\\\\0rg!\a SSKJr GNf=f!\a \R@"S S S 9 S rG N f=f)N)Anycast TYPE_CHECKINGUnion)_maybe_view_chunk_cat) DeviceMesh)get_proxy_mode)_functional_collectives_impl) tree_map_only)is_dynamo_compilingzdUnable to import torchdynamo util `is_torchdynamo_compiling`, so won't support torchdynamo correctly stacklevelcg)NFrc/home/james-whalen/.local/lib/python3.13/site-packages/torch/distributed/_functional_collectives.pyis_torchdynamo_compilingrsrzdist.tensor.DeviceMeshcT[RRRU5$)z Wait on a tensor returned by the collectives ops. Waiting follows device semantics, which means blocking on CPU and synchronizing streams on CUDA. )torchops_c10d_functional wait_tensor)tensors rrrs 99 % % 1 1& 99rselfsrcgrouptagc[X#5n[RRR XU5n[ U5$)a Broadcasts the tensor to all processes in the given process group. Args: src (int): Source rank group (ProcessGroup or List[int]): The process group to work on. tag (str, optional): A unique identifier for the collective. Default: empty string )_resolve_group_namerrr broadcast_maybe_wrap_tensor)rrrr group_namers rr"r"s5%U0J YY ' ' 1 1$Z HF f %%rreduceOpc[X#5n[RRR XR 5U5n[ U5$)a Reduces the tensor data across all machines in such a way that all get the final result. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. )r!rrr all_reducelowerr#)rr%rrr$rs rr'r's<"%U0J YY ' ' 2 249I: VF f %%r gather_dimreturnc`UR5(d [S5e[X#5n[R"U5n[ R RRXU5n[U5nUS:wa1[U[5(aUR5n[XuU5nU$)a Gather tensor data across from all machines and concatenate over ``gather_dim``. Note that it currently only supports gather_dim = 0. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. z/Tensor must be contiguous for all_gather_tensorr) is_contiguousAssertionErrorr!c10d_get_group_size_by_namerrrall_gather_into_tensorr# isinstanceAsyncCollectiveTensorwaitrrr)rrr$ group_sizerress rall_gather_tensorr7s,     NOO$U0J--j9J YY ' ' > > *F V $CQ c0 1 1((*C#CZ@ Jrcl[X#5n[R"U5n[RR R XU5n[RU5nUS:waM[U[5(aUR5n[R"[R"XuSS9US9nU$)a$ Gather tensor data across from all machines and concatenate over ``gather_dim``. Note that it currently only supports gather_dim = 0. This function is the same as all_gather_tensor but will propagate the backwards gradient across workers. See all_gather_tensor for more details on usage. rdim)r!r.r/rr_c10d_functional_autogradr0_FromTorchTensorapplyr1r2r3catchunkr4s rall_gather_tensor_autogradr@s %U0J--j9J YY 0 0 G G *F   (CQ c0 1 1((*Cii C;L Jr scatter_dimc[X45n[R"U5nURU5U-S:wa![ SURS5SUS35eUS:wa+[ R "XUS9n[ R"U5n[ RRRUUR5UU5n[U5n U $)a Reduces the tensor data across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. rinput dimension 0 (" must be a multiple of group_size )r9) r!r.r/sizer-rr?r>rrreduce_scatter_tensorr(r# rr%rArrr$r5 tensor_listrr6s rrGrGs,%U0J--j9J yy *a/!$))A,/QR\Q]]^ _  akk$ D yy% YY ' ' = =  F V $C Jrc[X45n[R"U5nURU5U-S:wa [ SURS5SU35eUS:wa+[ R "XUS9n[ R"U5n[ RRRUUR5UU5n[RU5n U $)a` Reduces the tensor data across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. This function is the same as reduce_scatter_tensor but will propagate the backwards gradient across workers. Currently only the "sum" reduceOp is supported. See reduce_scatter_tensor for more details on usage. rrCrDr9)r!r.r/rFr-rr?r>rr;rGr(r<r=rHs rreduce_scatter_tensor_autogradrK%s&%U0J--j9J yy *a/!$))A,/QR\Q] ^  akk$ D yy% YY 0 0 F F  F   (C Jrc[X#5n[RRR UUR 5U5n[ [[U55$)a Reduces a list of tensors across all machines in such a way that all get the final result. The all tensors in the input list are left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. ) r!rrrall_reduce_coalescedr(listmapr#)rr%rrr$rIs rrMrMMsM&%U0J)),,AA K & 4 55rc[X5n[R"U5n[RR R UUU5n[[[U55$)a Gather a list of tensors across from all machines. Note that it currently only supports gather_dim = 0. The input tensor is left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. ) r!r.r/rrr all_gather_into_tensor_coalescedrNrOr#)rrrr$r5rIs rrQrQisV&%U0J--j9J)),,MM K & 4 55rinputsc [X45n[R"U5n[U5[U5:wa$[ S[U5S[U5S35e[ [ X 55H{unupU RU5U-S:wa&[ SUSU RU5SUSU35eUS:wdMN[R"XUS 9n [R"U 5X'M} [RRRUUR5UU5n [[!["U 55$) a Reduces a list of tensors across all machines in such a way that all get the final result, then scatter the results to corresponding ranks. The input tensors are left unmodified. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. zLength of scatter_dim (z) must equal length of inputs (rErzinput dimension z (rDz for tensor at index r9)r!r.r/lenr- enumerateziprFrr?r>rrreduce_scatter_tensor_coalescedr(rNrOr#) rRr%rArrr$r5idxr:rrIs rrWrWsC,%U0J--j9J ;3v;&%c+&6%77VWZ[aWbVccd e  (K(@A]c ;;s j (A - "3%r&++c*:);;]^h]ii~@CDE  !8++fcBK))K0FKB)),,LL K & 4 55rc[U[RR5(d[ S[ U535e[R RUR5[RR5(agURn[UR5S:a?URSnURSL=(a URR(+$g)Nz$Expected torch._ops.OpOverload, got Fr)r1r_ops OpOverloadr-type_C%_dispatch_has_kernel_for_dispatch_keyname DispatchKeyCompositeImplicitAutograd_schemarT arguments alias_infois_write)tgtschema first_args r _is_view_opris c5::00 1 1CDI;OPP xx55  E%%?? [[F 6  q $$Q' ##4/U 8L8L8U8U4UU!routput_split_sizesinput_split_sizescUb%[SU55(d[SU35eUb%[SU55(d[SU35e[X45n[R"U5nUbUc)UcUb [S5eUR SU-/U-nUn[ RRRUUUU5n[U5$)a Each process splits input tensor and then scatters the split list to all processes in a group. Then concatenate the received tensors from all the processes in the group and return single output tensor. Group can be one of: List[int]: ranks participating in the collective. List[List[int]]: 2D mesh of ranks taking part of this collective in MPMD. ProcessGroup: Will perform a collective using the ranks and tag of the PG. DeviceMesh: Do a SPMD collective over all ranks of the mesh (DeviceMesh, int): Do a MPMD collective over one dimension of the DeviceMesh :: N.B. If you pass a PG or a 1D list to perform a MPMD collective, the compiler won't be able to recover that information and perform collective algebraic optimization. Use other forms of input for that. c3b# UH%n[U[[R45v M' g7fNr1intrSymInt.0rFs r $all_to_all_single..' >PdJtc5<<0 1 1>P-/2All output_split_sizes must be int or SymInt, got c3b# UH%n[U[[R45v M' g7frnrorrs rrtru%WEVT:dS%,,$788EVrw1All input_split_sizes must be int or SymInt, got ^output_split_sizes and input_split_sizes must either be specified together or both set to Noner) allr-r!r.r/shaperrrall_to_all_singler#rrjrkrrr$r5rs rrrs,% >P   !DEWDXY $WEVWWW CDUCVW %U0J--j9J!%6%>"*/@/H 9 #jjmz9:ZG. YY ' ' 9 9  F f %%rcUb%[SU55(d[SU35eUb%[SU55(d[SU35e[X45n[R"U5nUbUc)UcUb [S5eUR SU-/U-nUn[ RRRUUUU5n[RU5$)z2 Same as all_to_all_single but supports autograd. c3b# UH%n[U[[R45v M' g7frnrorrs rrt-all_to_all_single_autograd..rvrwrxc3b# UH%n[U[[R45v M' g7frnrorrs rrtrrzrwr{r|r) r}r-r!r.r/r~rrr;rr<r=rs rall_to_all_single_autogradrs% >P   !DEWDXY $WEVWWW CDUCVW %U0J--j9J!%6%>"*/@/H 9 #jjmz9:ZG. YY 0 0 B B  F  ! !& ))r grad_outputcU$)z Backward for wait_tensor: identity (no-op). Wait is just a synchronization primitive, so gradient flows through unchanged. Args: ctx: Context object grad_output: Gradient from downstream operations Returns: Gradient unchanged (identity) rctxrs rwait_tensor_backwardr,s  rcg)z Setup context for wait_tensor backward. Args: ctx: Context object to save state for backward inputs: Tuple of (tensor,) output: Output from forward pass NrrrRoutputs rwait_tensor_setup_contextr;s rz_c10d_functional::wait_tensor) setup_contextcURnURnUS:wa[SUS35e[RR R UR5X25n[U5SS4$)aS Backward for all_reduce: all_reduce with same reduce_op. Forward aggregates tensors, backward aggregates gradients. Args: ctx: Context object grad_output: Gradient from downstream operations Returns: Tuple of (grad_input, grad_group_name, grad_reduce_op) grad_group_name and grad_reduce_op are None (not differentiable) sumz8all_reduce backward only supports 'sum' reduction, got ''N) r$ reduce_op RuntimeErrorrrrr' contiguousr)rrr$rrs rall_reduce_backwardrMsvJ IEFykQR S  YY ' ' 2 2 )F v d **rcDUup4nXPlUR5Ulg)z Setup context for all_reduce backward. Args: ctx: Context object to save state for backward inputs: Tuple of (input, reduce_op, group_name) output: Output from forward pass Nr$r(r)rrRrinputrr$s rall_reduce_setup_contextris!$* EjNOO%CMrz_c10d_functional::all_reducecURnURn[RRR UR 5SUU5n[U5SS4$)a Backward for all_gather_into_tensor: reduce_scatter with sum. Forward gathers tensors from all ranks, backward scatters gradients back with sum reduction. Args: ctx: Context object with group_name and group_size grad_output: Gradient from downstream operations Returns: Tuple of (grad_input, grad_group_size, grad_group_name) grad_group_size and grad_group_name are None (not differentiable) rN)r$r5rrrrGrr)rrr$r5rs rall_gather_into_tensor_backwardr}s[JJYY ' ' = =   F v d **rc&Uup4nXPlX@lg)z Setup context for all_gather_into_tensor backward. Args: ctx: Context object to save state for backward inputs: Tuple of (input, group_size, group_name) output: Output from forward pass Nr$r5)rrRrrr5r$s r$all_gather_into_tensor_setup_contextrs%+!EzNNrz(_c10d_functional::all_gather_into_tensorcURnURnURnUS:wa[SUS35e[R R RUR5UU5n[U5SSS4$)a Backward for reduce_scatter_tensor: all_gather. Forward reduces and scatters tensors to ranks, backward gathers gradients from all ranks. Args: ctx: Context object with group_name, group_size, and reduce_op grad_output: Gradient from downstream operations Returns: Tuple of (grad_input, grad_reduce_op, grad_group_size, grad_group_name) grad_reduce_op, grad_group_size, grad_group_name are None (not differentiable) rzCreduce_scatter_tensor backward only supports 'sum' reduction, got 'rN) r$r5rrrrrr0rr)rrr$r5rrs rreduce_scatter_tensor_backwardrsJJ IEQR[Q\\] ^  YY ' ' > > F v dD 00rcPUup4pVX`lXPlUR5Ulg)z Setup context for reduce_scatter_tensor backward. Args: ctx: Context object to save state for backward inputs: Tuple of (input, reduce_op, group_size, group_name) output: Output from forward pass Nr$r5r(r)rrRrrrr5r$s r#reduce_scatter_tensor_setup_contextrs&06,EjNNOO%CMrz'_c10d_functional::reduce_scatter_tensorcURnURnURn[RR R UR5UUU5n[U5SSS4$)a Backward for all_to_all_single: all_to_all with reversed split sizes. Forward does all-to-all with specified split sizes, backward reverses them. Args: ctx: Context object with group_name, output_split_sizes, and input_split_sizes grad_output: Gradient from downstream operations Returns: Tuple of (grad_input, grad_output_split_sizes, grad_input_split_sizes, grad_group_name) All except grad_input are None (not differentiable) N) r$rjrkrrrrrr)rrr$rjrkrs rall_to_all_single_backwardrslJ//--YY ' ' 9 9  F v dD 00rc2Uup4pVX`lX@lXPlg)z Setup context for all_to_all_single backward. Args: ctx: Context object to save state for backward inputs: Tuple of (input, output_split_sizes, input_split_sizes, group_name) output: Output from forward pass N)r$rjrk)rrRrrrjrkr$s rall_to_all_single_setup_contextrs"@FUR5R5$rn) trigger_waittolistrs rrAsyncCollectiveTensor.tolists  "))++rc>Ub [S5eUSn[U5$)Nz5meta must be None for AsyncCollectiveTensor unflattenr)r-r2) inner_tensorsmeta outer_size outer_striders r__tensor_unflatten__*AsyncCollectiveTensor.__tensor_unflatten__s/   G V$$T**rNexpected_metadata expected_typecJU[RLagUR5$rn)rrr)rrrs r#__coerce_same_metadata_as_tangent__9AsyncCollectiveTensor.__coerce_same_metadata_as_tangent__s"  ,  ""rr*c*SUR5S3$)NzAsyncCollectiveTensor(rE)rrs r__repr__AsyncCollectiveTensor.__repr__ s'(9(9(;'>rcxUR(d[UR5nSUlU$UR$NT)rrr)routs rr"AsyncCollectiveTensor.trigger_waits-~~dii(C!DNJ99 rc,[UR5$rn)rrrs rr3AsyncCollectiveTensor.waits499%%rcUR$)zOThis method enables _functional_collectives_impl to test if a tensor is an ACS)rrs r_get_acs_underlying_tensor0AsyncCollectiveTensor._get_acs_underlying_tensors yyrc^ U[RRRRLa&U"USR US5n[ U5nU$[U5m S[ 4U 4SjjnS[R4Sjn[[ Xs5n [[ Xt5n U"U 0U D6n T (a[[RX5n U $)Nrr ecJ>T(dUR5$UR$rn)rr)r is_view_ops runwrap8AsyncCollectiveTensor.__torch_dispatch__..unwrap)s~~''66Mrc\[U[5(a [S5e[U5nU$)NzICannot wrap an AsyncCollectiveTensor inside another AsyncCollectiveTensor)r1r2r-)rr6s rwrap6AsyncCollectiveTensor.__torch_dispatch__..wrap/s1!233$_(*CJr) rratenviewdefaultrr2rirr ) rfunctypesargskwargsr6 wrapper_resrrunwrapped_argsunwrapped_kwargsrrs @r__torch_dispatch__(AsyncCollectiveTensor.__torch_dispatch__s 599>>&&.. .tAw||T!W-C/4K  &  +   ELL ''ON7&67  d8C rc>UR5R5$rn)r3numpyrs rr AsyncCollectiveTensor.numpyDsyy{  ""r)rrn)rN)__name__ __module__ __qualname____firstlineno____doc__rr__annotations__bool __slots__ staticmethodrrrrrr\rstrrrr3r classmethodrr __static_attributes__rrrr2r2s ,,O%I 5<<  ,++DH#!$#59D[#?#?&ell&$$L#rr2c [(aSnSnOSnSn[U[5(a[US[5(abU"U5n/nSnUHNnURU5 US:wa)U[ U5:wa[ SUS[ U535e[ U5nMP GOU"U5n[ U5nGO[U[ R5(aB[ R"U5n[ U5nU=(d [R"U5nGOG[U[5(alURS :wa [S 5eUR5n[ R"U5n[ U5nU=(d [R"U5nO[U[5(a[ U5S :Xa[US[5(at[US [ 5(a\USn US n U RU 5n[ R"U5n[ U5nU=(d [R"U5nO[ S 5e[ S 5eXU4$)a% _expand_group desugars the different RANK_TYPES types into a canonical format that is traceable. By having this be part of the explicit eager codepath, we avoid having to specialize behavior inside torchdynamo and can still interoperate with processgroup objects or other untraceable forms. c>[[[[U5$rnrrNrpxs rcast_listlistint'_expand_group..cast_listlistintYsT#Y+ +rc0[[[U5$rnrrs r cast_listint#_expand_group..cast_listint\sS 1% %rcU$rnrrs rrrcHrcU$rnrrs rr r!fr#rrz$group sizes must be identical found z and r JOnly 1D mesh is supported, pass in (DeviceMesh, int) together if mesh > 1Dr1Invalid tuple for group must be (DeviceMesh, int)z[Invalid type for group, must be one of List, Processgroup, DeviceMesh or (DeviceMesh, int).)rr1rNextendrT ValueErrorr ProcessGroupget_process_group_ranksr._get_group_tagrndimr- get_grouptuplerp) rrrr  nested_listrr5rspgdmeshr:s rrrMs} , &  % eAh % %*51KGJ!r"# c"g(=$>zl%PSTVPWyY!W "#5)GWJ E4,, - -..u5\ /T((/ E: & & ::? \ __ ..r2\ ,T((, E5 ! ! J!O58Z0058S))!HE(C%B2226GWJ0,,R0CPQ Q i   * %%rc,[U[R5(a UR$[U[5(a[ [ RU5$[U[5(a*URS:wa [S5eURS$[U[5(ac[U5S:XaI[US[5(a1[US[5(aUSnUSnURU$[S5e[U[ 5(aU[#5(d[$R&"S[(SS9 [ R*"[ [ [U5U5$[S [-U5S U35e) z3 Given group in RANK_TYPES, return the group name. r r&rrr'zThe combination of ranks + tag as process group identifier has been deprecated. Please switch to using ProcessGroup, DeviceMesh, or group name instead.rzUnsupported group type: z, )r1rr*r$rrr. GroupNamerr-r-_dim_group_namesr/rTrpr)rNrwarningswarn FutureWarning$_resolve_group_name_by_ranks_and_tagr\)rrr3r:s rr!r!sU %**++ E3   DNNE** E: & & ::? \ %%a(( E5 ! ! J!O58Z0058S))!HE(C))#. .PQ Q E4 ')) MMI  88d3i9OQTUU3DK=5'JKKrc\rSrSrSr\S\RS\R4Sj5r\S\RS\R4Sj5r Sr g ) r<iza _FromTorchTensor allows autograd to propagate from a normal Tensor to an AsyncCollectiveTensor. rr*c[U5$rn)r#)rrs rforward_FromTorchTensor.forwards "%((rrcU$rnrrs rbackward_FromTorchTensor.backwardsrrN) r r rrrrrrr>rArrrrr<r<s_ )||) )) 5<<ELLrr<z'_c10d_functional::_wrap_tensor_autogradrz(Tensor input) -> Tensor) mutates_argsrgrc[U5$)aO Custom op that allows autograd to propagate from a normal Tensor to an AsyncCollectiveTensor. This is the low-level implementation. Users should call _maybe_wrap_tensor directly. Args: input: Input tensor to wrap in AsyncCollectiveTensor Returns: AsyncCollectiveTensor wrapping the input (or wait_tensor result if tracing) )r2rs r_wrap_tensor_autogradrFs$ ! ''rc.[R"U5$)z( Meta kernel for _wrap_tensor_autograd. r empty_likerEs r_rJs   E ""rcU$)a Backward for _wrap_tensor_autograd: identity (no-op). The wrapping is just for async optimization, gradients flow through unchanged. Args: ctx: Context object (unused) grad_output: Gradient from downstream operations Returns: Gradient unchanged (identity) rrs r_wrap_tensor_autograd_backwardrLs  rcg)z Setup context for _wrap_tensor_autograd backward. Args: ctx: Context object to save state for backward (nothing to save) inputs: Tuple of (input,) output: Output from forward pass Nrrs r#_wrap_tensor_autograd_setup_contextrN s rcN[5(ag[RR[RRR 5bg[RR [RRR5(ag[5SL$r) rrr]_get_dispatch_mode_TorchDispatchModeKeyFAKE&_dispatch_tls_is_dispatch_key_includedr`PythonDispatcherr rrr_are_we_tracingrUss!! xx""588#A#A#F#FGS xx66 --  4 ''rcL[5(a [U5$[U5$rn)rUrrFrs rr#r#*s!4   &&rvaluec#f# [RRR5n[RRR U5 Sv [RRR U5 g![RRR U5 f=f7f)a Context manager to temporarily set whether inflight collectives are allowed as torch.compile graph inputs. Common use case is when the collective is issued in eager (with `async_op=True`) but waited in compiled region: ``` def all_reduce_eager(x): y = x * x req = dist.all_reduce(y, op=dist.ReduceOp.SUM, async_op=True) return y @torch.compile(fullgraph=True) def all_reduce_wait_compiled(y): torch.ops.c10d_functional.wait_tensor(y) return y * y x = torch.ones(1280, 1280, device="cuda") + self.rank # the context manager ensures that `wait_tensor(y)` will wait on the correct work object with allow_inflight_collective_as_graph_input_ctx(): y = all_reduce_eager(x) z = all_reduce_wait_compiled(y) ``` With this context manager, when a collective is called, under the hood the work object of the collective will be registered in the work registry, and the wait_tensor() in compiled region called on the output tensor of the collective will wait on the correct work object. N)rr]_distributed_c10d)_allow_inflight_collective_as_graph_input-_set_allow_inflight_collective_as_graph_input)rWpreviouss r,allow_inflight_collective_as_graph_input_ctxr]0sv8xx))SSUH  ""PPQVW  ""PP  ""PP  s)B1-B*B1+B..B1c[UR55n[U5S:XaURU5 O US==U-ss'UR U5nU$Nr)rNrFrTappend new_empty)rr5out_size out_tensors r_make_all_gather_out_tensorrdWsLEJJL!H 8} # z! *J rcDUVs/sHn[XC5PM sn$s snfrnrd)rrrr5rs r&_all_gather_into_tensor_coalesced_metargas @D E1 ' 6 EE Esc.[R"U5$rnrHrrs r_broadcast_metarjf   D !!rc.[R"U5$rnrHris r_all_reduce_metarmjrkrc.[R"U5$rnrHris r_wait_tensor_metaronrkrc[X5$rnrf)shardrrr5s r_all_gather_into_tensor_metarrr &u 99rcp[UR55nUS==U-ss'URU5$r_rNrFra)rrrrr5rbs r_reduce_scatter_tensor_metarvvs/EJJL!H QKJK ??8 $$rcZUVs/sHn[R"U5PM sn$s snfrnrH)rrrs r_all_reduce_coalesced_metarx|s%)- .AE  Q  .. .s (cU$rnrinprs r_all_reduce__metar| JrcU$rnrrzs r_broadcast__metarr}rcU$rnr)rRrs r_all_reduce_coalesced__metars MrcJ^U4SjnUVs/sH oe"U5PM sn$s snf)Ncv>[UR55nUS==T-ss'URU5nU$r_ru)rrbrcr5s r mk_out_tensor<_reduce_scatter_tensor_coalesced_meta..mk_out_tensors5 % " __X. rr)rRr%rrr5rrs ` r%_reduce_scatter_tensor_coalesced_metars' '- -fM! f -- -s cUcURUR55$UHn[R"US:5 M [ UR55n[ U5US'URU5$r_)rarFr_checkrNr)rrjrkrrsrbs r_all_to_all_single_metarsg!uzz|,,#A LLa $ %,- x((rc[X5$rnrf)rr5r$rs r'_all_gather_into_tensor_out_native_metarrsrc[X5$rnrf)rr5r$s r#_all_gather_into_tensor_native_metarrsrcFUVs/sHn[X1U5PM sn$s snfrn)r)rRr5r$rs r-_all_gather_into_tensor_coalesced_native_metars0 E ,EzJ  cp[UR55nUS==U-ss'URU5$r_ru)r{rr5r$r~s r"_reduce_scatter_tensor_native_metars/  E !HH == rcp[UR55nUS==U-ss'URU5$r_ru)r{rr5r$rr~s r&_reduce_scatter_tensor_out_native_metars1  E !HH == rc FUVs/sHn[XAX#5PM sn$s snfrn)r)rRrr5r$r{s r,_reduce_scatter_tensor_coalesced_native_metars0  C +3:R  rrIMPLr'Meta all_reduce_rMall_reduce_coalesced_rall_gather_into_tensor_outr0rQrGreduce_scatter_tensor_outrWrr" broadcast_c10d_functionalDEF) zObroadcast(Tensor self, int src, str tag, int[] ranks, int group_size) -> TensorzUall_reduce(Tensor self, str reduceOp, str tag, int[] ranks, int group_size) -> Tensorzcall_reduce_coalesced(Tensor[] self, str reduceOp, str tag, int[] ranks, int group_size) -> Tensor[]z"wait_tensor(Tensor self) -> TensorzTall_gather_into_tensor(Tensor shard, str tag, int[] ranks, int group_size) -> Tensorzball_gather_into_tensor_coalesced(Tensor[] input, str tag, int[] ranks, int group_size) -> Tensor[]zareduce_scatter_tensor(Tensor input, str reduceOp, str tag, int[] ranks, int group_size) -> Tensorzpreduce_scatter_tensor_coalesced(Tensor[] inputs, str reduceOp, str tag, int[] ranks, int group_size) -> Tensor[]zall_to_all_single(Tensor input, SymInt[]? output_split_sizes, SymInt[]? input_split_sizes, str tag, int[] ranks, int group_size) -> Tensor(rJ)tagsra output_tensor input_tensorasync_opcU(a [S5eU=(d [RRnUc [S5eUR [ XX$55$N@Can't remap async version of inplace op to functional collectivegroup cannot be None)r-rrWORLDcopy_r7)rrrrrr)s rall_gather_tensor_inplacersW N    %TZZ%%E }344   05V WWrrropc U(a [S5eU=(d [RRnUc [S5eUR [ XXSU55$r)r-rrrrrG)rrrrrrArs rreduce_scatter_tensor_inplacer&sW N    %TZZ%%E }344 <<-eSQ RRravgproductminmaxbandborbxorrcU(a [S5eU=(d [RRnUc [S5eUR [ XX$55$r)r-rrrrr')rrrrrs rall_reduce_inplacerGsT N    %TZZ%%E }344 << 6u: ;;rc U(a [S5eU=(d [RRnUc [S5eUR [ UUUUU55$r)r-rrrrr)rrrjrkrrrs rall_to_all_inplacerZsf N    %TZZ%%E }344 <<        rrIc$^U(a [S5eTR5S:wa%[U4SjU55(d [S5eU=(d [RR nUc [S5e[ TSX$5n/nSnUHVnUR5S:Hn U (aSOURS5n U (aXWOXWXz-n URU 5 Xz- nMX [X5HupU RU 5 M U$)Nrrc3h># UH'oRS5TRS5:Hv M) g7f)rN)rF)rsrrs rrt%all_gather_inplace..s$$V+QVVAY&++a.%@+s/2z7Remapping variable size all_gather is not yet supportedrr ) r-r:r}rrrr7rFr`rVr)rIrrrrr output_splitsoffsetr is_scalart_offsetrrrs ` rall_gather_inplacerws N  zz|q$V+$V!V!VVWW  %TZZ%%E }344 vq% 5FM F EEGqL !1qvvay )fnvv?P/QS! 3 #4 r)_all_gather_base_reduce_scatter_base all_gatherr0r'rrG))T)NFrr)rNFrr)rNFr)NNNFr)NFr) contextlibsysr8typingrrrrrtorch.distributed distributedr"torch.distributed.distributed_c10ddistributed_c10dr. torch._utilsrtorch.distributed.device_meshr"torch.fx.experimental.proxy_tensorr rr fun_col_impltorch.utils._cxx_pytreer ImportErrortorch.utils._pytreetorch.compilerr r Exceptionr9rNrpr*r/r6 RANK_TYPESrrrr"r'r7r@rGrKrMrQrWrirrrrlibraryregister_autogradrrrrrrrrrrrrrrrr2rr!autogradFunctionr< custom_oprF register_fakerJrLrNrrUr#contextmanagerr]rdrgrjrmrorrrvrxr|rrrrrrrrrrLibrarylib_implimplfxnodehas_side_effectrrr legacy_liblegacy_lib_implops_defsmodulesr  my_moduleop_defindexop_namegetattr backend_impldefineTagpt2_compliant_tagrrReduceOpSUMAVGPRODUCTMINMAXBANDBORBXORREDUCE_OP_TO_STRrrrrlegacy_all_gather_baserlegacy_reduce_scatter_baserlegacy_all_gatherr0legacy_allgatherlegacy_allreducelegacy_all_to_all_singlelegacy_reducescattertraceable_collective_remapsrrrrs  22 11.4=:25 N #J IcO "C '(NN  :: &ELL &s &: &C &&U\\&S&&#&4 & ,,&& & &  \\ &Z  ,,  L ( ,,(((  ( (` % ,,%%%  % %RLN6 u|| 6(+64>6EH6 %,,6:=?6 u|| 6%/6696 %,,6D -6  -6-6c-6  -6 -6  %,, -6d V* 2& ,,2&S D(2&Cy4'2&  2& 2&  \\ 2&t '* ,,'*S D('*Cy4''*  '* '*  \\ '*^ 5<<   #+ +%,,+8 & "* +ell+8   .#6 1U\\1D & -"5 118 . )1 =T%,,5G=> & ,!4 =ellAS=8   8-@ CU\\@RCD & 7,?  V ,,V #YV V V  \\ V>q#ELLq#h M&M&#M&uS$s)S=P7QM&`+Lz+L+LT^^+L\u~~..$- % ( (%,, (  ( $$#U\\#ell#%# U\\   ''"5( ( (' '  # # # LF """:% /. )::   == !3V < l,f5 m.7 $&@&I %'BFK m.7  "I6 &(KVT &1   %'I6R !G %0   !#:FC k?F3 l,f5 eii88DDLLM eii88DDE]] " "#4e < --''(96B  KK ! FQc*+G<1WI7Lf599#>#>?,0KL   X<<X,,X X X  X. S LLS <<S S  S  S S, MMuMMuMM9MMuMMuMMMMuMM   < LL< < < <,   LL << @   "ell#" LL" "J/7(0) =5m42112  MMn  s$d+ d>+ d;:d;>ee