΅i= rSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJrJ r SSK J r SSKJ r SSK Jr SSKJrJr SSKJr SSKJrJrJrJrJr SS KJr SSKrSSKJr SSKJ r SSK!J s J"r# SS K$J%r% SS K&J'r' SS K(J)r)J*r*J+r, SS K-J.r. SSK/J0r0J1r1 SSK2J3r3 SSK4J5r5J6r6J7r7 SSK8J9r9 SSK:J;r;JJ?r?J@r@JArA SSKBJCrCJDrDJErEJFrF SSKJGrGJHrH SSKIJJrK SSKLJMrMJNrNJOrOJPrP SSKQJRrRJSrSJTrTJUrUJVrVJWrW SSKXJYrY SrZ\U(aSr[Sr\\RR5rZO8\V(aSr[Sr\O,\W(aSr[S r\\RR5rZOS!r[S"r\S#rZ"S$S%\5r`"S&S'\5ra"S(S)\ R\5rcS*\ RS+\RS,\ 4S-jreS.rfSzS*\ RS/\g4S0jjrhS{S1jriS2rjS3rkS|S*\ RS4\g4S5jjrlS*\ RS6\g4S7jrmS*\ RS8\g4S9jrn"S:S;5ro"S<S=\c5rp"S>S?\c5rq"S@SA\q5rr"SBSC\q5rs"SDSE\c5rt"SFSG\t5ru"SHSI\ R5rv"SJSK\q5rw"SLSM\ R5rx"SNSO\ R5rz"SPSQ\ R5r{\RSR\ 4SSj5r}\RST\ 4SUj5r~\RSV\ 4SWj5r\RSX\ 4SYj5r\RSZ\ 4S[j5r\\RS\\ 4S]j55r\\RS^\ 4S_j55r\\RS`\ 4Saj55r\\RSb\ 4Scj55rSd\ S,\ Se\Sf\4SgjrS}Sh\ RSi\ RSj\\Sk44Sljjr\GR"\WSm5"SnSo\N55r"SpSq\M5rS~Sr\\4Ssjjr"StSu\ R5r"SvSw\ R5r"SxSy\ R5rg)N)ABCabstractmethod)Callable) nullcontext)deepcopy)autoEnumwraps)Anycast no_type_checkOptionalUnion)mock) checkpoint) DeviceMesh) CPUOffload fully_shardFullyShardedDataParallel) TrainingState)FSDPParamGroupRegisterPostBackwardFunction)#NO_RESHARD_AFTER_FORWARD_STRATEGIES)BackwardPrefetchMixedPrecisionShardingStrategy)ShardedGradScaler)always_wrap_policyModuleWrapPolicywrap)distribute_tensorDTensorShard)ColwiseParallelparallelize_moduleRowwiseParallelSequenceParallel)TransformerDecoderLayerTransformerEncoderLayer)DistributedDataParallel)MultiProcessTestCaseMultiThreadedTestCase run_subtests TEST_SKIPS) FILE_SCHEMAget_cycles_per_ms set_rng_seed TEST_CUDATEST_HPUTEST_XPU) has_tritoncudancclzhpu:0hcclxpuxcclcpuglooc0\rSrSr\"5r\"5rSrg) FSDPInitModeYN)__name__ __module__ __qualname____firstlineno__rNO_FSDP RECURSIVE__static_attributes__rC]/home/james-whalen/.local/lib/python3.13/site-packages/torch/testing/_internal/common_fsdp.pyrArAYsfGIrKrAc>\rSrSr\"5r\"5r\"5rSrg)DEVICEInitModebrCN) rDrErFrGr DEVICE_BEFORE DEVICE_AFTER DEVICE_NEVERrJrCrKrLrNrNbsFM6L6LrKrNc\rSrSrSr\S\\RS44Sj5r \S\R4Sj5r \S Sj5r \ \S \ S \ S\R4S j55rS rg) FSDPTestModelkzVThis defines the interface expected from all models used commonly for FSDP unit tests.return.cg)z+Returns an input for the model as as tuple.NrCselfdevices rL get_inputFSDPTestModel.get_inputo rKcg)z,Returns the loss given the input and output.NrC)rYinputoutputs rLget_lossFSDPTestModel.get_losstr]rKNcg)zU(aUR[5$U$r)rr)rurs rL_move_to_devicers$2588K ==rK wrap_fsdpc2U(dU$[U/UQ70UD6$rr)rurrhris rL_maybe_wrap_fsdprs!5CtE'CD'CF'CCrKcH\rSrSrS\S\4SjrS\4SjrS\4SjrSrS r g ) DummyProcessGrouprsizecXlX lgr_rank_size)rYrrs rL__init__DummyProcessGroup.__init__s   rKrVcUR$r)rrYs rLrDummyProcessGroup.rank zzrKcUR$r)rrs rLrDummyProcessGroup.sizerrKcB[R"5nSnXClU$)Ncd[RR5nURS5 U$)Nr?)rpfuturesFuture set_result)futures rL get_future/DummyProcessGroup.allreduce..get_futures'+0==+?+?+AF   a MrK)rMockr)rYrhri dist_waitrs rL allreduceDummyProcessGroup.allreduces IIK   *rKrN) rDrErFrGintrrrrrJrCrKrLrrs2Scc rKrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\\4S\S\S \\R*\44Sjj5rSrSrU=r$)TransformerWithSharedParamsirzdevice_init_modeadd_bn deterministicc>[TU]5 UR5UlUR5UlU(a[ R "S5 SnSn[R"XV5Ul [R"USSSSS9Ul [R"Xe5Ul URRURlURSURRR!U455 URS [ R""UR$[ R&S 95 SUlU(a)[ RR+UR(5O[ RR-5UlU[0R2:XaUR5[65nU(aUR95 gg) Nrg?)d_modelnum_encoder_layersnum_decoder_layersdim_feedforwarddropout vocab_bias long_buffer)dtype)superrrrrrp manual_seedrs Embedding embed_tokens Transformer transformerLinear output_projweightregister_buffernew_ones zeros_likerlongbs BatchNorm1dIdentitybnrNrPrreval)rYrzrrrd_vocabr __class__s rLr$TransformerWithSharedParams.__init__s] JJL **,    a LL:>>    99W6#'"3"3":":  $++22;;WJG      T__EJJ ? 39%((&&tww/uxx?P?P?R ~;; ;77;'D  IIK rKc$[R"SUR-5 [R"SUS9R SUR 5n[R"UR S-US9R SUR 5nX#4$)Nr? rZr7)rprrarangeviewr)rYrZsrctgts rLr[%TransformerWithSharedParams.get_input1sj !dii-(ll2f-221dgg>ll477Q;v6;;AtwwGzrKcURU5nX0R-URRU5-nURU5nUR U5nUR X45nUR U5$r)rrrtype_asrrr)rYsrc_idstgt_idsr r xs rLforward#TransformerWithSharedParams.forward7sr(OO#d&6&6&>&>s&CC(ggcl   S &""rKcUup4[RRURSUR S55URS5SS9$)Nsum) reduction)rs functional cross_entropyrr)rYr_r`rr s rLra$TransformerWithSharedParams.get_loss?sG}}** KKFKKO ,chhrle+  rKc$UR5 grbackwardrds rLrf(TransformerWithSharedParams.run_backwardE  rKfsdp_init_mode fsdp_kwargsrVcxUc0nU[R:Xa)[U[5(aUSnOUn[ XbXT5$U[R :XaSU;a[ [[15nOURS5nSU;a?US[R[R1;a[U[5(dSnOUn[U[5(aUSn OUn [ XXT5n [U U4SU0UD6n U[R:XaU R!["5n U $[%SU35e)a Initializes a :class:`TransformerWithSharedParams` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps with top-level FSDP. By default, the top-level FSDP uses the ``ModuleWrapPolicy`` for encoder and decoder layers, but a different auto wrap policy may be specified via ``fsdp_kwargs``. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. add_bn (bool): Whether to include batch norm in the model. Nrauto_wrap_policysharding_strategyUnsupported FSDP init mode: )rArH isinstancerorrIr r*r)popr HYBRID_SHARD_HYBRID_SHARD_ZERO2rrNrQrr ValueError) rzrrr rrpgr"fsdp_pg tformer_pgm fsdp_models rLrk TransformerWithSharedParams.initHsW6  K \11 1%''1X.f |55 5!4#3//$ $/??3E#F ${2 34$113C3W3WXY"5%00%''"1X " +fA"2 J  >#>#>>']];7  77GHIIrKcUR/$r)rrs rLget_ignored_modules/TransformerWithSharedParams.get_ignored_moduless  !!rK)rrrrrrr)NFT)rDrErFrGr ProcessGrouprNboolrr[rrarfrrrArrrr rrsrtrrkr1rJ __classcell__rs@rLrrs(  ()( (  (T #  15# KJ  KJ$KJ)KJd38n- KJ  KJ  KJ ryy$ KJKJZ""rKrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\\4S\S \R(4 Sjj5rSrU=r$)NestedWrappedModuleirzrrrc >^^^[TU]5 TR5UlTR5UlU[ R :HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[[R"SS5U55[[R"SS5U55Ul g)Nc0>T(a [UT40TD6$U$rrlayerr rzrs rL _maybe_wrap1NestedWrappedModule.__init__.._maybe_wrapE58K88LrKrrr7r)rrrrrrNrPrprrs Sequentialrrmodule rYrzrrrr rr=rs `` ` rLrNestedWrappedModule.__init__s JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   "a(8.I J BIIaO^ <   rKcv[R"SUR-5 [R"SSUS94$)Nr?r7rr)rprrrandrXs rLr[NestedWrappedModule.get_inputs. !dii-( 1a/11rKc$URU5$rrArYrs rLrNestedWrappedModule.forward{{1~rKc&UR5nU$r)rrYr_r`res rLraNestedWrappedModule.get_lossszz| rKc$UR5 grrrds rLrf NestedWrappedModule.run_backwardrrKrr rVcUc0nU[R:Xa [USUUS9$U[R:Xa;[U4SUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`NestedWrappedModule` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. FrrrTr$) rArHr8rIrNrQrrr))rzrrr rr.s rLrkNestedWrappedModule.inits.  K \11 1&!1+   |55 5,!1+   J >#>#>>']];7  77GHIIrKrArrNF)rDrErFrGrr3r4rNrr[rrarfrrrArrrr rsrtrkrJr5r6s@rLr8r8s     )    @2 15# +J  +J$+J)+Jd38n- +J  +J  +J+JrKr8c v^\rSrSr\S S\R S\S\S\ \ \ \ 4S\ 4 U4Sjjj5rSrU=r$) AlwaysWrapNestedWrappedModuleirzrrr rc:>[[[] U[RUUUS9nU[R:XaU$U[R :XaHU=(d 0n[ U4S[0UD6nU[R:XaUR[5nU$g)z Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, for the ``RECURSIVE`` init mode, this wraps with top-level FSDP and the ``always_wrap_policy()`` auto wrap policy. )rzrrr rr"N) rrWrkrArHrIrrrNrQrr)rzrrr rrur.rs rLrk"AlwaysWrapNestedWrappedModule.inits )+H '//-#'    \11 1L |55 5%+KeX6HXKXJ>#>#>>']];7   6rKrCrU)rDrErFrGrrrr3rArNrrrr r4rkrJr5r6s@rLrWrWs^ 15#   $)d38n-   rKrWc ^\rSrSrS\R S\S\S\4U4Sjjr\ S Sj5r \ S S\R S\ S\S \ \ \\4S\4 S jj5rS rU=r$)NonUniformReqGradNWMirzrrrc >^^^[[U] 5 TR5UlTR 5UlU[ R:HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[R"[[R"SS5U5[[R"SS5U5555Ulg)Nc0>T(a [UT40TD6$U$rrr;s rLr=2NonUniformReqGradNWM.__init__.._maybe_wrap.r?rKrrr7r)rr8rrrrrNrPrprrsr@rrrArBs `` ` rLrNonUniformReqGradNWM.__init__s !413JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   #BIIb!$4nE#BIIaO^D   rKcUR5H3up#[R"X5(aM"URS5 M5 grU)r}rematchrequires_grad_)ru req_grad_masknps rL_set_nonuniform_req_grad-NonUniformReqGradNWM._set_nonuniform_req_gradEs4**,DA88M--  '-rKrr c[R"S5nU[R:Xa#[ USUUS9n[R Xe5 U$U[R :XaUUc0n[ U4SUUS.UD6nU[R:XaUR[5n[R Xu5 U$[SU35e)a Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, it wraps a second :class:`torch.nn.Sequential` container to enable the desired non-uniform ``requires_grad`` ``use_orig_params=True`` tests. For both ``RECURSIVE`` and ``NO_FSDP`` init modes, freezes all parameters except the last two to validate ``ShardedGradScaler`` support for ranks with no (non-zero sized) local shards in FSDP ``use_orig_params=True`` mode. zmodule\.2.*\.1.*FrRTr$) racompilerArHr[rgrIrNrQrrr))rzrrr rreq_grad_pattern ddp_modelr.s rLrkNonUniformReqGradNWM.initKs(::&9: \11 1,!1+ I ! 9 9) V  |55 5" -!1+   J >#>#>>']];7 9 9* W 77GHIIrKrTrmrU)rDrErFrGrr3r4rNrrrrgrArrrr rkrJr5r6s@rLr[r[s(   ( ( ) (  ( T((  15# +J  +J$+J)+Jd38n- +J  +J+JrKr[c ^\rSrSrSrS\R S\S\4U4SjjrSr Sr S r S r \ S \\S \S\S\S \4 Sj5rSrU=r$)ModuleWithDelayizzThis class wraps a :class:`FSDPTestModel` to optionally add a delay after computing the loss and/or before the gradient reduction.rAdelay_after_loss_msdelay_before_reduction_mscF>[TU]5 X lX0lXlgr)rrrprqrA)rYrArprqrs rLrModuleWithDelay.__init__~s! #6 )B& rKc8URRU5$r)rAr[rXs rLr[ModuleWithDelay.get_inputs{{$$V,,rKc$URU5$rrHrIs rLrModuleWithDelay.forwardrKrKcbURRX5nURS:a[(d [(a%[ R "URS- 5 U$[(a=[RR[UR[5-55 U$Nr) rArarpr4r5timesleepr3rpr8_sleeprr1rMs rLraModuleWithDelay.get_losss}{{##E2  # #a 'x88 433d:;  !!#d&>&>ARAT&T"UV rKc^^[RRmUU4Sjn[R"SU5 TR R U5 SSS5 g!,(df  g=f)Nc8>TRS:a[(a>[RR [ TR[ 5-55 O9[(d [(a#[R"TRS- 5 T"U0UD6$ry) rqr3rpr8r}rr1r4r5r{r|)rhriorig_reduce_scatterrYs rL_delayed_reduce_scatter=ModuleWithDelay.run_backward.._delayed_reduce_scattersq--19JJ%%D::=N=PPQXJJt==DE&77 7rKz'torch.distributed.reduce_scatter_tensor)rp distributedreduce_scatter_tensorrpatchrArf)rYrerrs` @rLrfModuleWithDelay.run_backwardsR#//EE 8ZZ 57N  KK $ $T *   s A  A. module_class model_args model_kwargsc<[UR"U0UD6UU5$)a Args: module_class (Type[FSDPTestModel]): Wrapped module class to which to add delays. model_args: Positional arguments forwarded to the ``module_class`` ``init()``. delay_after_loss_ms (int): Delay after computing the loss/before the optimizer step (in ms). delay_before_reduction_ms (int): Delay before reduce-scattering gradients (in ms). model_kwargs: Keyword arguments forwarded to the ``module_class`` ``init()``. )rork)rrprqrrs rLrkModuleWithDelay.inits**   z :\ :  %  rK)rprqrA)rDrErFrGrnrsrtrrr[rrarfrrtyperTr rkrJr5r6s@rLrorozsF   ! $' -+$ =)  ! $'     rKroc\rSrSr\\R SSSS4S\RS\ S\S\ \ \ \ 4S \S \S \4S jj5rS rg)NestedWrappedModuleWithDelayiNFrrzrrr rrprqc >[R[UUUUUUUS9$)Nrzrrr rrprq)rorkr8rs rLrk!NestedWrappedModuleWithDelay.inits4## )-#' 3&?$  rKrC)rDrErFrGrrrNrQrr3rArrrr r4rrkrJrCrKrLrrs,:+F+F04##$)*    $ ) d38n-    !  $'  rKrc.^\rSrSrU4SjrSrSrU=r$)DummyDDPic.>[TU]5 Xlgr)rrrA)rYrArs rLrDummyDDP.__init__s  rKc&UR"U0UD6$rrHrYrhris rLrDummyDDP.forwards{{D+F++rKrHrDrErFrGrrrJr5r6s@rLrrs,,rKrc^\rSrSrS\R S\S\S\S\4 U4Sjjr Sr S r \ SS\R S \ S\S \\\\4S\S\4 S jj5rS rU=r$)MixtureOfExpertsirzrrdelay_before_free_msrc >[TU]UUUUS9 XlX@lX lU[ R :HUlU(a#[R"SUR-5 SnSnSn [[R"Xx5UR5n [SU R555UlU R5H n SU lM U(a[R"S5 [[R"X5UR5n U(aF[R$R'UR5/5n [)X40UD6n [)X40UD6n [R*"[[R"X5UR5U U [[R"X5UR55Ulg) N)rzrrr*rrrc3@# UHoR5v M g7fr)numel).0rfs rL ,MixtureOfExperts.__init__..s$L8K1WWYY8KTr)rrrzrrrNrPrrprrrrsrrrnum_expert_paramsexpertr new_grouprr@rA)rYrzrrrrr d_expertd_sharedd_inputrrfshared expert_grouprs rLrMixtureOfExperts.__init__s{ -'   $8!"..2N2NN    b499n - 8!>@S@ST!$$L8I8I8K$L!L""$AAH%    a  8!>@S@ST  ,,66L&>+>F&7;7Fmm BIIg8$:M:M N   BIIh8$:M:M N   rKc^^TRS:aTRSn[U[5(ag[R R RRmUU4Sjn[R"SU5 TRU5sSSS5 $TRU5$!,(df  N=f)Nrrc>[(a>[RR[ TR [ 5-55 O9[(d [(a#[R"TR S- 5 T"U0UD6$)Nrz) r3rpr8r}rrr1r4r5r{r|)rhri orig_reshardrYs rL_delayed_reshard2MixtureOfExperts.forward.._delayed_reshard"sc y )) 9 9 {{1~ s B// B=cUR5 UR(d[R"5 UR 5H|n[ US5(aMUR cM%UR RUR5 [RRUR URS9 M~ SSS5 gg!,(df  g=f)Nrry) rrrprrhasattrgraddiv_rr all_reducerz)rYrerfs rLrfMixtureOfExperts.run_backward4s ~~*Aq(++ vv) DOO4))44QVV4::4N +! s2C-AC C rr c Uc0nU[R:Xa [USUUUS9$U[R:Xa<[U4SUUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`MixtureOfExperts` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP, including the expert and shared layers, but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. delay_before_free_ms (int): Delay before resharding expert parameters in the forward pass (in ms). F)rrrrTr$) rArHrrIrNrQrrr))rzrrr rrr.s rLrkMixtureOfExperts.init@s4  K \11 1#!1%9+  |55 5)!1%9+   J >#>#>>']];7  77GHIIrK)rrzrArrr)NFr)rDrErFrGrr3r4rNrrrrfrrrArrrr rkrJr5r6s@rLrrs2   2 2 ) 2 " 2  2 h0 O 15#$% 0J  0J$0J)0Jd38n- 0J  0J " 0J0JrKrc ^\rSrSrSSSSS.S\S\\RS\S \S \4 U4S jjjjr S \RS \R4Sjr Sr Sr U=r$)MLPitTFr7)bias with_bufferdim_multiplierdimrZrrrc>[TU]5 [R"XU-X#S9Ul[R"XQ-XUS9UlU(a'UR S[R"U4US95 gSUl g)N)rZrrr) rrrsrin_projout_projrrprandnr)rYrrZrrrrs rLr MLP.__init__usf yys&:6U  ."6QUV   5;;vf+M NDKrKrrVcURU5n[R"U5nURU5n[R"U5nURbX R-nU$r)rFrelurr)rYrzs rLr MLP.forwardsQ LLO FF1I MM!  FF1I ;; "KKArKcURb4[RRR UR5 ggr)rrprsrknormal_rs rLreset_parametersMLP.reset_parameterss+ ;; " HHMM ! !$++ . #rK)rrrr)rDrErFrGrrrprZr4rrqrrrJr5r6s@rLrrts*. ! &    "%,,//rKrcV^\rSrSrSS.S\S\4U4SjjjrS\S\S \S S4S jrSr U=r $) MLPStackiF)with_seq_parallelmlp_dimrc>[USS9[U5[USS9/nU(a$UR[R"USS95 [TU]"U6 X lg)N)rFr)rappendrs LayerNormrrr)rYrrmodulesrs rLrMLPStack.__init__sV  * L  * $   NN2<<e< = '"!2rKtp_meshdp_meshuse_activation_checkpointingrVc [SS9[SS9[SS9[SS9[SS9UR(a[[S5S9O [5S.nUR(a [ SS9US'[ XUS9 UHDn[ U[R5(aM$U(a [U5 [U4S U0UD6 MF [U4S U0UD6 U$) NF)use_local_outputr?)output_layouts)z 0.in_projz 0.out_projz 1.in_projz 1.out_projz 2.in_projz 2.out_proj) sequence_dim3) device_meshparallelize_planmesh) r%r'rr$r(r&r%rsrrr)rYrrrr rrAs rL parallelizeMLPStack.parallelizes)%@)5A(%@)5A(%@%%*qB "    ! !$4!$D S !4GWXF&",,//+6"  [TU]5 [R"X5Ul[R"X5Ul[R "5UlX lgr) rrrsrlin1lin2ReLUrr)rYrrrs rLrDoubleLinear.__init__sA IIc' IIc' GGI !2rKrrVcUR(a@URURU55URURU554$URURU55$r)rrrrrIs rLrDoubleLinear.forwardsQ  ! !99TYYq\*DIIdiil,CC Cyy1&&rK)rrrrT)rDrErFrGrnrr4rrprqrrorrJr5r6s@rLrrs^ 3C3D33'' uU\\5<</0%,,> ?''rKrnew_all_gather_into_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rall_gather_into_tensorbarrier)rorig_all_gathers rLpatch_all_gatherrsN11OLLN"A!A>"A;;A>new_foreach_all_gatherc#@# [RRRRR n[ R"5 U[RRRRlSv [ R"5 U[RRRRlg![ R"5 U[RRRRlf=f7fr)rprr _fully_shard_fsdp_param_groupforeach_all_gatherrr)rorig_foreach_all_gathers rLpatch_foreach_all_gatherrs ++==PP LLN ''99L    # ++==P  # ++==P BDCA DA DDnew_foreach_reducec#@# [RRRRR n[ R"5 U[RRRRlSv [ R"5 U[RRRRlg![ R"5 U[RRRRlf=f7fr)rprrrrforeach_reducerr)rorig_foreach_foreach_reduces rLpatch_foreach_reducer s ++==LL  LLN ''99H    ' ++==L  ' ++==Lrnew_reduce_scatter_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrr)r rs rLpatch_reduce_scatterr  sO44LLN!:D9  %8" %8"rnew_all_reducec## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrr)rorig_all_reduces rLpatch_all_reducersIooOLLN$DO*  ) )r new_unshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)runshardrr)r orig_unshards rL patch_unshardr$P"))LLLN(N.  !- !-r new_reshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rreshardrr)rrs rL patch_reshardr1rrnew_post_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r post_backwardrr)rorig_post_backwards rLpatch_post_backwardr >sQ(55LLN#4N :  '9$ '9$r new_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrrr)r! orig_backwards rL*patch_register_post_backward_hook_backwardr$KsS199MLLN,8 )>  0=$- 0=$-rrrhric[U5S:aUSnOSU;aUSnO[SUSU35eU"U5 U"U0UD6$)Nrr`z,Cannot get reduce-scatter output from args: z kwargs: )lenAssertionError)clsrrwrhrir`s rLreduce_scatter_with_assertr)Xsa 4y1}a V !;D6F8 T  f  / //rKreplicated_modulesharded_moduleprefixes_to_ignore.c[UR5UR5SS9GHuupEupgUnUHn URU S5nM URXH5 UR U[ 5 [ U[ 5(deURURp[U 5[S5[S54:Xa [S5e[XZU 5n URUR5U R55 URcURUR5 GMUR!UR5 [URX5n UR UR[ 5 [ UR[ 5(deURURR5U R55 GM g)NTr{rzmFSDP's (Shard(0), Shard(0)) layout differs from distribute_tensor(), so we cannot check for equality using it)rr}replace assertEqualassertIsInstancer#r%r placementsror$r'r"to_localr assertIsNoneassertIsNotNone)r(r*r+r,replicated_namereplicated_param sharded_name sharded_paramclean_sharded_nameprefixrr2sharded_ref_paramsharded_ref_grads rLcheck_sharded_parityr>ks OR**,'')OJ+-Jl *(F!3!;!;FB!G ) < ]G4-1111(44m6N6Nj  q58 4 4 ; ..>jQ  ..02C2L2L2NO  (   ]// 0  M../,-=-B-BDU ]//9-,,g6666  **3357G7P7P7RS5OrKznot-support-multithreadcJ^\rSrSr\S5rU4SjrSrSrSr Sr U=r $)FSDPTestMultiThreadic[$r DEVICE_COUNTrs rLrFSDPTestMultiThread.world_sizerKcB>[TU]5 UR5 gr)rsetUp_spawn_threadsrYrs rLrGFSDPTestMultiThread.setUps   rKc [U/UQ70UD6$rr.rs rLr. FSDPTestMultiThread.run_subtestsD242622rKc@[RR5 grrp_dynamoresetrs rLperThreadSetUp"FSDPTestMultiThread.perThreadSetUp rKc@[RR5 grrPrs rLperThreadTearDown%FSDPTestMultiThread.perThreadTearDownrUrKrC) rDrErFrGpropertyrrGr.rSrWrJr5r6s@rLr@r@s. 3rKr@c$^\rSrSrU4Sjr\S5r\S5r\S\4Sj5r \S5r Sr S r S r S r\S 5rS)S\R$S\S\S\S\\S\S\\S\S\S\\\\44SjjrS SS\"5S S S SSSSS S 4 S\\S\S\S\\ S \S\S!\S"\\!S#\\"S\\S$\S%\S\S\S&\\\\4S\\\\44 S'jjr#S(r$U=r%$)*FSDPTestich>[TU]5 S[RS'UR 5 g)N0TORCH_NCCL_DESYNC_DEBUG)rrGosenviron_spawn_processesrIs rLrGFSDPTest.setUps)  14 ,- rKc[$rrBrs rLrFSDPTest.world_sizerErKc>[RR5$r)rdistributed_c10d_get_default_grouprs rLrvFSDPTest.process_groups$$7799rKrVcgrUrCrs rLdestroy_pg_upon_exitFSDPTest.destroy_pg_upon_exitsrKc*[UR3$r)r0 file_namers rL init_methodFSDPTest.init_methodst~~.//rKc:URX!R5 gr)r0r)rYr.rs rL_check_cpu_offloadFSDPTest._check_cpu_offloads &<&<=rKc:URX!R5 gr)r0backward_prefetch)rYr.rts rL_check_backward_prefetch!FSDPTest._check_backward_prefetchs *,H,HIrKc:URX!R5 gr)r0forward_prefetch)rYr.rxs rL_check_forward_prefetch FSDPTest._check_forward_prefetchs )+F+FGrKc [U/UQ70UD6$rrLrs rLr.FSDPTest.run_subtestsrNrKc U"U5nXlX6lURSS5n[SURSUR35 [ R R5UR:a4[R"[SUR3R5 U(a^[ RRRRR!5n["R$"SURUUS9 OC["R$"UR&[([+UR5URS9 Sn UR[0-n [2(d [4(a[ R R7U 5 U /n ["R8"U S 9 [ R:R=5 [?5 URAX$5 [ R:R=5 ["R8"U S 9 ["RB"5 g![,a@n S U R.S ;a'[R"[S R5 eSn A ff=f) Nfake_pgFz dist init r=z, world=z multi-gpu-fake)backendrrstore)rnrrr recompilerbackend_unavailable) device_ids)"rrmgetprintrrp accelerator device_countsysexitr/ exit_codetesting _internalrr~ FakeStorerinit_process_grouprnDISTRIBUTED_BACKENDr RuntimeErrorrhrCr3r5set_device_indexrrQrRr2run_testdestroy_process_group) r(r test_namermpiperirYr~rer device_ids rL_run FSDPTest._runs9~ "**Y. TYYKx/@AB    ) ) +doo = HHZ*T__,= >?II J  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$1 affQi'$9:DDE   s -A$H7AH77 J;I<<JNFru num_stepsautocastlrfsdp_cpu_offload save_modelmixed_precisionenable_sharded_grad_scaler use_pure_fp16sharded_grad_scaler_kwargsc U=(a URn [UR55Rn U c0n [ SSU0U D6n [ R RUR5USS9n[U5GHgnUR5 [ RR[US9 URR[ R"[55nU (dU(aW[U[ 5(dB[U[ R"5(aUR%5nO['SU55nU"U6nU (ap[U[ 5(a[UR([*;aGUR5H3nUR-UR[ R"S55 M5 URR/UU5R1U 5nSSS5 U R3W5nU(d-U (d&UR4[ R6:XdS5eOU (a+UR-UR4[ R85 Ok[U[ 5(a,UceUR-UR4UR:5 O*UR-UR4[ R65 URR=U5 U (a\[U[ 5(aGUR5H3nUR-UR[ R"S55 M5 U R?U5 U RA5 U(dGM URC5RE5VVs0sHunnUURG5_M nnn[IU5 URKU5 GMj [U[ 5(aURM[NRP5 WRS5$!,(df  GNA=fs snnf) Nenabledg?)rmomentum)rc3@# UHoR5v M g7fr)r)rrs rLr4FSDPTest._train_for_several_steps.. s%>1ffhhrr=zeloss data type should be float32, as the original parameter data type is float32.rC)*offload_paramsnextrrZrrpoptimSGDr zero_gradamprrrAr[r%rrqrror#rr0rarscalerfloat32float16 param_dtyperfstepupdaterrclonerload_state_dict _assert_staterIDLEr~)rYrurrrrrrrrrcpu_offload_params model_devicesharded_grad_scalerrrr_r`rfrekvrs rL_train_for_several_steps!FSDPTest._train_for_several_stepss/.Q2B2Q2QE,,./66 % -)+ &/ . 2L    0 0 2rCHy!A OO ##K#B ..u||K/HI _Zt=T=T!%66 %  %%>%> >'"5$////>?#--/((5<<3FG0||,,UF;>>|L-C.',,T2D"=zzU]]252 !$$TZZ?t,,*666$$TZZ1L1LM$$TZZ? LL % %d +!j&=&=))+A$$QXXu||E/BC, $ $U +  & & (z7<7G7G7I7O7O7QR7Qtq!al7Q RE"%%j1s"v eT " "    2 2 3{{}wCBfSs9EP3-Q3 Q rT model_classrr ref_init_fn num_itersrrtr#rxuse_orig_params init_kwargsc U[R:wdS5eUc0nSnURR5nUR"UR[R[ R 4SS0UD6nUc-[(a[U[/[S9nO[UU/US9nOU"U5nU(aUR5nURUUU SLUUU U UUS9 n[UR55nURUUU U U U S.5 UR"URUUU4SS0UD6n[%U[&5(d['UUR40UD6nU(aUR5nU[ R(:XaUR+[5nUSL=(a UR,nU=(a U[ R(:HnU=(a U[ R(:gnU(aI[.R0"S 5nUR5HnUR3UR0U5 M! U(aUR5[6S [35O [95nU URUUS UUUU U UUS9 n SSS5 U(agU(a^[.R0"S 5nUR5HnUR3UR0U5 M! W R+[5n [;U5n![.R<R?UW S S9 U cU(dUR3UU!SSS9 ggg![a n[!S US [#U535UeSnAff=f!,(df  N=f)a Tests FSDP training against a reference, which defaults to DDP but may be customized with ``ref_init_fn``. Args: model_class (Type[FSDPTestModel]): A model class that inherits from ``FSDPTestModel``, which defines the expected interface. fsdp_init_mode (FSDPInitMode): The mode to initialize the FSDP-wrapped model. This should not be ``NO_FSDP``. ref_init_fn (Optional[Callable]): A callable to invoke that wraps a non-wrapped model to construct the reference model, where this wrapper should provide data parallel semantics. If ``None``, then the callable defaults to the DDP constructor. z.Expects an FSDP init mode that wraps with FSDPN{Gz?rT)r output_device)rrrrrrr)rrtr#rrxrz Initializing z raised error r=zOAn FSDP-managed module with parameter CPU offloading enabled has parameters on F)rrrrrrrr) check_dtypezFSDP did not match DDP) exact_devicemsg) rArHrvrrkrNrPr4DDPrrrrrr Exceptionr)rr%rrQrrrprZr0assertRaisesRegexrrrr assert_close)"rYrrrrrrrrtr#rrxrrrrrr rrru ref_modelref_loss ddp_paramsr.rrexpects_device_errorexpects_cpu_device cpu_devicercontext fsdp_lossfsdp_unsharded_paramss" rL_test_fsdp_parityFSDPTest._test_fsdp_parityVsD!5!55 < 5  K !!&&(       ( (       x{m;  4&M #E*I !(I00  $D0(+'A''A1  )..01 *%6%6#2$4#2    Y$))""   #   J*d++j$*<*<L LJ #*J ~:: :#{3J$D0O[5O5O  N/>3N3NN   N/>3N3NN  e,J#..0  z:1$  " "%%0M3   55!,% /+E++E6 I  e,J#..0  z:1! [1I / ;  ""8YE"J  "=   %!,  ,9 "K Y}[MAxPQWX X YFWs$"M1M5 M2M--M25 N)rmr)rNFNFFN)&rDrErFrGrGrYrrvr4rjrnrqruryr. classmethodrrsrtrfloatrrrrrr rrrTrArNrrrrrJr5r6s@rLr[r[sI ::d00>JH34%4%v15 48+0#?CUyyUU U  U #:. UU".1U%)UU%-T#s(^$<Ux+/",,8<8<48!& %+0#04?C#g-(g%g) g h' g  gg g$$45g$$45g".1ggg%)gg d38n-!g"%-T#s(^$<#ggrKr[compile_compute_on_modulecD^^^U4Sjm"SS[5mUU4SjnU$)Nc>[RRR"U0UD6 Tb[ UST5(aUSR 5 gg)Nr)rprrrr%rj)rhrirs rL!fully_shard_with_compiled_compute=compiled_fsdp_test..fully_shard_with_compiled_computesS **D;F; $ , G.1 1  GOO 1 rKc0\rSrSr\"5r\"5rSrg)*compiled_fsdp_test..FullyShardModeirCN)rDrErFrGrEAGERCOMPILED_COMPUTErJrCrKrLFullyShardModers6rKrc6>^[T5UUU4Sj5nU$)Nc>[RRRnTGHnUTR:wa&[ 5(d[ R"SSS9 M:[RRRn[RRRn[RR5 UTR:XaUnO_UTR:XaAS[RRl S[RRl TnO[SU35eUT R UR"'T "U0UD6 [RR5 UT R UR"'U[RRl U[RRl GM g)Nz0Inductor on GPU needs Triton and recent GPU archr) stacklevelTr?z!Need to implement FullyShardMode=)rprrrrr6warningswarnrQconfigskip_fsdp_hooks _inductorcompile_threadsrrNotImplementedError __globals__rD) rhrioriginal_fully_shardmodeoriginal_skip_fsdp_hooksoriginal_compile_threadsfully_shard_patchrrfuncs rLwrapper6compiled_fsdp_test..decorator..wrapper sf(-(9(9(>(>(J(J &>/// MMJWX+0==+?+?+O+O(+0??+A+A+Q+Q(!!))+>///(<%^<<<;?EMM((8=>EOO**:(I%-;D6B CT  !5!>!>?d%f%!!))+BV  !5!>!>?7O $$49Q&&6='rKr )rrrrs` rL decorator%compiled_fsdp_test..decorator s# t R  RDrK)r )rrrrs` @@rLcompiled_fsdp_testrs"""$L rKc2^\rSrSrSU4SjjrSrSrU=r$) SkipModulei5cX>[TU]5 [R"SSSS9UlgN Fr)rrrsrlinrIs rLrSkipModule.__init__6s" 99R%0rKc$URU5$rrrIs rLrSkipModule.forward:sxx{rKrrmrr6s@rLrr5s1rKrc.^\rSrSrU4SjrSrSrU=r$) NestedLineari>c>[TU]5 U(a8[[R"SSSS9R [ 55Ulg[R"SSSS9R [ 5Ulgr)rrr!rsrrr nested_linear)rY fsdp_wraprs rLrNestedLinear.__init__?sV  !%biiBU&C&F&F{&S!TD !#2r!>!A!A+!ND rKc$URU5$rr rIs rLrNestedLinear.forwardFs!!!$$rKr rr6s@rLrr>sO%%rKrc.^\rSrSrU4SjrSrSrU=r$) SkipModeliJc>[TU]5 [R"SSSS9R [ 5Ul[5R [ 5Ul[[US9[ S9Ul g)NrFr)r )r) rrrsrrrlinearr linear_skipr!rr )rY double_nestrs rLrSkipModel.__init__KsW iiBU366{C %skipIfr@r[rrrrrrCrKrLr5s #$"<< 44 ?S IRRFF FH +  K ::**,L K  K 99))+LK L4T BIIs 4 99$$>% # 99##" " 2299 2t 2>299>d>DBIID$D.Q"-Q"h[J-[J|$7D]J.]J@J mJ Z ? .,ryy,JJ*JJZ/"))/@*r}}*Z'299'6 666  X  "  X  " 9H99 *X** .x.. .x.. :8:: >X>>0!00  0  0.+- Tyy TII Tc3h TF 45/6&[#[| 2(4.2j %299 %  rK