ȅi#XSSKrSSKJr SSKJrJr SSKrSSKJr SSK J r SSK J r SSK JrJ r Sr"S S \ 5rS \S \S \4Sjr"SS5r\ "\S9SS\R&S\\RR&S-S \R&4Sjj5rS\R&S \4Sjrg)N)Iterable)AnyNoReturn)_State)DistributedDataParallel) _get_registrycontractc ^\rSrSr%\R \S'SU4Sjjr\4S\ RS\ \ RS\ \ RS \ SS4 S jjrSS jrS\ RS\ \ RSS4S jrSS jrSSjrS\ RS\\S4S\\ \4S\4SjrS\ RS\\R0S\R0S\R04SjrSrU=r$)_ReplicateState _ddp_weakrefreturnNc>[TU]5 [R"5UlSUl[R"5UlURUl/UlSUl SUl 0Ul /Ul g)NF) super__init__nn ParameterListmodulehas_initialized _param_list _orig_module _param_names_no_sync _init_args _init_kwargs_comm_hook_args)self __class__s a/home/james-whalen/.local/lib/python3.13/site-packages/torch/distributed/_composable/replicate.pyr_ReplicateState.__init__sk !#!1!1!3 %*-/-=-=-?!KK')# 26,.*,rignored_modulesignored_paramsprefixcl[U5(agX;agU[:waUS3O[nURSS9HFupgXs;dM URR U5 UR R UU35 MH UR 5HupURU UUUU3S9 M g)N.F)recurse)r&)_is_fully_sharded_ROOT_MODULE_PREFIXnamed_parametersrappendrnamed_children_collect_params) rrr$r%r&recurse_prefixnpname child_modules r!r/_ReplicateState._collect_params"s V $ $   $ #&99vhaL?R ++E+:DA&  ''*!!((N+;A3)?@; #)"7"7"9 D  ()$0 ! #:r#cN^[R"SS9U4Sj5nU"5 g)NT) recursivec>TRceTR"TR0TRD6 TR5 STl0Tlg)N)rinitrregister_comm_hook)rsr! _lazy_init-_ReplicateState.lazy_init.._lazy_initCsI??. .. IIt <$*;*; <  # # % DO "D r#)torch_disable_dynamo)rr<s` r! lazy_init_ReplicateState.lazy_initBs'    . # / #  r#c UR(agSUlXlUVVs1sHoDR5HoUiM M nnnUR5H4n[ U5(dMUR UR55 M6 SSKJn U"XS9 URXU5 SU;a]USbAUSn [U [R5(aU RS:XaSUS'O U /US'OSUS'URS5 [UR40UD6Ul["R$"UR 5[&R)UR5lgs snnf)NTr)_localize_dtensor)r% device_idcpu device_ids)rr parametersmodulesr*update%torch.distributed.tensor.parallel.ddprCr/ isinstancer>devicetypepoprr_ddpweakrefref replicatestater) rrr$kwargsmr2r% submodulerCrDs r!r:_ReplicateState.initMs6    # %4M_lln!n!_M)I ++%%i&:&:&<=* L&@ VnE & k".";/ i669>>U;R+/F<(-6;F<('+|$ JJ{ #+D,<,<GG 4;KK 4J  $1;Ns"E1cURH!upURR"U0UD6 M# URR5 gN)rrOr;clear)r comm_args comm_kwargss r!r;"_ReplicateState.register_comm_hookws>&*&:&: "I II ( () C{ C'; ""$r#cXlX lgrY)rrrargsrTs r!record_init_args _ReplicateState.record_init_args|s "r#r`.rTcUR(dUR(aUR5 UR(+URlURR "U0UD6$rY)rrr@rrOrequire_backward_grad_sync _pre_forward)rrr`rTs r!forward_pre_hook _ReplicateState.forward_pre_hooksJ ??d// NN 37==/@ ,yy%%t6v66r#inputoutputc8URRU5$rY)rO _post_forward)rrrhris r!forward_post_hook!_ReplicateState.forward_post_hooks yy&&v..r#) rrOrrrrrrrrrN)__name__ __module__ __qualname____firstlineno__rPrQ__annotations__rr+rModuleset Parameterstrr/r@r:r;ratuplerdictrfr>Tensorrl__static_attributes__ __classcell__)r s@r!r r s!++ -&*  RYYBLL)     @ (K (KRYY(K  (KT% #7ii7',S#X7@DS#X7 7/ /U\\"/ /  //r#r r`rTrc[S5e)NzGDDP does not support deepcopy. Please use state dict for serialization.)AssertionError)r`rTs r!unimplemented_deepcopyrs Q r#c6\rSrSrSrS\SS4SjrS SjrSrg) DDPcLURSnUR"U/UQ70UD6$)z Override ``__new__`` to remove the DDP class and directly construct the original class for cases like indexing into a container module. )__mro____new__)clsr`rTorig_clss r!r DDP.__new__s,;;q>:4:6::r#requires_gradient_syncrNcBU(+[RU5lg)z Sets if the module should sync gradients. This can be used to implement gradient accumulation without communication. Args: requires_gradient_sync (bool): Whether to reduce gradients for the module's parameters. N)rRrSr)rrs r!set_requires_gradient_syncDDP.set_requires_gradient_syncs.D)C &r#cb[RU5RRX45 grY)rRrSrr-r_s r!r;DDP.register_comm_hooks!--44d^Dr#r9rn) rorprqrrrboolrr;r{r9r#r!rrs#; D D$ DEr#r) state_clsrr$c [RRS5 SU;aB[US[[R 45(d[ S[US535e[U5(a [ S5eUc0nO [U5n[RU5nURURSS9 URS5nUb?UR5nXT:wa*SS KJnJn URU5 UR'U5 UR'UR(5 UR*"X40UD6 UR,nS [.0n [S UR03[2U4U 5n XlU$) zReplicates a module Args: module (torch.nn.Module): module to replicate Example:: >>> # xdoctest: +REQUIRES(module:torch._C._distributed_c10d) >>> module = nn.Linear(3, 3) >>> replicate(module) ztorch.distributed.replicaterDz6Expected device_id to be int or torch.device, but got zGCannot apply `replicate()` on a Module already managed by `fully_shard`T) with_kwargs device_meshr)rC_reconstruct_dtensor __deepcopy__r)r>_C_log_api_usage_oncerKintrL RuntimeErrorrMr*rurRrSregister_forward_pre_hookrfget_get_root_meshrJrCrregister_forward_hookrlrar rror) rr$rTrSr root_meshrCrrdctnew_clss r!rRrRst  HH  !>?f&-U\\/BCC{ 3457     U  o. OOF #E $$U%;%;$N**]+K..0   #   , ,-A B  ( (): ;   !8!89 6=f=   C 1 2CS'#sS9G Mr#c*[U5nUcgSU;$)z+Check if module is marked with fully_shard.F fully_shard)r )rregistrys r!r*r*s V$H H $$r#rY)rPcollections.abcrtypingrrr>torch.nnr#torch.distributed._composable_statertorch.nn.parallelrr r r+r rrrtrRrr*r9r#r!rs$ 65-}/f}/@#EE4 O$9=C IICehhoo.5CYY C%CL%bii%D%r#