/h"zSrSSKrSSKrSSKJr SSKJrJr SSKJ r J r SSK J r SSK r SSKJrJr S S KJr S S KJrJr S S KJrJrJr \"5(a SSKJs Jr \"5(aSS KJr Sr Sr!Sr"Sr#Sr$\ SS.Sjr%SDSjr&Sr'Sr(Sr)Sr*Sr+Sr,Sr-Sr."SS \/5r0S!r1S"r2\1S#5r3S$\ 4S%jr4S$\ 4S&jr5SES'jr6SFS(jr7\ RpS \ RrS\ RtS)\ RvS*\ RxS+\ RzS,\ R|S-\ R~S.\ RS/\ RS00 rB\BR5VVs0sHupX_M snnrDS1rESGS2\ R4S3jjrG\1SES4\H4S5jj5rISES4\H4S6jjrJSHS7jrKSES8jrL"S9S:\M5rN\2SIS;j5rOSES<jrP\1SJS=j5rQS>rR"S?S@5rSSArTSBrU\SKSCj5rVgs snnf)LzB A set of basic tensor ops compatible with tpu, gpu, and multigpu N)Mapping)contextmanager nullcontext)update_wrapperwraps)Any)AcceleratorState PartialState)!TORCH_DISTRIBUTED_OPERATION_TYPES)DistributedTypeTensorInformation)is_npu_availableis_torch_distributed_availableis_torch_xla_available)ReduceOpc6[U[R5$N) isinstancetorchTensortensors U/home/james-whalen/.local/lib/python3.13/site-packages/accelerate/utils/operations.pyis_torch_tensorr,s fell ++c v[U[RR[RR[RR [RR [RR[RR[RR5$r) rrxpu FloatTensor ByteTensor IntTensor LongTensor HalfTensor DoubleTensorBFloat16Tensorrs ris_torch_xpu_tensorr'0sm            rc"[U[5$r)rr tensor_infos ris_tensor_informationr+=s k#4 55rcn[U[5=(a [US5=(a [US5$)z Checks if `data` is a `namedtuple` or not. Can have false positives, but only if a user is trying to mimic a `namedtuple` perfectly. _asdict_fields)rtuplehasattrdatas r is_namedtupler3As* dE " \wtY'? \GDR[D\\rcr[U5(a[U5"[U56$[U5"U5$)zG Cast a generator to the same type as obj (list, tuple, or namedtuple) )r3typelist)obj generators r honor_typer9Is1 SCy$y/**Cy##rF test_typeerror_on_other_typec ^^^^^[U[[45(a[UUUUUU4SjU55$[U[5(aF[ U5"UR 5VVs0sHupgU[TU/TQ7TTS.TD6_M snn5$T"U5(a T"U/TQ70TD6$T(a2[S[ U5STRSTRS35eU$s snnf)a Recursively apply a function on a data structure that is a nested list/tuple/dictionary of a given base type. Args: func (`callable`): The function to recursively apply. data (nested list/tuple/dictionary of `main_type`): The data on which to apply `func` *args: Positional arguments that will be passed to `func` when applied on the unpacked data. main_type (`type`, *optional*, defaults to `torch.Tensor`): The base type of the objects to which apply `func`. error_on_other_type (`bool`, *optional*, defaults to `False`): Whether to return an error or not if after unpacking `data`, we get on an object that is not of type `main_type`. If `False`, the function will leave objects of types different than `main_type` unchanged. **kwargs (additional keyword arguments, *optional*): Keyword arguments that will be passed to `func` when applied on the unpacked data. Returns: The same data structure as `data` with `func` applied to every object of type `main_type`. c3J># UHn[TU/TQ7TTS.TD6v M g7f)r:Nrecursively_apply).0oargsr<funckwargsr;s r $recursively_apply..msA A"!".7M`djs #r:zUnsupported types (z ) passed to `z?`. Only nested list/tuple/dicts of objects that are valid for `z` should be passed.) rr/r6r9rr5itemsr@ TypeError__name__)rDr2r;r<rCrEkvs` ```` rr@r@Ts,$ &&        D' " "Dz !JJL  )DA$!".7M`dj)     4D*4*6** !$t*]4==/J++4+=+=*>>Q S   K s. C* c^^^[U5(d[US5(aTS:XaSmURTTS9$[ U[[45(a[UUUU4SjU55$[ U[5(ad[ T[5(aT/mOTc/m[U5"UR5VVs0sHupVXUT;aUO [UTTTS9_M snn5$U$![a URT5s$[a5n[ 5(a[ T[5(aST3mSnAO UeSnAff=fURTTS9$![a URT5s$f=fs snnf) as Recursively sends the elements in a nested list/tuple/dictionary of tensors to a given device. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to send to a given device. device (`torch.device`): The device to send the data to. Returns: The same data structure as `tensor` with all tensors sent to the proper device. tonpuznpu:0) non_blockingznpu:Nc3<># UHn[UTTTS9v M g7f)rP skip_keysN)send_to_device)rAtdevicerPrSs rrF!send_to_device..s ohncd^AvLT]^hnsrR)rr0rNrIAssertionErrorrrintr/r6r9rstrr5rHrT)rrVrPrSerrorrKrUs ``` rrTrTs{v'&$"7"7 U?F 99V,9? ? FUDM * * ohno   FG $ $ i % %" I  IF|#LLN *DAY1N1fS_kt,uu*    = %99V$ $  !!fc**#F8_F   %99V,9? ? %99V$ $ % s;C, E< ,E E)EEE EE98E9cSn[X5$)a/ Recursively gathers the information needed to rebuild a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with [`~utils.TensorInformation`] instead of tensors. c>[URURS9$)N)shapedtype)rr^r_rs r_get_data_structure/get_data_structure.._get_data_structures v||6<<HHrr?)r2r`s rget_data_structurerbsI 0 77rcSn[X5$)a Recursively gathers the shape of a nested list/tuple/dictionary of tensors as a list. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to send to analyze. Returns: The same data structure as `data` with lists of tensor shapes instead of tensors. c,[UR5$r)r6r^rs r _get_shapeget_shape.._get_shapesFLL!!rr?)r2res r get_shapergs" Z ..rc$Sn[X[S9$)z Recursively initializes tensors from a nested list/tuple/dictionary of [`~utils.TensorInformation`]. Returns: The same data structure as `data` with tensors instead of [`~utils.TensorInformation`]. cV[R"URSUR06$Nr_)remptyr^r_r)s r_initialize_tensor.initialize_tensors.._initialize_tensors"{{K--G[5F5FGGrr;)r@r+)data_structurerls rinitialize_tensorsrpsH /K` aarc[U[[[45(a'[ U5S:Xa[ S[ U5S35e[U[[45(a[US5$[U[5(a%UR5Hn[X5s $ O7[U[R5(d[S[ U5S35eURS$)z Recursively finds the batch size in a nested list/tuple/dictionary of lists of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. rz&Cannot find the batch size from empty .z0Can only find the batch size of tensors but got ) rr/r6rlen ValueErrorr5find_batch_sizekeysrrrIr^)r2rKs rrurus$g.//SY!^A$t*QOPP$ &&tAw'' D' " "A"47+ + ell + +J4PT:,VWXYY ::a=rcF[U5$![[4a gf=f)a Same as [`utils.operations.find_batch_size`] except will ignore if `ValueError` and `TypeErrors` are raised Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to find the batch size. Returns: `int`: The batch size. N)rurtrIr1s rignorant_find_batch_sizerxs, t$$  "   s   cSn[X5$)a; Recursively finds tensors in a nested list/tuple/dictionary and converts them to a list of numbers. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data from which to convert to regular numbers. Returns: The same data structure as `data` with lists of numbers instead of `torch.Tensor`. cUR5R5nUR[R:XaUR [R 5nUR5$r)detachcpur_rbfloat16rNfloat32tolistrs r_convert_to_list!listify.._convert_to_list sF$$& <<5>> )YYu}}-F}}rr?)r2rs rlistifyrs - 44rcJSn[XSS9n[R"5 U$)NcURS:XaUR5SnUR5(dUR5n[R "U5$)Nr)ndimclone is_contiguous contiguousxm all_gatherrs r_tpu_gather_one$_tpu_gather.._tpu_gather_one-sL ;;! \\^D)F##%%&&(F}}V$$rTr<)r@r mark_step)rrress r _tpu_gatherr,s#% O NCLLN Jrc^^[5m[RRmTRR S:Xa[R R5 UU4Sjn[XSS9$)Nrc>URS:XaUR5SnUR5(dUR5nTRbTRS:waq[ R "TRUR5-URTRS9nT"X5 UR"S/UR5SSQ76$[TR5Vs/sHn[ R"U5PM nn[ RR!X5 [ R""USS9$s snf)Nrgloor_rVr dim)rrrrbackendrrk num_processesnumelr_rVviewsizerange empty_like distributedrcat)routput_tensors_ gather_opstates r_gpu_gather_one$_gpu_gather.._gpu_gather_oneCs  ;;! \\^D)F##%%&&(F == $&)@ #[[##flln4ll||N n -!&&r>FKKM!",=> > AFeFYFY@Z[@Z1e..v6@ZN[    ( ( @99^3 3\s/ ETr) r rrall_gather_into_tensorrVr5r synchronizer@)rrrrs @@r _gpu_gatherr;sS NE!!88I ||E! 48 _$ OOrc\rSrSrSrSrg)DistributedOperationExceptionibz} An exception class for distributed operations. Raised if the operation cannot be performed due to the shape of the tensors. N)rJ __module__ __qualname____firstlineno____doc____static_attributes__rrrrrbs   rrc0^[T5U4Sj5nU$)zn Verifies that `tensor` is the same shape across all processes. Only ran if `PartialState().debug` is `True`. c R>[5R[R:Xd[5R(dT "U0UD6$T R ST R 3nSU;aUSnOUSn[5RR[U5R:wag[SUSURRS[5RRS[5RRSUS3 5e[U5n[U/5nUSbmURUS5[U5:HnU(dFS R[!U5VVs/sH upxS US U3PM snn5n [S US U 35eT "U0UD6$s snnf)Nrrrrz%One or more of the tensors passed to z were not on the z+ while the `Accelerator` is configured for z. Please move it to the z before calling z - zProcess z: znCannot apply desired operation due to shape mismatches. All shapes across devices must be valid. Operation: `z` Input shapes: - )r distributed_typerNOdebugrrJrVr5 find_devicerrg gather_objectcountrsjoin enumerate) rCrE operationrshapesoutputare_sameir^process_shape_strfunctions rwrapper!verify_operation..wrapperps > * *o.@.@ @ H\H\T,V, ,**+1X->->,?@ v H%F!WF > % %V)<)A)A A/7 {BSTZTaTaTfTfSghST`TbTiTiTnTnSoo))5)>)>)C)C(DDTU^T__`b 6"x( !9 ||F1I.#f+=H$,MM[dek[l2m[lxqXaS5'3J[l2m$n!3''0k1GHYGZ\ ((( 3ns.F# rrrs` rverify_operationrks"  8_))4 Nrc0^[T5U4Sj5nU$)z Checks that `verify_operation` failed and if so reports a more helpful error chaining the existing `DistributedOperationException`. c>T"U0UD6$![a0nTRSTR3n[SUS35UeSnAff=f)NrrzError found while calling `z1`. Please see the earlier error for more details.)rrrJ)rCrEerrs rr"chained_operation..wrappersc T,V, ,, #../q1B1B0CDI/-i[8ij  s A+AArrs` rchained_operationrs"  8_ Nrc[5R[R:Xa [ U5$[5R[ ;a [ U5$U$)a Recursively gather tensor in a nested list/tuple/dictionary of tensors from all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. Returns: The same data structure as `tensor` with all tensors sent to the proper device. )r rrXLArr rrs rgatherrsF~&&/*=*==6""  ( (,M M6"" robjectc[[5R5Vs/sHnSPM nn[RR X 5 UVVs/sH o3HoDPM M snn$s snfs snnfr)rr rrrall_gather_object)rroutput_objectsyxs r_gpu_gather_objectrsa$),.*F*F$GH$Gqd$GNH ''?% 1~!q!AqA~ 11I 2s A*A/c[5R[R:Xa [ S5e[5R[ ;a [ U5$U$)a Recursively gather object in a nested list/tuple/dictionary of objects from all devices. Args: object (nested list/tuple/dictionary of picklable object): The data to gather. Returns: The same data structure as `object` with all the objects sent to every device. z&gather objects in TPU is not supported)r rrrNotImplementedErrorr r)rs rrrsG~&&/*=*==!"JKK  ( (,M M!&)) rc"SSjn[X SUS9$)Nc@[RRXS9 U$)Nsrc)rr broadcast)rrs r_gpu_broadcast_one*_gpu_broadcast.._gpu_broadcast_ones ##F#4 rT)r<rrr?)r2rrs r_gpu_broadcastrs /4UX YYrc l^^[U[[45(a[UU4Sj[ U555$[U[ 5(aB[ U5"UR5VVs0sHup4U[UTSU3S9_M snn5$[R"TUU4Sj5$s snnf)Nc3F># UHup[UTSU3S9v M g7f)rnameN)_tpu_broadcast)rArrUrs rrF!_tpu_broadcast..s&"gUfTQ>!TF!A3-#HUfs!rrc>UT$rr)rrs r _tpu_broadcast..s !C&r) rr6r/r9rrr5rHrr mesh_reduce)rrrrKrLs `` rrrs&4-((&"gU^_eUf"ghh FG $ $F|RXR^R^R`aR`$!Qq$q} EER`abb >>$(8 99bs0B0  cSn[5n[R"U[RURS9nUbTUR n[ URn[R"[U5U/-[S9US[U5S-&[USS9nX3R5n[USSS 5nUSSnX64$) z] Grabs the shape of `tensor` only available on one process and returns a tensor of its shape irNr_r sum reductionrr) r rrkrYrVr^TENSOR_TYPE_TO_INTr_rr6rsreducenonzero)rmax_tensor_dimensionr base_tensorr^ tensor_dtyper_s rgather_tensor_shapers ! NE++2%))ELLYK  )&,,7 (- T%[L>5QY\(] $c%j1n%6K1134K  BC # $Ecr"K  rreturnc[5n[U5up#Uc5[R"U[US9R UR 5n[USS9$)as Copys a tensor that only exists on a single device and broadcasts it to other devices. Differs from `broadcast` as each worker doesn't need to know its shape when used (and tensor can be `None`) Args: tensor (`torch.tensor`): The tensor that should be sent to all devices. Must only have it be defined on a single device, the rest should be `None`. rrr)r rrzerosTENSOR_INT_TO_DTYPErNrVr)rrr^r_s rcopy_tensor_to_devicesrsN NE&v.LE ~U*=e*DEHHV &E **r from_processc[5R[R:Xa [ XSS9$[5R[ ;a [ XS9$U$)a Recursively broadcast tensor in a nested list/tuple/dictionary of tensors to all devices. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. from_process (`int`, *optional*, defaults to 0): The process from which to send the data Returns: The same data structure as `tensor` with all tensors broadcasted to the proper device. zaccelerate.utils.broadcast)rrr)r rrrrr r)rrs rrrsI~&&/*=*==f=YZZ  ( (,M Mf77 rc0^[5R[R:Xa4[ U5H#up#[ R "SUU4Sj5X'M% U$[5R[;a[RRUTS9 U$)ar Broadcast a list of picklable objects form one process to the others. Args: object_list (list of picklable objects): The list of objects to broadcast. This list will be modified inplace. from_process (`int`, *optional*, defaults to 0): The process from which to send the data. Returns: The same list containing the objects from process 0. z&accelerate.utils.broadcast_object_listc>UT$rr)rrs rr'broadcast_object_list..>s efgsetrr) r rrrrrrr rrbroadcast_object_list) object_listrrr7s ` rr r /s}~&&/*=*== ,FA^^,TVY[tuKN-   ( (,M M // /N rc Sn[X@U5$)a* Recursively takes a slice in a nested list/tuple/dictionary of tensors. Args: data (nested list/tuple/dictionary of `torch.Tensor`): The data to slice. tensor_slice (`slice`): The slice to take. Returns: The same data structure as `data` with all the tensors slices. c X$rr)r tensor_slices r _slice_tensor$slice_tensors.._slice_tensorRs ##rr?)r2r process_indexrrs r slice_tensorsrDs$ ], ??rc.^^[TS[[45(a/[TSUU4Sj[ [ TS5555$[TS[ 5(aS[TS5"TSR5VVs0sH o"[TVs/sHo3UPM snTS9_M" snn5$[TS[R5(d[S[TS535e[R"TTS9$s snfs snnf)a Recursively concatenate the tensors in a nested list/tuple/dictionary of lists of tensors with the same shape. Args: data (nested list/tuple/dictionary of lists of tensors `torch.Tensor`): The data to concatenate. dim (`int`, *optional*, defaults to 0): The dimension on which to concatenate. Returns: The same data structure as `data` with all the tensors concatenated. rc3h># UH"n[TVs/sHo"UPM snTS9v M$ gs snf7f)rN) concatenate)rArdr2rs rrFconcatenate..fs/#lXkSTKt0Dt!1t0D#$NXk0Ds 2- 2rz%Can only concatenate tensors but got )rr/r6r9rrsrr5rvrrrrIr)r2rrKrs`` rrrXs$q'E4=))$q'#lX]^abfghbi^jXk#lmm DGW % %DG}UYZ[U\UaUaUcdUcPQD-ADqdD-As!KKUcdee Q . .?T!W OPP 99Ts ##.Bds D D ) D D c\rSrSrSrg)CannotPadNestedTensorWarninginrN)rJrrrrrrrrrnsrrc $SSjn[X@SXUS9$)a Recursively pad the tensors in a nested list/tuple/dictionary of tensors from all devices to the same size so they can safely be gathered. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to gather. dim (`int`, *optional*, defaults to 0): The dimension on which to pad. pad_index (`int`, *optional*, defaults to 0): The value with which to pad. pad_first (`bool`, *optional*, defaults to `False`): Whether to pad at the beginning or the end. c^^ ^ [USS5(a[R"S[5 U$T[ UR 5:dT[ UR 5*:aU$TS:aT[ UR 5- m[ R"UR URS9Sn[U5R5n[U4SjU55m T UR T:XaU$UR m [T 5nT UT'UR[U55U-nU(a*[UU U 4Sj[[ U5555nO([UU 4Sj[[ U5555nXU'U$) N is_nestedFzHCannot pad nested tensors without more information. Leaving unprocessed.r)rVc3,># UH oTv M g7frr)rAsrs rrFFpad_across_processes.._pad_across_processes..s-u!vusc3h># UH'oT:Xa[TTT- T5O [S5v M) g7frslice)rArrmax_sizeold_sizes rrFrs4`t[\#Xh#.95QU;V`ts/2c3b># UH$oT:Xa[STT5O [S5v M& g7frNr!rArrr$s rrFrs,oZnUVCxE!Xc]3U4[PZn,/)getattrwarningswarnrrsr^rrrVrr|maxr6 new_zerosr/r) rr pad_index pad_firstrsizesnew_size new_tensorindicesr#r$s ` @@r_pad_across_processes3pad_across_processes.._pad_across_processessL 6; . . MMZ, M #fll# #sc&,,.?-?'?M 7 3v||$ $C||FLL?Et   "-u-- v||C( (M<<>  %%eHo6B `efijrfs`tGoZ_`cdl`mZnooG$7rT)r<rr.r/rrFr?)rrr.r/r4s rpad_across_processesr7rs!" D 4Sir rc (SSjn[UUSUUUS9$)z Takes a `tensor` of arbitrary size and pads it so that it can work given `num_processes` needed dimensions. New tensors are just the last input repeated. E.g.: Tensor: ([3,4,4]) Num processes: 4 Expected result shape: ([4,4,4]) c<^^ X-nXU-- nX-S:XaX!- nOX!U-- nXVU-s=:aS:aO OXV- nURm [T 5nX-US'UR[U55n[UU 4Sj[ [ U5555n XU 'U$)Nrr c3b># UH$oT:Xa[STT5O [S5v M& g7fr&r!r's rrF@pad_input_tensors.._pad_input_tensors..s,kVjQR8a#/tLVjr()r^r6r-r/rrs) r batch_sizerr remainder last_inputsto_padr1r2r3r$s ` @r_pad_input_tensors-pad_input_tensors.._pad_input_tensorss/  $=>  &! +"/F"M&ABF & ,1 , )F<<> ) %%eHo6 kV[\_`h\iVjkk$7rT)r<r<rrrr?)rr<rrr@s rpad_input_tensorsrBs(&  #   rc"SSjn[X0SXS9$)a( Recursively reduce the tensors in a nested list/tuple/dictionary of lists of tensors across all processes by the mean of a given operation. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to reduce. reduction (`str`, *optional*, defaults to `"mean"`): A reduction method. Can be of "mean", "sum", or "none" scale (`float`, *optional*): A default scaling value to be applied after the reduce, only valied on XLA. Returns: The same data structure as `data` with all the tensors reduced. c[5nUR5nUR[R:XaU$UR[R :XaR[ R"5 [ R"[ RU/U5 [ R"5 OLURR[;a.[RRU[R5 US:XaXCR -nU$)Nmean)r rrrrrrr all_reduce REDUCE_SUMvaluer rrrSUMr)rrscaler cloned_tensors r_reduce_across_processes(reduce.._reduce_across_processess   ! !_%7%7 7  ! !_%8%8 8 LLN MM"---% @ LLN  # # ) )-N N    ( ( E   00 0MrT)r<rrJrEg?r?)rrrJrLs rrrs$&  di rc"SnSn[XUS9$)aZ Recursively converts the elements nested list/tuple/dictionary of tensors in FP16/BF16 precision to FP32. Args: tensor (nested list/tuple/dictionary of `torch.Tensor`): The data to convert from FP16/BF16 to FP32. Returns: The same data structure as `tensor` with all tensors that were in FP16/BF16 precision converted to FP32. c"UR5$r)floatrs r_convert_to_fp32)convert_to_fp32.._convert_to_fp32s||~rc[U5=(d [US5=(a- UR[R[R 4;$rj)rr0r_rfloat16r}rs r_is_fp16_bf16_tensor-convert_to_fp32.._is_fp16_bf16_tensor s@'C767+C  MM NNZ J  rrnr?)rrRrVs rconvert_to_fp32rXs -AU VVrc*\rSrSrSrSrSrSrSrg)ConvertOutputsToFp32iaD Decorator to apply to a function outputing tensors (like a model forward pass) that ensures the outputs in FP16 precision will be convert back to FP32. Args: model_forward (`Callable`): The function which outputs we want to treat. Returns: The same function as `model_forward` but with converted outputs. c&Xl[X5 gr) model_forwardr)selfr\s r__init__ConvertOutputsToFp32.__init__!s*t+rc8[UR"U0UD65$r)rXr\)r]rCrEs r__call__ConvertOutputsToFp32.__call__%st114B6BCCrc.[R"S5e)NzCannot pickle a prepared model with automatic mixed precision, please unwrap the model with `Accelerator.unwrap_model(model)` before pickling it.)pickle PicklingError)r]s r __getstate__!ConvertOutputsToFp32.__getstate__(s"" `  r)r\N) rJrrrrr^rarfrrrrrZrZs ,D rrZc8^[T5mU4SjnTUlU$)Nc>T"U0UD6$rr)rCrEr\s rforward(convert_outputs_to_fp32..forward1sd-f--r)rZ __wrapped__)r\rjs` rconvert_outputs_to_fp32rm.s!(7M.(G NrcF[U[5(a*UR5Hn[U5nUcMUs $ g[U[[ 45(aUHn[U5nUcMUs $ g[U[ R5(a UR$g)z Finds the device on which a nested dict/list/tuple of tensors lies (assuming they are all on the same device). Args: (nested list/tuple/dictionary of `torch.Tensor`): The data we want to know the device of. N) rrvaluesrr/r6rrrV)r2r7rVs rrr:s$  ;;=C %F! ! D5$- ( (C %F!  D%,, ' '{{ (rc#b# [5R[R:wd<[5Rb2[5RR 5(d [ 5nOSSKnURRXX#S9nU Sv SSS5 g!,(df  g=f7f)z| Wrapper around `deepspeed.runtime.zero.GatheredParameters`, but if Zero-3 is not enabled, will be a no-op context manager. Nr) modifier_rank fwd_moduleenabled) r rr DEEPSPEEDdeepspeed_pluginis_zero3_init_enabledr deepspeedzeroGatheredParameters)paramsrqrrrsgather_param_contextrws rryryOs**o.G.GG++7 "33IIKK*}(~~@@ J A      sBB/B B/ B,(B/)FNr)rzbroadcast tensorr)NNr6rN)NNT)Wrrdr*collections.abcr contextlibrr functoolsrrtypingrrrr r constantsr dataclassesrrimportsrrrtorch_xla.core.xla_modelcore xla_modelrtorch.distributedrrr'r+r3r9r@rTrbrgrprurxrrr ExceptionrrrrrrrrrQdoublehalfr}uint8int8int16int32int64boolrrHrrrrrYrr rr UserWarningrr7rBrrXrZrmrry)rKrLs00rrs;#2+ 28;))!##*, 6]$4CX]0f1h8$/$ b."5. $PN I  F&&2s2#&Z: KK LL! JJ NNA KK JJ KK KK KK JJ );(@(@(BC(Bqt(BC2+5<<+"C*S*@($, ; 44n%P&&RW0  4 *I Ds$G/