+hsSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r J r J r J r J r JrJr SSKrSSKrSSKJr SSKJr SSKJr SSKJrJrJrJrJrJrJ r \ "5(a1SSK!r!\!RDRFRI5(aSSK#r#\"5(aSS K%J&r& \"5(aSS K'J(r( \"5(aSSK)r)S \*4S jr+S r,S\-4Sjr.S6S\S\S\R^S\R^S\R^S\R^S\R^S\0S\\ \R^\ \R^44Sjjr1S\S\ \-\R^44Sjr2\Rf4S\\RhRj\ \RhRj44Sjjr6S\ \-\R^4S\-S\RhRj4S jr7S!\ \-\RhRj4S\ \-\ 44S"jr8S7S#\-S$\*S%\0S&\0S'\0S(\\Rr\-4S)\ \Rt4S*jjr;S8S#\-4S+jjr4S1jj5r?S2r@S3rA"S4S55rBg)9N)contextmanager)AnyDictIterableListOptionalTupleUnion)UNet2DConditionModel)DiffusionPipeline)SchedulerMixin)convert_state_dict_to_diffusersconvert_state_dict_to_peft deprecateis_peft_availableis_torch_npu_availableis_torchvision_availableis_transformers_available)set_peft_model_state_dict) transformsseedc6[R"U5 [RRU5 [R"U5 [ 5(a [R RU5 g[RRU5 g)z Helper function for reproducible behavior to set the seed in `random`, `numpy`, `torch`. Args: seed (`int`): The seed to set. Returns: `None` N) randomrnptorch manual_seedrnpumanual_seed_allcuda)rs R/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/training_utils.pyset_seedr",s[ KKIINN4 d !!$' ""4(cURnUS-nSU- S-nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nURURS9UR5n[ UR 5[ UR 5:a3USn[ UR 5[ UR 5:aM3UR UR 5nXV- S-nU$)a_ Computes SNR as per https://github.com/TiankaiHang/Min-SNR-Diffusion-Training/blob/521b624bd70c67cee4bdf49225915f5945a872e3/guided_diffusion/gaussian_diffusion.py#L847-L849 for the given timesteps using the provided noise scheduler. Args: noise_scheduler (`NoiseScheduler`): An object containing the noise schedule parameters, specifically `alphas_cumprod`, which is used to compute the SNR values. timesteps (`torch.Tensor`): A tensor of timesteps for which the SNR is computed. Returns: `torch.Tensor`: A tensor containing the computed SNR values for each timestep. ??device).N)alphas_cumprodtor(floatlenshapeexpand)noise_scheduler timestepsr*sqrt_alphas_cumprodsqrt_one_minus_alphas_cumprodalphasigmasnrs r! compute_snrr7@sF %33N(#-%(>%9c$A!.00 8H8H0I)TZZ\ !'' (3y+? ?1)< !'' (3y+? ?  & &y 7E$A$D$DIL\L\$D$]^g$h$n$n$p! +11 2S5I I(Ei(P% +11 2S5I I ) 0 0 AE =Q C Jr#interpolation_typec0[5(d [S5eUS:Xa[RRnU$US:Xa[RR nU$US:Xa[RR nU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$US:Xa[RRnU$[S US 35e) a Maps a string describing an interpolation function to the corresponding torchvision `InterpolationMode` enum. The full list of supported enums is documented at https://pytorch.org/vision/0.9/transforms.html#torchvision.transforms.functional.InterpolationMode. Args: interpolation_type (`str`): A string describing an interpolation method. Currently, `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos` are supported, corresponding to the supported interpolation modes in torchvision. Returns: `torchvision.transforms.InterpolationMode`: an `InterpolationMode` enum used by torchvision's `resize` transform. zhPlease make sure to install `torchvision` to be able to use the `resolve_interpolation_mode()` function.bilinearbicubicboxnearest nearest_exacthamminglanczoszThe given interpolation mode z is not supported. Currently supported interpolation modes are `bilinear`, `bicubic`, `box`, `nearest`, `nearest_exact`, `hamming`, and `lanczos`.) r ImportErrorrInterpolationModeBILINEARBICUBICBOXNEAREST NEAREST_EXACTHAMMINGLANCZOS ValueError)r8interpolation_modes r!resolve_interpolation_moderLes> $ % % v  Z''99BB& % y ('99AA" ! u $'99==  y ('99AA   .'99GG  y ('99AA  y ('99AA  +,>+?@m n  r#unetr0r1noise noisy_latentstargetencoder_hidden_statesdream_detail_preservationreturnc\URRUR5USSS4nSU- S-n X-n Sn [R"5 U"XBU5R n SSS5 SupUR RS:XaLU nX>- R5nURU 5 URX-5n URU5n X4$UR RS:Xa [S5e[SUR R35e!,(df  N=f) a Implements "DREAM (Diffusion Rectification and Estimation-Adaptive Models)" from https://huggingface.co/papers/2312.00210. DREAM helps align training with sampling to help training be more efficient and accurate at the cost of an extra forward step without gradients. Args: `unet`: The state unet to use to make a prediction. `noise_scheduler`: The noise scheduler used to add noise for the given timestep. `timesteps`: The timesteps for the noise_scheduler to user. `noise`: A tensor of noise in the shape of noisy_latents. `noisy_latents`: Previously noise latents from the training loop. `target`: The ground-truth tensor to predict after eps is removed. `encoder_hidden_states`: Text embeddings from the text model. `dream_detail_preservation`: A float value that indicates detail preservation level. See reference. Returns: `tuple[torch.Tensor, torch.Tensor]`: Adjusted noisy_latents and target. Nr&r%)NNepsilon v_predictionz/DREAM has not been implemented for v-predictionzUnknown prediction type ) r*r+r(rno_gradsampleconfigprediction_typedetachmul_addNotImplementedErrorrJ)rMr0r1rNrOrPrQrRr*r3 dream_lambdapred_noisy_latents_targetpredicted_noise delta_noises r! compute_dream_and_update_latentsres*:%3366y7G7GHTXZ^`dIdeN%(>%9c$A!1KL D M.CDKK +N--:.668 &&**+H+VW**[)  ""    / /> A!"STT3O4J4J4Z4Z3[\]] s D D+c0nUR5H[up#[US5(dM[US5nUcM)UR5nUR 5HupgXqUSU3'M M] U$)z@ Returns: A state dict containing just the LoRA parameters. set_lora_layer lora_layerz.lora.) named_moduleshasattrgetattr state_dictitems)rMlora_state_dictnamemodulerhcurrent_lora_layer_sdlora_layer_matrix_name lora_params r!unet_lora_state_dictrts O**,  6+ , , 6J%(2(=(=(?%:O:U:U:W6*OYtfF3I2J$KL;X - r#modelc[U[5(dU/nUHCnUR5H,nUR(dMUR U5UlM. ME g)z Casts the training parameters of the model to the specified data type. Args: model: The PyTorch model whose parameters will be cast. dtype: The data type to which the model parameters will be cast. N) isinstancelist parameters requires_gradr+data)rudtypemparams r!cast_training_paramsrsM eT " " \\^E""""XXe_ $r#rnprefix text_encodercUR5VVs0sH1up4URU5(dMURUS5U_M3 nnn[[ U55n[ X%SS9 gs snnf)a, Sets the `lora_state_dict` into `text_encoder` coming from `transformers`. Args: lora_state_dict: The state dictionary to be set. prefix: String identifier to retrieve the portion of the state dict that belongs to `text_encoder`. text_encoder: Where the `lora_state_dict` is to be set. default) adapter_nameN)rm startswithreplacerrr)rnrrkvtext_encoder_state_dicts r!!_set_state_dict_into_text_encoderrss3B2G2G2I2I$!Q\\Z`Ma%199VR !A%2I99XYp9qrlR[\ s A.A.modules_to_savec0nUR5H,up#UcM URSR5XS3'M. U$)Nr_lora_adapter_metadata)rm peft_configto_dict)r metadatas module_namerps r!_collate_lora_metadatarsPI.446  @F@R@RS\@]@e@e@gI %;< = 7 r#weighting_scheme batch_size logit_mean logit_std mode_scaler( generatorcvUS:XaB[R"X#U4XVS9n[RRR U5nU$US:XaT[R "U4XVS9nSU- U[R "[RU-S- 5S-S- U--- nU$[R "U4XVS9nU$)z Compute the density for sampling the timesteps when doing SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. logit_normal)meanstdsizer(rmode)rr(rr r)) rnormalnn functionalsigmoidrandcosmathpi)rrrrrr(rus r!%compute_density_for_timestep_samplingrs >) LLjzmTZ p HH   ' ' * H V # JJZM& N EJ%))DGGaK!O"<"AA"E"IJ J H JJZM& N Hr#cUS:XaUS-R5nU$US:Xa)SSU-- SUS---nS[RU-- nU$[R"U5nU$)z Computes loss weighting scheme for SD3 training. Courtesy: This was contributed by Rafie Walker in https://github.com/huggingface/diffusers/pull/8528. SD3 paper reference: https://huggingface.co/papers/2403.03206v1. sigma_sqrtgcosmapr r))r,rrr ones_like)rsigmas weightingbots r!compute_loss_weighting_for_sd3r!sz<'T\((*  X %!f*nq619},3'  OOF+ r#cV[R"5 [RR 5(a[RR 5 g[R RR 5(a[RR 5 g[5(a[RR 5 g[[S5(aC[RR 5(a[RR 5 ggg)zN Runs garbage collection. Then clears the cache of the available accelerator. xpuN) gccollectrr is_available empty_cachebackendsmpsr torch_npurrjrr#r! free_memoryr3sJJL zz       ( ( * *   ! ! !!#   599#9#9#;#; $< r#T)offloadmodulesrc' # U(a[SU55(+nU(a4UVs/sH&n[UR55RPM( nnO![ U5S:XdeUSR/nUHnUR U5 M Sv U(a'[ UW5HupFUR U5 M ggs snf!U(a'[ UW5HupFUR U5 M ff=f7f)a Context manager that, if offload=True, moves each module to `device` on enter, then moves it back to its original device on exit. Args: device (`str` or `torch.Device`): Device to move the `modules` to. offload (`bool`): Flag to enable offloading. c3B# UHn[U[5v M g7fN)rwr ).0r}s r! !offload_models..PsMW:a):;;Wsr rN)anynextryr(r-r+zip)r(rris_modelr}original_devicesorig_devs r!offload_modelsrCsMWMMM ELMWQ\\^ 4 ; ;W M w<1$ $$ ' 1 12 A DDL "7,<= X >  N "7,<= X > s(*D-C >DC4D0DDc^U(d [S5eUR5RS5n/nUHn[R"SU5nU(d[SUS35e[ UR S55n[ UR S55nUS::dUS::a [S 5eUS -S:wd US -S:wa[R"S US US 35 URXV45 M U(d [S5eU$![an[SUSU35UeSnAff=f)zGParses a string defining buckets into a list of (height, width) tuples.zBucket string cannot be empty.;z^\s*(\d+)\s*,\s*(\d+)\s*$zInvalid bucket format: 'z'. Expected 'height,width'.r r)rz,Bucket dimensions must be positive integers.zBucket dimension (,z.) not divisible by 8. This might cause issues.z Invalid integer in bucket pair 'z': Nz.No valid buckets found in the provided string.) rJstripsplitrematchintgroupwarningswarnappend) buckets_str bucket_pairsparsed_bucketspair_strrheightwidthes r!parse_buckets_stringres/ 9::$$&,,S1LN 5x@7zA\]^ ^ YQ(F A'E{eqj !OPPzQ%!)q. 26(!E7Bpqr  ! !6/ 2! IJJ   Y?zQCPQWX X Ys'B D  D,D''D,c[S5nSn[U5H%unupg[X-X-- 5nX::dM!UnUnM' U$)z6Finds the closes bucket to the given height and width.infN)r, enumerateabs) hwbucket_options min_metricbest_bucket_idx bucket_idxbucket_hbucket_wmetrics r!find_nearest_bucketrsSuJO,5n,E( (XQ\AL01  J(O -F r#c>\rSrSrSrSS\\RRS\ S\ S\ S\ S \ \ \ 4S \ \ \ 4S \ S \ \S \\\44Sjjr\SS Sjj5rSrS\ S\ 4Sjr\R,"5S\\RR4Sj5rS\\RRSS4SjrS!SjrS"S!SjjrS\4SjrS\\RRSS4SjrS\\RRSS4SjrS\SS4SjrSr g)#EMAModeliz. Exponential Moving Average of models weights Nrydecay min_decayupdate_after_stepuse_ema_warmup inv_gammapowerforeach model_cls model_configc  [U[RR5(a Sn [ SSU SS9 UR 5nSnU R SS5bS n [ SSU SS9 U SnU R S S5bS n [ S SU SS9 U S n[U5nUV s/sH oR5R5PM" sn Ul U R S S5b S n [ S SU SS9 URU S S9 SUl X l X0lX@lXPlX`lXplSUlSUlXlXlXlgs sn f)a Args: parameters (Iterable[torch.nn.Parameter]): The parameters to track. decay (float): The decay factor for the exponential moving average. min_decay (float): The minimum decay factor for the exponential moving average. update_after_step (int): The number of steps to wait before starting to update the EMA weights. use_ema_warmup (bool): Whether to use EMA warmup. inv_gamma (float): Inverse multiplicative factor of EMA warmup. Default: 1. Only used if `use_ema_warmup` is True. power (float): Exponential factor of EMA warmup. Default: 2/3. Only used if `use_ema_warmup` is True. foreach (bool): Use torch._foreach functions for updating shadow parameters. Should be faster. device (Optional[Union[str, torch.device]]): The device to store the EMA weights on. If None, the EMA weights will be stored on CPU. @crowsonkb's notes on EMA Warmup: If gamma=1 and power=1, implements a simple average. gamma=1, power=2/3 are good values for models you plan to train for a million or more steps (reaches decay factor 0.999 at 31.6K steps, 0.9999 at 1M steps), gamma=1, power=3/4 for models you plan to train for less (reaches decay factor 0.999 at 10K steps, 0.9999 at 215.4k steps). zzPassing a `torch.nn.Module` to `ExponentialMovingAverage` is deprecated. Please pass the parameters of the module instead.z9passing a `torch.nn.Module` to `ExponentialMovingAverage`1.0.0F standard_warnT max_valueNzCThe `max_value` argument is deprecated. Please use `decay` instead. min_valuezGThe `min_value` argument is deprecated. Please use `min_decay` instead.r(z=The `device` argument is deprecated. Please use `to` instead.r'r)rwrrModulerrygetrxcloner[ shadow_paramsr+temp_stored_paramsrrrrrroptimization_stepcur_decay_valuerrr)selfryrrrrrrrrrkwargsdeprecation_messageps r!__init__EMAModel.__init__sgF j%((// 2 2D  K##   $..0J"N ::k4 ( 4"g  k7,?u U;'E ::k4 ( 4"k  k7,?u U{+I*% :DE*Qggi..0*E ::h % 1"a  h)> !ij j    $lm m**4+<+<=__& -   .:. U%%'( d#r#rc[SXR- S- 5nUS::agUR(a#SSX R- -UR*-- nO SU-SU-- n[ X0R 5n[X0R5nU$)z> Compute the decay factor for the exponential moving average. rr )maxrrrrminrr)rrsteprs r! get_decayEMAModel.get_decays1'*@*@@1DE 19   1tnn'<#<$**"LLO 4xBI6Oozz:o~~>r#c T[U[RR5(aSn[ SSUSS9 UR 5n[ U5nU=RS- slURUR5nX0l SU- n[R"5nUR(Ga~[5(aK[RR R#5(a[ R$R'USS9nU UVs/sHofR((dMUPM nn[+UR,U5VVs/sHupUR((dMUPM n nn[/U5[/U5:au[R0"[+UR,U5VVs/sHupUR((aMUPM snnUVs/sHofR((aMUPM snSS 9 [R2"U [R4"X5US 9 SSS5 g[+UR,U5Hup[5(aK[RR R#5(a[ R$R'USS9nU UR((aUR7XHU- -5 OUR9U5 SSS5 M gs snfs snnfs snnfs snf!,(df  g=f!,(df  M=f) NzPassing a `torch.nn.Module` to `ExponentialMovingAverage.step` is deprecated. Please pass the parameters of the module instead.z>passing a `torch.nn.Module` to `ExponentialMovingAverage.step`rFrr ) modifier_rankT) non_blocking)r4)rwrrrrryrxrr$r contextlib nullcontextrr transformers integrations deepspeedis_deepspeed_zero3_enabledzeroGatheredParametersrzrrr-_foreach_copy_ _foreach_sub_ _foreach_subsub_copy_) rryr rone_minus_decaycontext_managerr~ params_grads_param s_params_grads r!r# EMAModel.stepso j%((// 2 2D  P##   $..0J*%  !#t556$e)$002 <<<(**|/H/H/R/R/m/m/o/o"+.."C"CJ^b"C"c 2<T*@S@Su* T25d6H6H*2U!2UY^YlYlG2U!{#c*o5((7:4;M;Mz7Zv7Z^Wbgbubu7Zv,6RJ5>Q>QJR%) ##!5#5#5m#QYh!$#&d&8&8*"E,..<3L3L3V3V3q3q3s3s&/nn&G&G]a&G&bO$** _%%HI e, %_ #F#U! wR!,%_sgLK1.K14LK6*K60AL4K< K< LL 4L :0L*:L1L L L' c [U5nUR(a~[R"UVs/sHo"RPM sn[ UR U5VVs/sH*up2URUR5RPM, snn5 g[ UR U5HCup2URRURUR5R5 ME gs snfs snnf)a1 Copy current averaged parameters into given collection of parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored moving averages. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. N) rxrrr1r{rrr+r(r5)rryr~r9s r!rEMAModel.copy_toMs*% <<  )344EHI[I[]gEhiEh>7ELL)..Ehi  #&d&8&8*"E   ELL!9!>!>?#F 5is C9!1C> clURVs/sHoR5PM snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to pinned memory. Useful for non-blocking transfers for offloading EMA params to the host. N)r pin_memory)rr s r!r?EMAModel.pin_memory`s* 7;6H6HI6Hlln6HIIs1c URVs/sH7nUR5(aURXUS9OURXS9PM9 snUlgs snf)z Move internal buffers of the ExponentialMovingAverage to `device`. Args: device: like `device` argument to `torch.Tensor.to` )r(r|r()r(r(N)ris_floating_pointr+)rr(r|r(r s r!r+ EMAModel.tohse''  (""$$ DD,D GV? @(   s>Ac URURURURURUR UR URS.$)z Returns the state of the ExponentialMovingAverage as a dict. This method is used by accelerate during checkpointing to save the ema state dict. rrrrrrrrrE)rs r!rlEMAModel.state_dictwsNZZ!%!7!7!%!7!7"11ZZ!//  r#cUVs/sH.o"R5R5R5PM0 snUlgs snf)z Saves the current parameters for restoring later. Args: parameters: Iterable of `torch.nn.Parameter`. The parameters to be temporarily stored. N)r[cpurr)rryr~s r!storeEMAModel.stores6NX"XZE<<>#5#5#7#=#=#?Z"X"Xs5AcURc [S5eUR(aV[R"UVs/sHo"R PM snURVs/sHo3R PM sn5 OD[ URU5H*up2UR RUR 5 M, SUlgs snfs snf)a Restore the parameters stored with the `store` method. Useful to validate the model with EMA parameters without: affecting the original optimization process. Store the parameters before the `copy_to()` method. After validation (or model saving), use this to restore the former parameters. Args: parameters: Iterable of `torch.nn.Parameter`; the parameters to be updated with the stored parameters. If `None`, the parameters with which this `ExponentialMovingAverage` was initialized will be used. NzGThis ExponentialMovingAverage has no `store()`ed weights to `restore()`)r RuntimeErrorrrr1r{rr5)rryr~c_params r!restoreEMAModel.restores  " " *hi i <<  )344SWSjSj6kSj||Sj6k #&d&=&=z"J   .#K#'56ks C #C rlcH[R"U5nURSUR5UlURS:dURS:a [ S5eURSUR 5Ul[ UR [5(d [ S5eURSUR5Ul[ UR[5(d [ S5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[5(d [ S 5eURS UR5Ul [ UR[[45(d [ S5eURSUR5Ul[ UR[[45(d [ S5eURSS5nUb]X l[ UR[ 5(d [ S5e[#SUR55(d [ S5egg)z Loads the ExponentialMovingAverage state. This method is used by accelerate during checkpointing to save the ema state dict. Args: state_dict (dict): EMA state. Should be an object returned from a call to :meth:`state_dict`. rrr&zDecay must be between 0 and 1rzInvalid min_decayrzInvalid optimization_steprzInvalid update_after_steprzInvalid use_ema_warmuprzInvalid inv_gammarz Invalid powerrNzshadow_params must be a listc3V# UHn[U[R5v M! g7fr)rwrTensor)rr s r!r+EMAModel.load_state_dict..s O@4@&J   D &Y);); <YY'(588+=+=">'4'2.F$.F4.Fr#r)r&)NNNrHNr)Cr)rTrrrrrrtypingrrrrrr r numpyrrmodelsr pipelinesr schedulersrutilsrrrrrrrr+r,r-r.peftr torchvisionrrrr"r7r_rLrRr,rertfloat32rrrrrr( GeneratorrrrrVrrrrrr#r!rms %DDD ((&  **EEGG.&)3)("J)3)h(+3# 3##3#||3# << 3# << 3# LL 3#!<<3# %3# 8ELL !8ELL#9 9:3#l3S%,,=N8O&V[UbUb-ehhootEHHOO7L&L M-"]#u||+,]69]IN]&Dehhoo1E,F4PSUXPX>',+/        %,,# $  ( 6S$  koEHHOO%667AFsELLGXAYdhB8 LFLFr#