+hMSSKJr SSKJr SSKJrJrJrJrJ r J r SSK r SSK r SSK J r Jr SSKJrJr SSKJrJr SS KJr S S KJrJrJrJrJrJrJr S S KJ r J!r! S S K"J#r# S SK$J%r%J&r&J'r'J(r(J)r)J*r*J+r+ S SK,J-r- SSK.J/r/ \R`"\15r2\"SS\55r3"SS\Rh5r5"SS\Rh5r6"SS\Rh5r7S;S\8S\8S\8S\8S\8S\\8S \\ \8\ \84S!\\8S"\\8S#\9S$\\9S%\\94S&jjr:S"S2S3\Rh5r?"S4S5\Rh5r@"S6S7\Rh5rAS=S8jrBS9rCS:rDg)>) dataclass)gcd)AnyDictListOptionalTupleUnionN)Tensornn) ConfigMixinregister_to_config) BaseOutputlogging) apply_freeu)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)CrossAttnDownBlock2DCrossAttnUpBlock2D Downsample2D ResnetBlock2DTransformer2DModelUNetMidBlock2DCrossAttn Upsample2D)UNet2DConditionModel)ControlNetConditioningEmbeddingc(\rSrSr%SrSr\\S'Srg)ControlNetXSOutput4a% The output of [`UNetControlNetXSModel`]. Args: sample (`Tensor` of shape `(batch_size, num_channels, height, width)`): The output of the `UNetControlNetXSModel`. Unlike `ControlNetOutput` this is NOT to be added to the base model output, but is already the final output. Nsample) __name__ __module__ __qualname____firstlineno____doc__r+r __annotations____static_attributes__r,d/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/controlnets/controlnet_xs.pyr)r)4sFFr4r)c ^\rSrSrSrS S\R S\R S\R S\\R S\\R4 U4Sjjjr S r U=r $) DownBlockControlNetXSAdapterBzyComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnDownBlock2D`resnets base_to_ctrl ctrl_to_base attentions downsamplerc^>[TU]5 XlX lX0lX@lXPlgN)super__init__r9r:r;r< downsamplers)selfr9r:r;r<r= __class__s r5rA%DownBlockControlNetXSAdapter.__init__Fs,  (($'r4)r<r:r;rBr9)NN) r-r.r/r0r1r ModuleListrConv2drAr3 __classcell__rDs@r5r7r7Bsh*/3+/ ( (mm (mm ( R]]+ ( bii( ( (r4r7cd^\rSrSrSrS\S\RS\R4U4SjjrSr U=r $)MidBlockControlNetXSAdapterVzxComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnMidBlock2D`midblockr:r;cF>[TU]5 XlX lX0lgr?)r@rArMr:r;)rCrMr:r;rDs r5rA$MidBlockControlNetXSAdapter.__init__Zs  ((r4)r:r;rM) r-r.r/r0r1r#r rFrAr3rHrIs@r5rKrKVs3))!8) )egerer))r4rKcH^\rSrSrSrS\R 4U4SjjrSrU=r $)UpBlockControlNetXSAdapterazwComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnUpBlock2D`r;c.>[TU]5 Xlgr?)r@rAr;)rCr;rDs r5rA#UpBlockControlNetXSAdapter.__init__ds (r4r;) r-r.r/r0r1r rFrAr3rHrIs@r5rQrQasB)R]]))r4rQbase_in_channelsbase_out_channelsctrl_in_channelsctrl_out_channels temb_channelsmax_norm_num_groupstransformer_layers_per_blocknum_attention_headscross_attention_dimadd_downsampleupcast_attentionuse_linear_projectionc FSn /n/n/n/n[U[5(aU/U -n[U 5HnUS:XaUOUnUS:XaUOUnUR[ X55 UR[ X -UU[ X -US9[ X5S9SS95 U(a+UR[UX8-UUUU U U [ X5S9S95 UR[ X155 M U (aCUR[ X55 [X1-SUSS 9nUR[ X155 OSn[[R"U5[R"U5[R"U5S 9nU(a[R"U5Ul UbUUl U$) Nrr max_factorh㈵> in_channels out_channelsrZgroups groups_outepsrg num_layersr^rar`norm_num_groupsTopuse_convrhname)r9r:r;) isinstanceintrangeappendmake_zero_convr!find_largest_factorr"r r7r rFr<rB)rVrWrXrYrZr[ has_crossattnr\r]r^r_r`rarmr9r<r;r:irBdown_block_componentss r5get_down_block_adapterr|isJGJLL.44(D'E 'R$ : /0Av+;L/0Av+;L N+;NO ,?.+*+;+N[no./@a       "'%< 1;A>(;*?%5$78I$j   N+<PQGJ N+<PQ#  1DO`gk N+<PQ 8 g&]]<0]]<0 +-==+D(-9*  r4 base_channels ctrl_channelsc [X5n [UX-UU[[XU-5U5UUUUS9 n [X5n [ XU S9$)N r\rgrhrZ resnet_groupsr^r]rar`)r:rMr;)rwr#rxrrK) r}r~rZr[r\r]r^r`rar:rMr;s r5get_mid_block_adapterrsb"-?L&%A!1"#)#m]=Z*[]pq//3) H"-?L &Lbn oor4rhprev_output_channelctrl_skip_channelsc/nSn[U5H*nUS:XaUOUnUR[X%U55 M, [[R "U5S9$)Nr rrU)rurvrwrQr rF)rhrrr;rmrzresnet_in_channelss r5get_up_block_adapterrs\ LJ : 45F0 N+=+@BTUV &2==3N OOr4c L^\rSrSrSr\SS\S\S\\S\ S\ S\ \\\4S \\S \\S \S \\S \ \S\ \\\4S\ S\S\ 4U4Sjjj5r \SS\S\ \ S \ \\S\ \\S\ S\S\S\S\\4Sjj5rSrSrU=r$)ControlNetXSAdaptera! A `ControlNetXSAdapter` model. To use it, pass it into a `UNetControlNetXSModel` (together with a `UNet2DConditionModel` base model). This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Like `UNetControlNetXSModel`, `ControlNetXSAdapter` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. Parameters: conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channels for each block in the `controlnet_cond_embedding` layer. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapters's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. learn_time_embedding (`bool`, defaults to `False`): Whether a time embedding should be learned. If yes, `UNetControlNetXSModel` will combine the time embeddings of the base model and the control adapter. If no, `UNetControlNetXSModel` will use the base model's time embedding. num_attention_heads (`list[int]`, defaults to `[4]`): The number of attention heads. block_out_channels (`list[int]`, defaults to `[4, 8, 16, 16]`): The tuple of output channels for each block. base_block_out_channels (`list[int]`, defaults to `[320, 640, 1280, 1280]`): The tuple of output channels for each block in the base unet. cross_attention_dim (`int`, defaults to 1024): The dimension of the cross attention features. down_block_types (`list[str]`, defaults to `["CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"]`): The tuple of downsample blocks to use. sample_size (`int`, defaults to 96): Height and width of input/output sample. transformer_layers_per_block (`Union[int, Tuple[int]]`, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. upcast_attention (`bool`, defaults to `True`): Whether the attention computation should always be upcasted. max_norm_num_groups (`int`, defaults to 32): Maximum number of groups in group normal. The actual number will be the largest divisor of the respective channels, that is <= max_norm_num_groups. conditioning_channelsconditioning_channel_order#conditioning_embedding_out_channelstime_embedding_mixlearn_time_embeddingr]block_out_channelsbase_block_out_channelsr^down_block_types sample_sizer\r`r[rac2>[T!U]5 USnUSS-nUS;a[SU35e[U5[U 5:wa[SUSU S35e[ U [ [ 45(dU /[U 5-n [ U [ [ 45(dU /[U 5-n [ U[ [ 45(dU/[U 5-n[U5[U 5:wa[SUSU S35e[USUUS 9UlU(a[UU5Ul OSUl [R"/5Ul [R"/5Ul[R"SUSS S S 9Ul[#USUS5UlUSnUSn['U 5HfunnUnUUnUnUUnS U;nU[U 5S - :HnURR)[+UUUUUUUU UUUU UU(+U US9 5 Mh [-USUSUU SUSU SU US9UlUS/n['U5H1unnU[U5S - :aS OSnUR1U/U-5 M3 [ [3U55nUSn[5[U 55HWnUnUUn[5S 5Vs/sHnUR75PM n nURR)[9UUU S95 MY gs snf)Nr)rgbbgrz&unknown `conditioning_channel_order`: zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: conditioning_embedding_channelsrrr r& kernel_sizepadding CrossAttn) rVrWrXrYrZr[ryr\r]r^r_r`ra)r}r~rZr\r]r^r`rar)rhrr)r@rA ValueErrorlenrslisttupler'controlnet_cond_embeddingrtime_embeddingr rF down_blocksup_connectionsrGconv_inrwcontrol_to_base_for_conv_in enumeratervr|r mid_blockextendreversedrupopr)"rCrrrrrr]rrr^rrr\r`r[ratime_embedding_input_dimtime_embedding_dimrWrYrzdown_block_typerVrXryis_final_blockrrhnumber_of_subblocks reversed_base_block_out_channelsprev_base_output_channel_ctrl_skip_channels_rDs" r5rAControlNetXSAdapter.__init__$s0 #:1#= 4Q7!; &^ ;EF`Eabc c ! "c*:&; ;tvHuII_`p_qqrs 6u FF,H+ICP`La+a (-e}==#6"7#>N:O"O -e}==#6"7#>N:O"O  " #s+;'< <vxKwLLbcsbttuv  *I,>q,AB"7* & "34LN`"aD "&D ==, mmB/yy$6q$9qRST +9:LQ:OQhijQk+l(4A6.q1"+,<"= A0  7 : 0  21 5 '?:M#&6"7!";;N    # #&%5&7%5&7"4(;"/1Ma1P(;A(>(;A(>'5#5%5*? #>6/1"5,R0,)Eb)I 3B 7 3B 7-"7  134();S!jr\S \\ \44S"j5rS#\ \\\ \444S$jrS%rS&\ S'\ S(\ S)\ 4S*jrS+rS,rS-rS?S.\S/\ \ R>\ \4S0\ R>S1\\ R>S2\\ S3\\ R>S4\\ R>S5\\ R>S6\\\ \!4S7\\\ \ R>4S8\ S9\ S \ \"\ 44S:jjr#S;r$U=r%$)@UNetControlNetXSModelia A UNet fused with a ControlNet-XS adapter model This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). `UNetControlNetXSModel` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. It's parameters are either passed to the underlying `UNet2DConditionModel` or used exactly like in `ControlNetXSAdapter` . See their documentation for details. Trrup_block_typesrrnr^r\r]addition_embed_typeaddition_time_embed_dimr`ratime_cond_proj_dim%projection_class_embeddings_input_dimrctrl_conditioning_channels(ctrl_conditioning_embedding_out_channelsctrl_conditioning_channel_orderctrl_learn_time_embeddingctrl_block_out_channelsctrl_num_attention_headsctrl_max_norm_num_groupsc>[T0U]5 US:dUS:a [S5eUS:aU(d [S5eU bU S:wa [S5e[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-n[U[[ 45(dU/[ U5-nUnSUl[R"SUSSSS 9Ul [USUUS 9Ul [R"SUSSSS 9Ul [USUS5UlUSnUSS-n[!USS SS 9Ul[%UUU S 9UlU(a[%UUS9UlOSUlU cSUlSUlO![!U S SS 9Ul[%UU5Ul/nUSnUSn[/U5HaunnUnUUnUn UUnSU;n!U[ U5S- :Hn"UR1[3UUU UUUUU!UUUUUUUUU"(+U U S95 Mc [5USUSUUUUSUSUSUSU U S9 Ul/n#[ [9U55n$[ [9U55n%[ [9U55n&US/n'[/U5H1unn(U[ U5S- :aSOSn)U'R;U(/U)-5 M3 [ [9U55n*U*Sn([/U5Hunn+U(n,U*Un(U*[=US-[ U5S- 5n-[?S5V.s/sHn.U'RA5PM n/n.SU+;n!U[ U5S- :Hn"U#R1[CU-U(U,U/UUU!U$UU%UU&UU"(+U UU S95 M [RD"U5Ul#[RD"U#5Ul$[RJ"USUS9Ul&[RN"5Ul([R"USSSSS 9Ul)gs sn.f)Nrr&z1`time_embedding_mix` needs to be between 0 and 1.zKTo use `time_embedding_mix` < 1, `ctrl_learn_time_embedding` must be `True` text_timezAs `UNetControlNetXSModel` currently only supports StableDiffusion and StableDiffusion-XL, `addition_embed_type` must be `None` or `'text_time'`.rr rrT)flip_sin_to_cosdownscale_freq_shift) cond_proj_dim)rgtime_embed_dimrrVrWrXrYrZrnrryr\base_num_attention_headsrr^r_r`rar r}r~rZrnrr\rrr^r`rar)rgrhrrrZresolution_idxryr\r]r^ add_upsampler`rnra) num_channels num_groups)*r@rArrsrrrrgr rG base_conv_inr'r ctrl_conv_inrwrrbase_time_projrbase_time_embeddingctrl_time_embeddingbase_add_time_projbase_add_embeddingrrv ControlNetXSCrossAttnDownBlock2DControlNetXSCrossAttnMidBlock2DrrrminrurControlNetXSCrossAttnUpBlock2DrFr up_blocks GroupNormbase_conv_norm_outSiLU base_conv_act base_conv_out)1rCrrrrrnr^r\r]rrr`rarrrrrrrrrrrtime_embed_input_dimrrrWrYrzrrVrXryrr rev_transformer_layers_per_blockrev_num_attention_headsrev_cross_attention_dimrrhrreversed_block_out_channels up_block_typerrgrrrDs1 r5rAUNetControlNetXSModel.__init__s?B   !%7!%;PQ Q  !*Cjk k  */Bk/Qd 6u FF,H+ICP`La+a (-e}==#6"7#>N:O"O -e}==#6"7#>N:O"O 2T5MBB(@'ACHXDY'Y $#6 IIa);A)>AWXY)H,CA,FG"<* & IIa)@)CQR\]^+9:QRS:TVhijVk+l( 2!4+A.2'(:1(=tjkl#4 ,$  %'80(D $(,D $  &&*D #&*D #&/0GY]tu&vD #&78]_m&nD # .q13A6"+,<"= A0  21 5 0  7 : '?:M#&6"7!";;N   0%5&7%5&7"0$3-E"/1Ma1P-Ea-H-Ea-H(;A(>'5#5%5*? #>:9,R01"5(+%=)Eb)I%=b%A%=b%A 3B 7-"7   +/9U0V+W("&x0H'I"J"&x0C'D"E6a89()@AOA|4599q   % %|n7J&J K B '+84F+G&H#215 ). 9 A}". 6q9L5c!a%EWAX[\A\6]^KEJ1X"NX#5#9#9#;X "N'=8M#&8"9A"==N   . +!-(;':"0#$"/1QRS1T(?(B(?(B%3!3%5$3*? !:8==5y1"$,,2UNetControlNetXSModel.from_unet..s'v! 'vszWhen a controlnet is passed, none of these parameters should be passed: size_ratio, ctrl_block_out_channels, time_embedding_mix, ctrl_optional_kwargs.) rrrrrnr^r\rrr`rarrr])rrrrrr]r[ctrl_r)rr conv_norm_outconv_outbase_ add_time_proj add_embeddingc3P# UHup[RX5v M g7fr?)r from_modulesrrcs r5rr7s'* E - 9 9! ? ?E$&c3P# UHup[RX5v M g7fr?)rrr s r5rr<s'( F + 7 7 = =Fr")rranyrritemsrr from_configgetattrload_state_dict state_dicthasattrrrrrrrr rFziprrrrrrrr)rrrrrrrparams_for_unetkvparams_for_controlnetrmodules_from_unetmoptional_modules_from_unets r5rUNetControlNetXSModel.from_unets<  ,66"9=QJ(2M_'v!m  -1KK,=,=,?X,?DA1CW414,?X151O1O-.! =G[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processorr)r*rEnamed_children)rrrBrCsub_namechildfn_recursive_add_processorss r5rJJUNetControlNetXSModel.attn_processors..fn_recursive_add_processorsnsZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; r4)rtorchr ModulerrrG)rCrCrrrBrJs @r5attn_processors%UNetControlNetXSModel.attn_processorscsb  c 588?? X\]`bt]tXu !//1LD 'j A2r4 processorc d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrrBc >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorrFr)r*rsdictrSrrG)rrrBrPrHrIfn_recursive_attn_processors r5rUMUNetControlNetXSModel.set_attn_processor..fn_recursive_attn_processorssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r4N) rrNkeysrsrTrrrLr rMrG)rCrPcountrrrBrUs @r5set_attn_processor(UNetControlNetXSModel.set_attn_processor}sD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r4c ~[SURR555(a [5nOr[SURR555(a [ 5nO8[ S[ [URR55535eURU5 g)zU Disables custom attention processors and sets the default attention implementation. c3F# UHoR[;v M g7fr?)rDrrprocs r5rCUNetControlNetXSModel.set_default_attn_processor..siKh4~~!>>Kh!c3F# UHoR[;v M g7fr?)rDrr]s r5rr_shJg$#==Jgr`zOCannot call `set_default_attn_processor` when attention processors are of type N) allrNvaluesrrrnextiterrY)rCrPs r5set_default_attn_processor0UNetControlNetXSModel.set_default_attn_processors i4K_K_KfKfKhi i i,.I h$J^J^JeJeJgh h h%IabfgklpmAmAmHmHmJhKcLbMN   *r4s1s2b1b2c[UR5H9upV[USU5 [USU5 [USU5 [USU5 M; g)aEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. rhrirjrkN)rrsetattr)rCrhrirjrkrzupsample_blocks r5 enable_freeu"UNetControlNetXSModel.enable_freeusJ$"+4>>!: A ND" - ND" - ND" - ND" - ";r4c1Skn[UR5H9up#UH.n[X45(d[X4S5cM"[ X4S5 M0 M; g)zDisables the FreeU mechanism.>rjrkrhriN)rrr*r'rm)rC freeu_keysrzrnr-s r5 disable_freeu#UNetControlNetXSModel.disable_freeusH- !*4>>!: A>--D1Q1]Nt4 ";r4cSUlURR5H3upS[URR 5;dM*[ S5e URUlUR5H)n[U[5(dMURSS9 M+ UR[55 g)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsrNr%rrDr-rmodulesrsrfuse_projectionsrYr)rCrattn_processorrBs r5fuse_qkv_projections*UNetControlNetXSModel.fuse_qkv_projectionss)-%!%!5!5!;!;!= A#n66??@@ !tuu">)-(<(<%llnF&),,''T'2%  5 78r4cVURbURUR5 gg)umDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rxrY)rCs r5unfuse_qkv_projections,UNetControlNetXSModel.unfuse_qkv_projectionss)  ( ( 4  # #D$A$A B 5r4r+timestepencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskcross_attention_kwargsadded_cond_kwargs return_dict apply_controlc  URRS:Xa[R"US/S9nUb2SUR UR 5- S-nUR S5nUn [R"U 5(dURRS:HnURRS:Hn[U[5(a/U(dU(a[RO[RnO.U(dU(a[RO[Rn[R "U /UURS9n O7[#U R$5S :XaU SR UR5n U R'UR$S 5n UR)U 5nUR UR S 9nURR*(aSU (aLUR-UU5nUR/UU5nURR0S -nUU-USU- --nOUR/U5nSnURR2cGOURR2S :XaS U ;a[5UR6S35eU R9S 5nSU ;a[5UR6S35eU R9S5nUR;UR=55nUR?UR$S S45n[R@"UU/SS9nUR UR 5nURCU5nO#[5SURR2S35eUbUU-OUnUnU=nn//nnUREU5n URGU5nURIU5nU bUU - nU (aUURKU5U--nURMU5 URMU5 URNH7n!U!"UUUUUU UU S9unnn"n#URQU"5 URQU#5 M9 URSUUUUUU UU S9unnURTH>n$[#U$RV5n%UU%*Sn&UU%*Sn'USU%*nUSU%*nU$"UU&U'UUUU UU S9 nM@ URYU5nUR[U5nUR]U5nU (dU4$[_US9$)au The [`ControlNetXSModel`] forward method. Args: sample (`Tensor`): The noisy input tensor. timestep (`Union[torch.Tensor, float, int]`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): How much the control model affects the base model outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnets.controlnet.ControlNetOutput`] instead of a plain tuple. apply_control (`bool`, defaults to `True`): If `False`, the input is run only through the base model. Returns: [`~models.controlnetxs.ControlNetXSOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnetxs.ControlNetXSOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rr&)dimsNgmpsnpu)rdevicer)rg333333?r text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rdimzgControlNet-XS currently only supports StableDiffusion and StableDiffusion-XL, so addition_embed_type = z is currently not supported.)hidden_states_basehidden_states_ctrltembrrrrr) hidden_statesres_hidden_states_tuple_baseres_hidden_states_tuple_ctrlrrrrrr)r+)0rrrLfliprr unsqueeze is_tensorrtypersrfloat32float64int32int64tensorrshapeexpandrrrrrrrrDgetrflattenreshapeconcatrrrrrrvrrrrr9rrrr))(rCr+rrrrrrrrrrr timestepsis_mpsis_npurt_emb ctrl_temb base_tembinterpolation_paramraug_embrr time_embeds add_embedscembh_ctrlh_basehs_basehs_ctrl guided_hintdown residual_hb residual_hcup n_resnetsskips_hbskips_hcs( r5rUNetControlNetXSModel.forwardsp ;; 6 6% ?#jjsCO  %."3"3FLL"AAXMN+55a8N y))]]''50F]]''50F(E***0F (.& u{{ i[fmmTI  !Q &!$**6==9I$$V\\!_5 ##I. v||, ;; 0 0]00 FI00 FI"&++"@"@#"E 22Y!FYBY5ZZD++E2D ;; * * 2  [[ , , ;$55 ~~&'{|,// >K!22 ~~&'xy),,Z8H11(2B2B2DEK%--{/@/@/CR.HIK{K&@bIJ#tzz2J--j9Gyz~{F{F{Z{Zz[[wx ")!4tg~$%! r44_E ""6*""6*  " k !F d>>vFI[[[Fvv$$D7;#)#)&*#5'=-+ 8 4FFK NN; ' NN; '%%%"&1#9)'(  ..BBJJI z{+H z{+Hk z*Gk z*G$-5-5&*#5'=-+ F !&((0##F+##F+9 !00r4)rrrrrrrrrrrrrrgrrxr)rr) UpBlock2Drrrrrrr&rNNTTNNrr rrFrrr)NNNNNr4N) NrNNNNNTT)&r-r.r/r0r1 _supports_gradient_checkpointingrrrtr rr rrrArr%rrrrr?propertyrrNrYrfrorsr|rr rLrr)rr3rHrIs@r5rrs (,$&(( &u)?)+6:?@67-115!%&*,0?C$'*+?P/4*/.<;<(*=|[c]|[ * |[c |["#J|["#|[#3c ?3|[',CsO&<|[#3c ?3|[ &c]!|[""*##|[$%|[& $'|[(%SM)|[*08}+|[."/|[0%(1|[238*3|[4*-5|[6$(7|[8"'s9|[:#(U3Z"8;|[<#&=|[|[|59&*9=.2/3t"t01tUO t "*$u+!6 t %UO t'tnttl#:c+=&=!>0 AE2Dd3PbKbFc2c,d AF + .u.%.U..2594 C$37.1/30415;??C "J1J1 eS01J1 %|| J1 "%,,/ J1 %UO J1u||,J1  -J1!.J1!)c3h 8J1$Dell):$;<J1J1J1 !5( )J1J1r4rcj^\rSrSrS"S\S\S\S\S\S\S\S \\\\\4S \\S \\S \\S \S\\S\\4U4Sjjjr \ S\ S\ 4Sj5r S#SjrS$S\S\S\\S\\S\\S\\S\\\\4S\\S\S\\\\\S4\\S444S jjrS!rU=r$)%rirVrWrXrYrZrnrr\rrr^r_r`rac>[TU]5 /n/n/n/n/n/nSn[U [5(aU /U-n [ U5HnUS:XaUOUnUS:XaUOUnUR [ X55 UR [UUUUS95 UR [X1-UU[X1-US9[XGS9SS95 U(aOUR [U X*-UU UU UUUS95 UR [U XK-UU UU UU[XGS9S95 UR [ XB55 M U (aYUR [ X"55 [USUS S 9Ul [XB-SUS S 9Ul UR [ XB55 OSUl SUl [R"U5Ul[R"U5UlU(a[R"U5OS/U-UlU(a[R"U5OS/U-Ul[R"U5Ul[R"U5UlS Ulg) NrrrgrhrZrircrerfrlTrorpF)r@rArsrtrurvrwr!rxr"r base_downsamplersctrl_downsamplersr rF base_resnets ctrl_resnetsbase_attentionsctrl_attentionsr:r;gradient_checkpointing)rCrVrWrXrYrZrnrryr\rrr^r_r`rarrrrr;r:rmrzrDs r5rA)ControlNetXSCrossAttnDownBlock2D.__init__s$      2C 8 8,H+IJ+V (z"A346/?P 346/?P    /? R S    0!2"/*      0 C!2"/.(;H` 33Dj  &&&0)E$5#?#B,?.C)9(7   &&&0)E$5#?#B,?.C)9(;.get_first_cross_attention@$##A&99!<BB Br4rr<TFr)r9rgrh time_emb_proj in_featuresnorm1rr*rr<rheadsr^r`rarBrr(r)rrrrrr:r;)rrrrrVrWrXrYrZrctrl_num_groupsryr\rrr^r`rar_rs r5r-ControlNetXSCrossAttnDownBlock2D.from_modules=s C*11!4@@*2215BB  " "1 % 1 14D D +2215BB&..q1??KK #++A.44?? (00399DD >< 0 0 M+.~/H/H/K/^/^+_ ('@'P'V'V $'@'P'V'V $";N"K"_"_ 8HYY $2$=$=a$@$V$V !!M+/ ('+ $'+ $"& # $( !'44D@-/-/'&%4')E%=%= 3)-"7 & **>+A+A+L+L+NO **>+A+A+L+L+NO   ! ! 1 1.2K2K2V2V2X Y  ! ! 1 1.2K2K2V2V2X Y   # # 3 3N4O4OPQ4R4]4]4_ `  # # 3 3N4O4O4Z4Z4\ ] **>+F+F+Q+Q+ST **>+F+F+Q+Q+ST r4r4cUR5H nSUlM UR/n[UR[ R 5(aURUR5 URbURUR5 UH!nUR5H nSUlM M# gr6TNF) r7r8rrsrr rFrvrrCr:r;r<s r5r93ControlNetXSCrossAttnDownBlock2D.freeze_base_paramss__&E"&E '''( d**BMM : :   d22 3  ! ! -   d44 5D*&+#+r4rrrrrrrencoder_attention_maskr.c Ub(URSS5b[RS5 Un Un Sn Sn [[ UR UR 55n[[ URUR55n[ XURUR5GHuunnunnnnU (a[R"U U"U 5/SS9n [R"5(a$UR(aURUX5n OU"X5n UbU"U UUUUSS9Sn U (aW[R"5(a$UR(aURUX5n OU"X5n UbU"U UUUUSS9Sn U (aU U"U 5U--n X4-n X4-n GM UR bURS nURS nU (a[R"U U"U 5/SS9n UR!U 5n U (aUR#U 5n U (aU U"U 5U--n X4-n X4-n XX4$) NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r,r&rFrrrrrrr)rloggerwarningrr+rrrrr:r;rLcatis_grad_enabledr_gradient_checkpointing_funcrr)rCrrrrrrrrrrrbase_output_statesctrl_output_states base_blocks ctrl_blocksb_resb_attnc_resc_attnb2cc2bs r5r(ControlNetXSCrossAttnDownBlock2D.forwardsi " -%))'48Dtu##3t00$2F2FGH 3t00$2F2FGH := d&7&79J9J; 6OUF_eVc3FCK#8a@$$&&4+F+F::5&Ov,!*?+A#1+A % ((**t/J/J!>>ufSF"60F%#.C/E'5/E$) F#f+0B"BB!3i!? !3i!? Y; \  ! ! -##B'C##B'CFCK#8a@++F3F//7#f+0B"BB!3i!? !3i!? 1EEr4) rrrr:rrrr;r) rrTr&r&r&rTFTr)NNrNNNT)r-r.r/r0rtrr r rrArrr7rr9r rrrrrr3rHrIs@r5rrs "(*IJ2323-1#+004!t,t,t, t,  t,  t,t,#&t,'/uS%*_/E&Ft,#+3-t,#+3-t,&c]t,t,#4.t, (~!t,t,l?*>?Pl??B,*37/3.1+/;?37"ZF"ZFZF (/ ZF %V, ZF %UO ZF!(ZF!)c3h 8ZF!) 0ZFZF vvuVS[153EE FZFZFr4rc&^\rSrSrSS\S\S\\S\S\S\S\\S \\S \\S \S \\4U4S jjjr\S\ S\ 4Sj5r SSjr S S\ S\ S\ S\\ S\\S\\\\4S\\ S\\ S\S\\ \ 44SjjrSrU=r$)!rir}r~rZrnrr\rrr^r`rac >[T U]5 [X5Ul[ UUUUU UU U S9Ul[ UX!-UU[ [X"U-5U5U UU U S9 Ul[X!5Ul SUl g)N)r\rgrZrr^r]rar`rF) r@rArwr:r# base_midblockrxr ctrl_midblockr;r) rCr}r~rZrnrr\rrr^r`rarDs r5rA(ControlNetXSCrossAttnMidBlock2D.__init__s +=H4)E%') 3 8"7-  5)E%5&'-M=#@AC[!4 8"7-  "+=H&+#r4rrcURnURnURnSnURnURn[ UR SR5nURSRRn URSRRn URSRRn U"U5Rn U"U5Rn U"U5RnU"U5RnUR SR nU"UUU U U UU U UUUS9 nURR#UR%55 UR&R#UR%55 UR(R#UR%55 URR#UR%55 U$)NcNURSRSR$rr)rMs r5rOControlNetXSCrossAttnMidBlock2D.from_modules..get_first_cross_attention-s$&&q)<>JJ "**1-33>> '//288CC#<]#K#Q#Q #<]#K#Q#Q 7 FZZ4]CTT - 8 8 ; Q Q'''&%4)E%=%= 3-"7   **<+B+B+DE ++M,D,D,FG ++M,D,D,FG **<+B+B+DE r4r4cUR5H nSUlM URR5H nSUlM g)r6TFN)r7r8r)rCr:s r5r92ControlNetXSCrossAttnMidBlock2D.freeze_base_paramsSs@__&E"&E '''224E"'E 5r4rrrrrrrrrc TUb(URSS5b[RS5 Un Un UUUUUS.n U (a%[R"XR U 5/SS9n UR "U 40U D6n U (a)UR"U 40U D6n XRU 5U--n X4$)Nrr)rrrrrr&r) rrrrLrr:rrr;) rCrrrrrrrrrrr joint_argss r5r'ControlNetXSCrossAttnMidBlock2D.forward^s " -%))'48Dtu##%:,&<&<   YY(9(9&(ABJF##F9j9 ''=*=F//7:LLLF~r4)rr:rr;r) Nrrr&r&r&rFTr)NrNNNT)r-r.r/r0rtrrrArr#rKrr9r rrrrr rr3rHrIs@r5rrs (,!(*,-2323-1!&042,2,2, } 2,  2, #& 2,'*2,#+3-2,#+3-2,&c]2,2, (~2,2,h...3..` ( 04.1;?+/37""""" & " %V, " %UO "!)c3h 8"!("!) 0"" vv~ ""r4rc<^\rSrSrS#S\S\S\S\\S\S\S\\S \S \S \S \S \S\\4U4Sjjjr\ S\ S\ 4Sj5r S$Sjr S%S\S\\S4S\\S4S\S\\S\\S\\\\4S\\S\\S\\S \S\4S!jjrS"rU=r$)&rirgrhrrrZrnrr\r]r^rr`rac>[TU]5 /n/n/nSnXlXl[ U [ 5(aU /U-n [ U5HnUUS- :XaUOUnUS:XaUOUnUR[UUU55 UR[UU-UUUS95 U(dM_UR[U X*-UU UU UU US95 M [R"U5Ul U(a[R"U5OS/U-Ul[R"U5UlU (a[!USUS9UlOSUlSUlXplg) Nr r&rrrlT)rqrhF)r@rAhas_cross_attentionr]rsrtrurvrwr!r"r rFr9r<r;r$ upsamplersrr)rCrgrhrrrZrnrryr\r]r^rr`rar9r<r;rmrzres_skip_channelsrrDs r5rA'ControlNetXSCrossAttnUpBlock2D.__init__sd"    #0 #6 2C 8 8,H+IJ+V (z"A01Z!^0C , 89Q!4L    /A!/DFX Y Z NN 25F F!-"/*  }!!&+$;$0#?#B,?.C)9(7  !#:}}W- 7D"-- 34&S]J]MM,7 (S_`DO"DO&+#,r4 base_upblock ctrl_upblockcURnSnURSRnURSRU- nURSRU- nUVs/sHoRPM n nURSRR n URSR Rn URn [US5(atSn [URSR5nU"U5RnU"U5RnU"U5RnURSR nO Sn SnSnSnSnSnUR"SLnU"UUUU U U U U UUUUUUS9nURR%URR'55 U (a3URR%URR'55 U(a6UR"R%UR"SR'55 URR%UR'55 U$s snf)NcNURSRSR$rrrs r5rNControlNetXSCrossAttnUpBlock2D.from_modules..get_first_cross_attentionrr4rrr<TF)rgrhrrrZrnrryr\r]r^rr`ra)r;r9rhrgrrrrrr*rr<rrr^r`rarr(r))rrrctrl_to_base_skip_connectionsrrhrgprev_output_channelsr!ctrl_skip_channelssrZrrryr\r]r^r`rarrs r5r+ControlNetXSCrossAttnUpBlock2D.from_moduless>(4(A(A% C$++A.;; "**2.::\I +33A6BB\Q6ST6S}}6ST$,,Q/==II !))!,22== %44 < . . M+.|/F/Fq/I/\/\+] (";L"I"O"O ";L"I"]"] 8FWW $0$;$;A$>$T$T !!M+/ ("& "& # $( !#..d: #% 42'&)')E 3 3%-"7 $  %%l&:&:&E&E&GH     , ,\-D-D-O-O-Q R     , ,\-D-DQ-G-R-R-T U **+H+S+S+UV ]Us%Ir4cUR5H nSUlM UR/n[UR[ R 5(aURUR5 URbURUR5 UH!nUR5H nSUlM M# gr) r7r8r9rsr<r rFrvrrs r5r91ControlNetXSCrossAttnUpBlock2D.freeze_base_paramss__&E"&E 'll^ door}} 5 5   doo . ?? &   doo .D*&+#+r4rr.rrrrrr upsample_sizerrc ^^Ub(URSS5b[RS5 [TSS5=(a5 [TSS5=(a! [TSS5=(a [TSS5mUU4Sjn [ TR TR TR[U5[U55HupnnnU (a X"U5U-- nU "UU5unn[R"UU/SS 9n[R"5(a$TR(aTRXU5nOU "X5nUcMU"UUUUU S S 9S nM TRbTRX5nU$) Nrrrhrirjrkc >T(aA[TRUUTRTRTRTR S9$X4$)N)rhrirjrk)rrrhrirjrk)r res_h_baseis_freeu_enabledrCs r5maybe_apply_freeu_to_subblockMControlNetXSCrossAttnUpBlock2D.forward..maybe_apply_freeu_to_subblock0sI"''!wwwwwwww%00r4r&rFrr)rrrr'r+r9r<r;rrLrrrrr)rCrrrrrrrrrrrr!resnetattnrr res_h_ctrlr s` @r5r&ControlNetXSCrossAttnUpBlock2D.forwards{ " -%))'48Dtu D$ % *dD) *dD) *dD)  1:= LL OO    1 2 1 2 : 5F#z:Z3E!EE (EmU_(` %M:!II}j&AqIM$$&&4+F+F $ A A&Y] ^ &} ;  $!*?+A#1+A % !! ': 8 ?? & OOMIMr4)r<r;rr r]r9rr) rNTr&r&rTFTr)NrNNNNT)r-r.r/r0rtrrrrArrrQrr9r r rrrrrr3rHrIs@r5rrs "(,,-#$#'!!&04E-E-E-! E- !I E-  E-E-! E-'*E-!E-!E-E-E- (~E-E-N8(:8Jd88t,.37.1;?+/'+37"GG',FCK&8G',FCK&8 G  G (/ G%UOG!)c3h 8G!(G }G!) 0GG GGr4rc @[[R"XSSS95$)Nr&r)r) zero_moduler rG)rgrhs r5rwrwas ryyAqI JJr4crUR5H"n[RRU5 M$ U$r?)r7r initzeros_)rBps r5r(r(es*     q! Mr4cVUnX :aU$US:waX-nUS:XaU$US-nUS:waMgg)Nrr&r,)numberrdfactorresiduals r5rxrxks> F  A+? q=M!  A+r4)rTr&r&rTFT)Nrr&r&rFTr?)E dataclassesrmathrtypingrrrrr r rLtorch.utils.checkpointr r configuration_utilsrrutilsrrutils.torch_utilsrattention_processorrrrrrrr embeddingsrrmodeling_utilsrunets.unet_2d_blocksrrr r!r"r#r$unets.unet_2d_conditionr%rr' get_loggerr-rr)rMr7rKrQrtrr|rrrrrrrrwr(rxr,r4r5r>s":: B(,6';7   H %     (299(()")))))*,EF)*)-',,0X!X!X!X! X!  X! "# X!#+5eCj+A"BX!"#X!"#X!X!tnX!$D>X!|$()+())*)-""& p p pC= p"# p #& p "# p"# p p  pF P P PS  PH *kH VC 1J C 1LdFryydFN SbiiSl[RYY[|K r4