+hZVSSKJr SSKJrJrJrJrJrJr SSK r SSK J r SSK J r Jr SSKJrJr SSKJrJrJrJr SS KJr SS KJrJrJr SS KJrJr SS K J!r! SS K"J#r# SSK$J%r% SSK&J'r'J(r( \RR"\*5r+\"SS\'55r,"SS\#\ \\5r-"SS\#5r.g)) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)USE_PEFT_BACKENDloggingscale_lora_layersunscale_lora_layers)JointTransformerBlock) AttentionAttentionProcessorFusedJointAttnProcessor2_0)"CombinedTimestepTextProjEmbeddings PatchEmbed)Transformer2DModelOutput) ModelMixin)SD3SingleTransformerBlock) BaseOutput zero_modulec:\rSrSr%\\R \S'Srg)SD3ControlNetOutput%controlnet_block_samplesN) __name__ __module__ __qualname____firstlineno__rtorchTensor__annotations____static_attributes__r$e/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/controlnets/controlnet_sd3.pyr!r!%s#ELL11r-r!c$^\rSrSrSrSr\S-S\S\S\S\S \S \S \S \S \S\S\S\S\\S4S\ \ S\ \ S\ S\ 4"U4Sjjj5r S.S\ \S\SS4Sjjr \S\\ \44Sj5rS\\\\ \444SjrSrS rS!r\S/S"j5rS0S#\R4S$\R4S%\S&\R4S'\R4S(\R8S)\ \\ \4S*\ S\\R4\44S+jjrS,r U=r!$)1SD3ControlNetModel*a ControlNet model for [Stable Diffusion 3](https://huggingface.co/papers/2403.03206). Parameters: sample_size (`int`, defaults to `128`): The width/height of the latents. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, defaults to `2`): Patch size to turn the input data into small patches. in_channels (`int`, defaults to `16`): The number of latent channels in the input. num_layers (`int`, defaults to `18`): The number of layers of transformer blocks to use. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. num_attention_heads (`int`, defaults to `18`): The number of heads to use for multi-head attention. joint_attention_dim (`int`, defaults to `4096`): The embedding dimension to use for joint text-image attention. caption_projection_dim (`int`, defaults to `1152`): The embedding dimension of caption embeddings. pooled_projection_dim (`int`, defaults to `2048`): The embedding dimension of pooled text projections. out_channels (`int`, defaults to `16`): The number of latent channels in the output. pos_embed_max_size (`int`, defaults to `96`): The maximum latent height/width of positional embeddings. extra_conditioning_channels (`int`, defaults to `0`): The number of extra channels to use for conditioning for patch embedding. dual_attention_layers (`Tuple[int, ...]`, defaults to `()`): The number of dual-stream transformer blocks to use. qk_norm (`str`, *optional*, defaults to `None`): The normalization to use for query and key in the attention layer. If `None`, no normalization is used. pos_embed_type (`str`, defaults to `"sincos"`): The type of positional embedding to use. Choose between `"sincos"` and `None`. use_pos_embed (`bool`, defaults to `True`): Whether to use positional embeddings. force_zeros_for_pooled_projection (`bool`, defaults to `True`): Whether to force zeros for pooled projection embeddings. This is handled in the pipelines by reading the config value of the ControlNet model. TN sample_size patch_size in_channels num_layersattention_head_dimnum_attention_headsjoint_attention_dimcaption_projection_dimpooled_projection_dim out_channelspos_embed_max_sizeextra_conditioning_channelsdual_attention_layers.qk_normpos_embed_type use_pos_embed!force_zeros_for_pooled_projectionc>[TU]5 UnU bU OUUlXe-UlU(a[ UUUUURU US9UlOSUl[ URU S9UlUbo[R"Xx5Ul [R"[U5Vs/sH#n[URUUSUUU ;aSOSS9PM% sn5UlOOSUl [R"[U5Vs/sHn[URUUS9PM sn5Ul[R"/5Ul[[#UR55HTn[R"URUR5n[%U5nUR R'U5 MV [ UUUX<-URSS9n[%U5UlSUlgs snfs snf)N)heightwidthr3r4 embed_dimr<r@) embedding_dimr:FT)dimr7r6context_pre_onlyr?use_dual_attention)rHr7r6)rDrEr3r4rFr@)super__init__r; inner_dimr pos_embedrtime_text_embednnLinearcontext_embedder ModuleListrangertransformer_blocksrcontrolnet_blockslenrappendpos_embed_inputgradient_checkpointing)selfr2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBdefault_out_channelsi_controlnet_blockrY __class__s r.rLSD3ControlNetModel.__init__Ws* *,8,DLJ^,A '"!%'..#5-DN"DNA..@U   *$&II.A$ZD !')mm#:. /* NN,?+=). '348M3M4SX /  'D #%)D !&(mm#:.  / . NN,?+= /  'D #"$r!2s42234A!yyH *+;<   " " ) )*: ;5%!#Ann   +?;&+#S s (*G1G6 chunk_sizerHreturnc^US;a[SU35eU=(d SnS[RRS[S[4U4SjjmUR 5H nT"X1U5 M g) a Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rmodulerbrHc>[US5(aURXS9 UR5H nT"X1U5 M g)Nset_chunk_feed_forward)rbrH)hasattrrgchildren)rerbrHchildfn_recursive_feed_forwards r.rkMSD3ControlNetModel.enable_forward_chunking..fn_recursive_feed_forwards=v788---M*)%SA+r-N) ValueErrorr)rPModuleintri)r[rbrHrerks @r.enable_forward_chunking*SD3ControlNetModel.enable_forward_chunkingsn f LSERS S _1  Behhoo B3 BUX BmmoF %f# >&r-c^0nS[S[RRS[[[ 44U4SjjmUR 5Hup#T"X#U5 M U$)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namere processorsc>[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processor.)rhrvnamed_children)rsrertsub_namerjfn_recursive_add_processorss r.r{GSD3ControlNetModel.attn_processors..fn_recursive_add_processorssZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; r-)strr)rPrnrrry)r[rtrsrer{s @r.attn_processors"SD3ControlNetModel.attn_processorssb  c 588?? X\]`bt]tXu !//1LD 'j A2r- processorc d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rsrec >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorrwrx)rh isinstancedictrpopry)rsrerrzrjfn_recursive_attn_processors r.rJSD3ControlNetModel.set_attn_processor..fn_recursive_attn_processorssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r-N) rWr~keysrrrmr}r)rPrnry)r[rcountrsrers @r.set_attn_processor%SD3ControlNetModel.set_attn_processorsD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r-cSUlURR5H3upS[URR 5;dM*[ S5e URUlUR5H)n[U[5(dMURSS9 M+ UR[55 g)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsr~itemsr}r`r%rmmodulesrrfuse_projectionsrr)r[r^attn_processorres r.fuse_qkv_projections'SD3ControlNetModel.fuse_qkv_projections s)-%!%!5!5!;!;!= A#n66??@@ !tuu">)-(<(<%llnF&),,''T'2%  : <=r-cVURbURUR5 gg)umDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rr)r[s r.unfuse_qkv_projections)SD3ControlNetModel.unfuse_qkv_projections&s)  ( ( 4  # #D$A$A B 5r-c N[URRURRURRURRUR URR S9nURURR5SS9 U$)N)rDrEr3r4rFr<Tstrict) rconfigr2r3r4rMr<load_state_dictrN state_dict)r[ transformerrNs r._get_pos_embed_from_transformer2SD3ControlNetModel._get_pos_embed_from_transformer5s%%11$$00"))44#**66!++*11DD   !!+"7"7"B"B"DT!Rr-cPURnU=(d URUS'X5S'URU5nU(aURR URR 55 UR R UR R 55 URR URR 55 URR URR 5SS9 [UR5Ul U$)Nr5r=Fr) rr5 from_configrNrrrOrRrUrrY)clsrr5num_extra_conditioning_channelsload_weights_from_transformerr controlnets r.from_transformer#SD3ControlNetModel.from_transformerAs##)>V->->|0O,-__V, (  0 01F1F1Q1Q1S T  & & 6 6{7R7R7]7]7_ `  ' ' 7 7 8T8T8_8_8a b  ) ) 9 9+:X:X:c:c:ens 9 t)4Z5O5O)PJ &r- hidden_statescontrolnet_condconditioning_scaleencoder_hidden_statespooled_projectionstimestepjoint_attention_kwargs return_dictc Ub#UR5nURSS5n OSn [(a [X 5 O+Ub(UR SS5b[ R S5 URbURS:wa [S5eURcURS:wa [S5eURbUc [S 5eURcUb [S 5eURbURU5nURXe5n URbURU5nXRU5-nS n URHn [R"5(aHUR (a7URbUR#U UUU 5upAO2UR#XU 5nOURb U "XU S 9upAOU "X5nX4-n M S n [%XR&5HupU"U5nX4-n M U Vs/sHnUU-PM n n[(a [)X 5 U(dU 4$[+U S 9$s snf)a The [`SD3Transformer2DModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.Tensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.Tensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nscale?z\Passing `scale` via `joint_attention_kwargs` when not using the PEFT backend is ineffective.z/hidden_states must be 4D when pos_embed is usedr z3hidden_states must be 3D when pos_embed is not usedzDencoder_hidden_states must be provided when context_embedder is usedzNencoder_hidden_states should not be provided when context_embedder is not usedr$)rrtemb)r#)copyrrrgetloggerwarningrNndimrmrRrOrYrUr)is_grad_enabledrZ_gradient_checkpointing_funcziprVrr!)r[rrrrrrrr lora_scalerblock_res_samplesblockcontrolnet_block_res_samplesblock_res_sampler_samples r.forwardSD3ControlNetModel.forwardTs~N " -%;%@%@%B "/33GSAJJ   d /%16L6P6PQXZ^6_6kr >> %-*<*<*ANO O^^ # (:(:a(?RS S  ,1F1Ncd d  " " */D/Pmn n >> % NN=9M##HA  ,$($9$9:O$P !&(<(<_(MM ,,E$$&&4+F+F((4;?;\;\%- <8)=%)$E$Ee\`$aM((4;@&3gk<8)= %*-$>M 14D D --0(*$256GI_I_2` . /0@A +GJ]+] (3a So'oRn1C(CRn$'o    102 2"0 AE2Dd3PbKbFc2c,d AF>4 C jn,%(.2+/%);? xZ||xZxZ" xZ %|| xZ "LL xZ""xZ!)c3h 8xZxZ u||55 6xZxZr-r0c^\rSrSrSrU4SjrSS\RS\\RS\\ S\RS\RS \RS \ \ \\4S \S \\\44S jjrSrU=r$)SD3MultiControlNetModelia `SD3ControlNetModel` wrapper class for Multi-SD3ControlNet This module is a wrapper for multiple instances of the `SD3ControlNetModel`. The `forward()` API is designed to be compatible with `SD3ControlNetModel`. Args: controlnets (`List[SD3ControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `SD3ControlNetModel` as a list. cX>[TU]5 [R"U5Ulg)N)rKrLrPrSnets)r[ controlnetsr`s r.rL SD3MultiControlNetModel.__init__s MM+. r-rrrrrrrrrcc [[X#UR55HVun upn U "UUUUU U UUS9n U S:XaU nM [WSU S5VVs/sH unnUU-PM nnn[U54nMX W$s snnf)N)rrrrrrrrr) enumeraterrtuple)r[rrrrrrrrr]imagerr block_samplescontrol_block_samplescontrol_block_sample block_samples r.rSD3MultiControlNetModel.forwards.7s?`d`i`i7j-k )A)j&+!&;#5 %#('=' MAv(5%?BBWXYBZ\ijk\l>m)>m:,l)<7>m&)*//D)E(G%).l,%$ )sA<)r)NNNT)r%r&r'r(rrLr)r*rtensorrrrrr}rrr r!rrr,rrs@r.rrs //3%);? !%||!%ell+!%!K !% "LL !% %|| !%""!%!)c3h 8!%!% "E) *!%!%r-r)/ dataclassesrtypingrrrrrr r)torch.nnrPconfiguration_utilsr r loadersr rutilsrrrr attentionrattention_processorrrr embeddingsrrmodeling_outputsrmodeling_utilsrtransformers.transformer_sd3rrrr get_loggerr%rr!r0rr$r-r.rs ":: B?VV-[[G7'D/   H % 2*2 2bZ[2BDZbZJ 2%j2%r-