+h K VSSKJrJrJrJrJrJr SSKrSSKJ r SSK J r J r SSK JrJrJr SSKJrJrJrJr SSKJr SS KJrJr SS KJrJrJrJr SS K J!r!J"r" SS K#J$r$ SS K%J&r& SSK'J(r(J)r) \RT"\+5r,\"SS\ RZ55r."SS\&\ \\\5r/g))AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixinSD3Transformer2DLoadersMixin)USE_PEFT_BACKENDloggingscale_lora_layersunscale_lora_layers)maybe_allow_in_graph) FeedForwardJointTransformerBlock) AttentionAttentionProcessorFusedJointAttnProcessor2_0JointAttnProcessor2_0)"CombinedTimestepTextProjEmbeddings PatchEmbed)Transformer2DModelOutput) ModelMixin)AdaLayerNormContinuousAdaLayerNormZerocr^\rSrSrS\S\S\4U4SjjrS\RS\R4SjrS r U=r $) SD3SingleTransformerBlock'dimnum_attention_headsattention_head_dimc >[TU]5 [U5Ul[ UUUUS[ 5SS9Ul[R"USSS9Ul [XSS9Ul g)NTư>) query_dimdim_headheadsout_dimbias processorepsFelementwise_affiner/zgelu-approximate)r$dim_out activation_fn) super__init__r norm1rrattnnn LayerNormnorm2rff)selfr$r%r& __class__s g/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/transformers/transformer_sd3.pyr5"SD3SingleTransformerBlock.__init__)se %c* '%+-  \\#%TJ #BTU hidden_statestembcBURXS9up4pVnURUSS9nURS5U-nX-nURU5nUSURS5--URS5-nUR U5n URS5U -n X-nU$)N)emb)rAencoder_hidden_states)r6r7 unsqueezer:r;) r<rArBnorm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlp attn_output ff_outputs r>forward!SD3SingleTransformerBlock.forward?sGKzzR_zGjDiHii.@X\i] ((+k9 %3 "ZZ 6/1y7J7J17M3MNQZQdQdefQggGG./ &&q)I5 %1 r@)r7r;r6r:) __name__ __module__ __qualname____firstlineno__intr5torchTensorrO__static_attributes__ __classcell__r=s@r>r"r"'sHV V!V V,U\\r@r"c^\rSrSrSrSrS/rSS/r\S+S\ S \ S \ S \ S \ S \ S\ S\ S\ S\ S\ S\ \ S4S\ \ 4U4Sjjj5r S,S\ \ S\ SS4SjjrSr\S\\ \44Sj5rS\\\\ \444SjrSrS rS-S!\R2S"\R2S#\R2S$\R4S%\S&\ \\ \4S'\S(\ \\ S\\R2\44S)jjrS*r U=r!$).SD3Transformer2DModelPax The Transformer model introduced in [Stable Diffusion 3](https://huggingface.co/papers/2403.03206). Parameters: sample_size (`int`, defaults to `128`): The width/height of the latents. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, defaults to `2`): Patch size to turn the input data into small patches. in_channels (`int`, defaults to `16`): The number of latent channels in the input. num_layers (`int`, defaults to `18`): The number of layers of transformer blocks to use. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. num_attention_heads (`int`, defaults to `18`): The number of heads to use for multi-head attention. joint_attention_dim (`int`, defaults to `4096`): The embedding dimension to use for joint text-image attention. caption_projection_dim (`int`, defaults to `1152`): The embedding dimension of caption embeddings. pooled_projection_dim (`int`, defaults to `2048`): The embedding dimension of pooled text projections. out_channels (`int`, defaults to `16`): The number of latent channels in the output. pos_embed_max_size (`int`, defaults to `96`): The maximum latent height/width of positional embeddings. dual_attention_layers (`Tuple[int, ...]`, defaults to `()`): The number of dual-stream transformer blocks to use. qk_norm (`str`, *optional*, defaults to `None`): The normalization to use for query and key in the attention layer. If `None`, no normalization is used. Tr pos_embednormN sample_size patch_size in_channels num_layersr&r%joint_attention_dimcaption_projection_dimpooled_projection_dim out_channelspos_embed_max_sizedual_attention_layers.qk_normcf>[TU]5 U bU OUUlXe-Ul[ UUUUURU S9Ul[ URU S9Ul[R"Xx5Ul [R"[U5Vs/sH'n[URUUXS- :HU X;aSOSS9PM) sn5Ul[URURSSS9Ul[R"URX"-UR-SS 9UlSUlgs snf) N)heightwidthrarb embed_dimrh) embedding_dimrfrFTF)r$r%r&context_pre_onlyrjuse_dual_attentionr(r0)r-)r4r5rg inner_dimrr^rtime_text_embedr8Linearcontext_embedder ModuleListrangertransformer_blocksrnorm_outproj_outgradient_checkpointing)r<r`rarbrcr&r%rdrerfrgrhrirjir=s r>r5SD3Transformer2DModel.__init__xs+& ,8,DL+,A#!#nn1   B..@U !# *= V"$--z* +A&(;'9%&q.%8#/0/Itu +  # /t~~t~~bgmqr  $..*2IDL]L]2]dhi &+#! s.D. chunk_sizer$returnc^US;a[SU35eU=(d SnS[RRS[S[4U4SjjmUR 5H nT"X1U5 M g) a Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrFz-Make sure to set `dim` to either 0 or 1, not rFmoduler~r$c>[US5(aURXS9 UR5H nT"X1U5 M gNset_chunk_feed_forward)r~r$hasattrrchildrenrr~r$childfn_recursive_feed_forwards r>rPSD3Transformer2DModel.enable_forward_chunking..fn_recursive_feed_forward=v788---M*)%SA+r@N) ValueErrorrVr8ModulerUr)r<r~r$rrs @r>enable_forward_chunking-SD3Transformer2DModel.enable_forward_chunkingsn f LSERS S _1  Behhoo B3 BUX BmmoF %f# >&r@c^S[RRS[S[4U4SjjmUR 5H nT"USS5 M g)Nrr~r$c>[US5(aURXS9 UR5H nT"X1U5 M grrrs r>rQSD3Transformer2DModel.disable_forward_chunking..fn_recursive_feed_forwardrr@r)rVr8rrUr)r<rrs @r>disable_forward_chunking.SD3Transformer2DModel.disable_forward_chunkingsH Behhoo B3 BUX BmmoF %fdA 6&r@c^0nS[S[RRS[[[ 44U4SjjmUR 5Hup#T"X#U5 M U$)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namer processorsc>[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processor.)rrnamed_children)rrrsub_namerfn_recursive_add_processorss r>rJSD3Transformer2DModel.attn_processors..fn_recursive_add_processorssZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; r@)strrVr8rrrr)r<rrrrs @r>attn_processors%SD3Transformer2DModel.attn_processorssb  c 588?? X\]`bt]tXu !//1LD 'j A2r@r.c d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrc >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorrr)r isinstancedictrpopr)rrr.rrfn_recursive_attn_processors r>rMSD3Transformer2DModel.set_attn_processor..fn_recursive_attn_processorssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r@N) lenrkeysrrrrrVr8rr)r<r.countrrrs @r>set_attn_processor(SD3Transformer2DModel.set_attn_processorsD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r@cSUlURR5H3upS[URR 5;dM*[ S5e URUlUR5H)n[U[5(dMURSS9 M+ UR[55 g)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsritemsrr=rQrmodulesrrfuse_projectionsrr)r<_attn_processorrs r>fuse_qkv_projections*SD3Transformer2DModel.fuse_qkv_projectionss)-%!%!5!5!;!;!= A#n66??@@ !tuu">)-(<(<%llnF&),,''T'2%  : <=r@cVURbURUR5 gg)umDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rr)r<s r>unfuse_qkv_projections,SD3Transformer2DModel.unfuse_qkv_projections0s)  ( ( 4  # #D$A$A B 5r@rArEpooled_projectionstimestepblock_controlnet_hidden_statesjoint_attention_kwargs return_dict skip_layersc 4Ub#UR5nURSS5n OSn [(a [X 5 O+Ub(UR SS5b[ R S5 URSSupURU5nURXC5n URU5nUb9SU;a3URS5n URX5upURXS9 [UR5HunnUbUU;aSOS n[R "5(a0UR"(aU(dUR%UUUU U5up!OU(d U"UUU US 9up!UcMtUR&S LdM[)UR5[)U5- nX[+UU- 5-nM UR-X5nUR/U5nUR0R2nU U-n U U-n UR5URS XUUUR64S 9n[R8"S U5nUR5URS UR6U U-U U-4S 9n[(a [;X 5 U(dU4$[=US9$)a  The [`SD3Transformer2DModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. encoder_hidden_states (`torch.Tensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.Tensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep (`torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. skip_layers (`list` of `int`, *optional*): A list of layer indices to skip during the forward pass. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nscaleg?z\Passing `scale` via `joint_attention_kwargs` when not using the PEFT backend is ineffective.ip_adapter_image_embeds)ip_hidden_statesrBTF)rArErBrr)shapeznhwpqc->nchpwq)sample)copyrrrgetloggerwarningrr^rsru image_projupdate enumeraterxrVis_grad_enabledr{_gradient_checkpointing_funcrprrUryrzconfigrareshapergeinsumrr)r<rArErrrrrr lora_scalerlrmrBrrip_temb index_blockblockis_skipinterval_controlraoutputs r>rOSD3Transformer2DModel.forward=sN " -%;%@%@%B "/33GSAJJ   d /%16L6P6PQXZ^6_6kr&++BC0 }5 ##HA $ 5 56K L ! -2KOe2e&<&@&@AZ&[ #(,8O(Z %  " ) );K ) Z"+D,C,C"D K)5+:TdZ_G$$&&4+F+Fw7;7X7X!)* 84%}7<"/*?+A 84%.9e>T>TX]>]#&t'>'>#?#FdBe#e -sS^aqSqOr0s s /#E2 m:  m4 [[++ :%#%-- &&q)6*jRVRcRcd.  %5}E && &&q)4+<+QSX[eSef'     19 'v66r@) rur{rrryrrgr^rzrsrx) r@riiir`N)Nr)NNNNNTN)"rQrRrSrT__doc__ _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patternsr rUrrrr5rrpropertyrrrrrrrrVrW LongTensorrrboolrrOrXrYrZs@r>r\r\Ps(B(,$01(3V'<$"$#%#'&*%)"$ !%!4,4,4, 4,  4,  4,!4,!4,!$4, #4,4, 4, % H 4, #!4,4,n?(3-?S?Y]?< 7c+=&=!>0 AE2Dd3PbKbFc2c,d AF>4 C /3+/%)/3;? +/r7||r7 %||r7"LL r7 "" r7 )- r7!)c3h 8r7r7d3i(r7 u||55 6r7r7r@r\)0typingrrrrrrrVtorch.nnr8configuration_utilsr r loadersr r rutilsrrrrutils.torch_utilsr attentionrrattention_processorrrrr embeddingsrrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerrQrrr"r\rr@r>rs;: B]]VV5: H7'D   H %% %%P_7 -/EGc_7r@