+h' SSKrSSKJrJrJrJrJrJr SSKr SSK r SSK J r SSK J s Jr SSKJrJr SSKJrJrJr SSKJrJrJrJrJr SSKJr SSKJ r S S K!J"r"J#r#J$r$ S S K%J&r& S S K'J(r( S S K)J*r*J+r+J,r,J-r- S SK.J/r/ S SK0J1r1 S SK2J3r3J4r4J5r5 \Rl"\75r8S"S#Sjjr9S"S#Sjjr:S"S#Sjjr;"SS5r<"SS\ RRz5r>"SS\ RRz\#5r?\ "SS\ Rz55r@\ "SS\ Rz55rA"SS\ Rz5rB"S S!\1\\\\\(\"5 rCg)$N)AnyDictListOptionalTupleUnion) ConfigMixinregister_to_config)FluxTransformer2DLoadersMixinFromOriginalModelMixinPeftAdapterMixin)USE_PEFT_BACKEND deprecateloggingscale_lora_layersunscale_lora_layers)is_torch_npu_available)maybe_allow_in_graph)AttentionMixinAttentionModuleMixin FeedForward)dispatch_attention_fn) CacheMixin)*CombinedTimestepGuidanceTextProjEmbeddings"CombinedTimestepTextProjEmbeddingsapply_rotary_embget_1d_rotary_pos_embed)Transformer2DModelOutput) ModelMixin)AdaLayerNormContinuousAdaLayerNormZeroAdaLayerNormZeroSingle FluxAttentioncURU5nURU5nURU5nS=n=pxUb@URb3UR U5nUR U5nUR U5nX4XVXx4$N)to_qto_kto_vadded_kv_proj_dim add_q_proj add_k_proj add_v_proj attn hidden_statesencoder_hidden_statesquerykeyvalue encoder_query encoder_key encoder_values h/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/transformers/transformer_flux.py_get_projectionsr:-s IIm $E ))M "C IIm $E266M6K(T-C-C-O(=> oo&;< (=> u[ GGcURU5RSSS9up4nS=n=pxUb3[US5(a"URU5RSSS9upgnX4XVXx4$)Nr dimr' to_added_qkv)to_qkvchunkhasattrr@r/s r9_get_fused_projectionsrD;sy M2888CE299M9K(WT>-J-J484E4EF[4\4b4bcdjl4b4m1 M u[ GGr;cTUR(a [XU5$[XU5$r')fused_projectionsrDr:)r0r1r2s r9_get_qkv_projectionsrGEs' %d;PQQ D1F GGr;c \rSrSrSrSrS SSS\RS\RS\\RS \\RS \R4 S jjr S r g)FluxAttnProcessorKNcr[[S5(d"[URRS35eg)Nscaled_dot_product_attentionz; requires PyTorch 2.0. Please upgrade your pytorch version.)rCF ImportError __class____name__)selfs r9__init__FluxAttnProcessor.__init__Ns3q899!8!8 99tuv v:r;r0r%r1r2attention_maskimage_rotary_embreturncd[XU5upgppURSURS45nURSURS45nURSURS45nURU5nUR U5nUR bU RSURS45n U RSURS45n U RSURS45n UR U 5n URU 5n [R"X/SS9n[R"X/SS9n[R"X/SS9nUb[XeSS9n[XuSS9n[XgXURS9nURSS5nURUR5nUb|UR!UR"SUR"SUR"S- /SS9up2UR$S"U5nUR$S"U5nUR'U5nX#4$U$) Nr=r> sequence_dim) attn_maskbackendrr r)rG unflattenheadsnorm_qnorm_kr+ norm_added_q norm_added_ktorchcatrr_attention_backendflattentodtypesplit_with_sizesshapeto_out to_add_out) rQr0r1r2rTrUr3r4r5r6r7r8s r9__call__FluxAttnProcessor.__call__Rs2H\ !6H DE+TZZ$45mmBR 01TZZ$45 E"kk#  ! ! -)33BR8HIM%//TZZ4DEK)33BR8HIM --m[TU]5 [[S5(d"[ UR R S35eXlX l[U[[45(dU/n[U[5(dU/[U5-n[U5[U5:wa [S5eX@l[R "[#[U55Vs/sHn[R$"X!SXVS9PM sn5Ul[R "[#[U55Vs/sHn[R$"X!SXVS9PM sn5Ulgs snfs snf)NrLz@ requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.zJ`scale` should be a list of integers with the same length as `num_tokens`.T)biasdevicerh)superrRrCrMrNrOrPryrz isinstancetuplelistlen ValueErrorscalenn ModuleListrangeLinearto_k_ipto_v_ip) rQryrz num_tokensrr}rh_rOs r9rR#FluxIPAdapterAttnProcessor.__init__s9 q899>>**++kl '#6 *udm44$J%&&Gc*o-E u:Z (ij j }}s:/ /A -fb/   }}s:/ /A -fb/      s $ E#6 E(r0r%r1r2rTrUip_hidden_statesip_adapter_masksrVc URSn[XU5upppU RSURS45n U RSURS45n U RSURS45n UR U 5n UR U 5n U nUbU RSURS45n U RSURS45n URSURS45nUR U 5n URU 5n [R"X/SS9n [R"X/SS9n [R"X/SS9n Ub[XSS9n [XSS9n [U U U USSURS9nURS S 5nURU R5nUGbUR!URSURSURS- /SS9up2UR"S"U5nUR"S"U5nUR%U5n[R&"U5n[)X`R*UR,UR.5HunnnnU"U5nU"U5nUR1USURUR25nUR1USURUR25n[UUUSSSURS9nUR5USURUR2-5nURUR5nUUU-- nM X#U4$U$) Nrr=rXr>rYF)r[ dropout_p is_causalr\rr )rjrGr]r^r_r`rarbrcrdrrrerfrgrhrirkrl zeros_likeziprrrviewhead_dimreshape)rQr0r1r2rTrUrr batch_sizer3r4r5r6r7r8ip_queryip_attn_outputcurrent_ip_hidden_statesrrrip_keyip_values r9rm#FluxIPAdapterAttnProcessor.__call__sX#((+ G[ !6H DE+TZZ$45mmBR 01TZZ$45 E"kk# ,)33BR8HIM%//TZZ4DEK)33BR8HIM --m[TU]5 X0lU bU OX2-UlXlXPlX@lU bU OUUlXlXl U bX-OUUl X`l Xpl [RRX9U S9Ul[RRX9U S9Ul[RR%XRUS9Ul[RR%XRUS9Ul[RR%XRUS9UlUR(d[RR-/5UlUR.R1[RR%URURUS95 UR.R1[RR3U55 Ub[RRX9S9Ul[RRX9S9Ul[RR%X`RUS9Ul[RR%X`RUS9Ul[RR%X`RUS9Ul[RR%URXS9UlUcURA5nURCU5 g)N)rrr|)r)"r~rRr inner_dimruse_biasrrrrr^r+rrcrRMSNormr_r`rr(r)r*rrkappendDropoutrarbr,r-r.rl_default_processor_cls set_processor)rQrr^rrr|r+rrrrrrr processorrOs r9rRFluxAttention.__init__s5"  $+$7X=M"  ")"5w9 0 ,3,?W(U !2.hh&&xM_&` hh&&xM_&` HHOOI~~DOI HHOOI~~DOI HHOOI~~DOI }}((--b1DK KK  uxxt~~t||RZ[ \ KK  uxx//8 9  ( % 0 0 0 CD  % 0 0 0 CD #hhoo.?VeofDO#hhoo.?VeofDO#hhoo.?VeofDO#hhoodnnioWDO  335I 9%r;r1r2rTrUrVc .[[R"URR5R R 55nSS1nUR5VV s/sHupX;dM X;dMUPM n nn [U 5S:a:[RSU SURRRS35 UR5VV s0sHupX;dM X_M nnn UR"XX#U40UD6$s sn nfs sn nf)Nrrrzjoint_attention_kwargs z are not expected by z and will be ignored.) setinspect signaturerrm parameterskeysitemsrloggerwarningrOrP) rQr1r2rTrUkwargsattn_parametersquiet_attn_parameterskr unused_kwargsws r9forwardFluxAttention.forwardGsg//0G0GHSSXXZ[!35G H'-||~u~tq9QVWVt~ u }  ! NN)-8MdnnNfNfNoNoMpqFG $*<<>J>41Q5I$!$>J~~d3HZjuntuu v Ks# D 2D 9D  D-D)r-r,r.r+rrrrr^rrbrar`r_rrrrlr)rkr(r*r) @rFNTTgh㈵>NNFTNrp)rPrqrrrsrIrrw_available_processorsrfloatboolrrRrcrtrrurrs@r9r%r% sF."+/*.+/#'4&4&4& 4&  4&  4&$C=4&"$4&4&4&4&#4.4&4&!4&4&r9=1537 v||v ( 5v!. v #5<<0 v vvr;c ^\rSrSrSS\S\S\S\4U4SjjjrSS\RS\RS \RS \ \ \R\R4S \ \ \ \ 4S \ \R\R44 S jjrSrU=r$)FluxSingleTransformerBlockiZr?num_attention_headsattention_head_dim mlp_ratioc >[TU]5 [X-5Ul[ U5Ul[ R"XR5Ul[ R"SS9Ul [ R"XR-U5Ul [5(aSSK Jn Sn[SSU5 U"5nO [!5n[#UUUUSUS SS 9Ulg) Ntanh) approximater)FluxAttnProcessor2_0_NPUzDefaulting to FluxAttnProcessor2_0_NPU for NPU devices will be removed. Attention processors should be set explicitly using the `set_attn_processor` method. npu_processorz0.34.0Tư>)rrr^rr|rrr)r~rRrmlp_hidden_dimr$normrrproj_mlpGELUact_mlpproj_outrattention_processorrrrIr%r0) rQr?rrrrdeprecation_messagerrOs r9rR#FluxSingleTransformerBlock.__init__\s !#/2*3/  #':':; ww62  #(;(;";SA ! # # FR  ox1D E02I)+I!'%   r;r1r2tembrUjoint_attention_kwargsrVcURSn[R"X!/SS9nUnURXS9upUR UR U55n U=(d 0nUR "SUUS.UD6n [R"X/SS9nU RS5n XRU5-nXq-nUR[R:XaURSS5nUSS2SU24USS2US24pX!4$) NrXr>emb)r1rUr ro) rjrcrdrrrr0 unsqueezerrhfloat16clip) rQr1r2rrUr text_seq_lenresidualnorm_hidden_statesgatemlp_hidden_states attn_outputs r9r"FluxSingleTransformerBlock.forward|s-2215  #8"HaP  #'99]9#E  LL7I)JK!7!=2ii ,- %  ;"BJ ~~a }}];;  0   %-- /)..vu=M/[TU]5 [U5Ul[U5Ul[ UUUUUSS[ 5US9 Ul[R"USSS9Ul [XSS9Ul [R"USSS9Ul [XSS9Ulg)NFT) rr+rr^rrr|rrrrrzgelu-approximate)r?dim_out activation_fn)r~rRr#norm1 norm1_contextr%rIr0r LayerNormnorm2rff norm2_context ff_context)rQr?rrrrrOs r9rRFluxTransformerBlock.__init__s %c* -c2!!'%"')   \\#%TJ #BTU\\#%TR%#J\]r;r1r2rrUrrVc URXS9upgpn URX#S9uppnU=(d 0nUR"SUU US.UD6n[U5S:XaUunnO[U5S:XaUunnnUR S5W-nUU-nUR U5nUSU SS2S4--USS2S4-nUR U5nU R S5U-nUU-n[U5S:XaUW-nU R S5W-nUU-nURU5n U SUSS2S4--U SS2S4-n URU 5nX/R S5U--nUR[R:XaURSS5nX!4$) Nr)r1r2rUrr rXrrro) rrr0rrrrrrrhrcrr)rQr1r2rrUrrgate_msa shift_mlp scale_mlpgate_mlpnorm_encoder_hidden_states c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlpattention_outputsrcontext_attn_outputr ff_outputcontext_ff_outputs r9rFluxTransformerBlock.forwards HLzzR_zGjDiHW[WiWi !XjX T" *"8!=2!II ,"<- %    !Q &/@ ,K, " #q (?P $>vu$M !$33r;)r0rrrrrr)rms_normrr)rPrqrrrsrrrrRrcrtrrrrrrurrs@r9rrstx^^-0^FI^TW^kp^^>IM;? 64||64 %||64ll 64 #5u||)C#DE 64 !)c3h 8 64 u||U\\) *6464r;rct^\rSrSrS\S\\4U4SjjrS\RS\R4Sjr Sr U=r $) FluxPosEmbedithetaaxes_dimc:>[TU]5 XlX lgr')r~rRrr)rQrrrOs r9rRFluxPosEmbed.__init__s   r;idsrVc URSn/n/nUR5nURRS:HnURRS:HnU(dU(a[R O[R n[U5HSn [URU USS2U 4URSSUS9upURU 5 URU 5 MU [R"USS9RUR5n [R"USS9RUR5n X4$)Nr=mpsnpuT)rrepeat_interleave_realuse_real freqs_dtyper>)rjrr}typercfloat32float64rrrrrrdrg)rQrn_axescos_outsin_outposis_mpsis_npur$icossin freqs_cos freqs_sins r9rFluxPosEmbed.forwards2iikE)E)(.&emmu}} vA. a AqD jj'+' HC NN3  NN3 IIg2.11#**= IIg2.11#**= ##r;)rr) rPrqrrrsrrrRrcrtrrurrs@r9rrs:!c!T#Y! $5<<$ELL$$r;rc^\rSrSrSrSrSS/rSS/rSS/r\ S!S\ S \ S \ \ S \ S \ S \ S\ S\ S\ S\ S\ \ \ \ 44U4Sjjj5rS"S\R S\R S\R S\R"S\R S\R S\R S\ \\\4S\ S\ S\\R \44SjjrS rU=r$)#FluxTransformer2DModelia The Transformer model introduced in Flux. Reference: https://blackforestlabs.ai/announcing-black-forest-labs/ Args: patch_size (`int`, defaults to `1`): Patch size to turn the input data into small patches. in_channels (`int`, defaults to `64`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `None`): The number of channels in the output. If not specified, it defaults to `in_channels`. num_layers (`int`, defaults to `19`): The number of layers of dual stream DiT blocks to use. num_single_layers (`int`, defaults to `38`): The number of layers of single stream DiT blocks to use. attention_head_dim (`int`, defaults to `128`): The number of dimensions to use for each attention head. num_attention_heads (`int`, defaults to `24`): The number of attention heads to use. joint_attention_dim (`int`, defaults to `4096`): The number of dimensions to use for the joint attention (embedding/channel dimension of `encoder_hidden_states`). pooled_projection_dim (`int`, defaults to `768`): The number of dimensions to use for the pooled projection. guidance_embeds (`bool`, defaults to `False`): Whether to use guidance embeddings for guidance-distilled variant of the model. axes_dims_rope (`Tuple[int]`, defaults to `(16, 56, 56)`): The dimensions to use for the rotary positional embeddings. Trr pos_embedr patch_size in_channels out_channels num_layersnum_single_layersrrjoint_attention_dimpooled_projection_dimguidance_embedsaxes_dims_ropec L>[TU]5 U=(d UUlXv-Ul[ SU S9UlU (a[ O[n U "URU S9Ul[R"XR5Ul [R"X R5Ul [R"[U5V s/sHn [URUUS9PM sn 5Ul[R"[U5V s/sHn [#URUUS9PM sn 5Ul['URURSSS9Ul[R"URX-UR-SS 9UlSUlgs sn fs sn f) Ni')rr) embedding_dimr=)r?rrFrrTr)r~rRr9rrr6rrtime_text_embedrrcontext_embedder x_embedderrrrtransformer_blocksrsingle_transformer_blocksr"norm_outrgradient_checkpointing)rQr7r8r9r:r;rrr<r=r>r?text_time_guidance_clsrrOs r9rRFluxTransformer2DModel.__init__=sw (7K,A%ENK;J 6Oq  6..@U !# *=~~ N))K@"$--z*   +A %(;'9 +   # *,01   2A +(;'9 2   * &/t~~t~~bgmqr  $..*2IDL]L]2]dhi &+#1  s FF!r1r2pooled_projectionstimestepimg_idstxt_idsguidancer return_dictcontrolnet_blocks_repeatrVc Ub#UR5nURSS5n OSn [(a [X 5 O+Ub(UR SS5b[ R S5 URU5nURUR5S-nUbURUR5S-nUcURXC5OURXGU5nURU5nURS:Xa[ R S5 USnURS:Xa[ R S 5 USn[R"Xe4SS 9nURU5nUb;S U;a5URS 5nUR!U5nUR#S U05 [%UR&5Hunn[R("5(a*UR*(aUR-UUUUUU5up!O U"UUUUUS 9up!U cM[[/UR&5[/U 5- n[1[2R4"U55nU (aXU[/U 5--nMXUU--nM [%UR65Hunn[R("5(a*UR*(aUR-UUUUUU5up!O U"UUUUUS 9up!U cM[[/UR65[/U 5- n[1[2R4"U55nXUU--nM UR9X5nUR;U5n[(a [=X 5 U (dU4$[?US9$)a The [`FluxTransformer2DModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch_size, image_sequence_length, in_channels)`): Input `hidden_states`. encoder_hidden_states (`torch.Tensor` of shape `(batch_size, text_sequence_length, joint_attention_dim)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.Tensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states: (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nrrz\Passing `scale` via `joint_attention_kwargs` when not using the PEFT backend is ineffective.ir zrPassing `txt_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch TensorrzrPassing `img_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch Tensorr>ip_adapter_image_embedsr)r1r2rrUr)sample) copypoprrgetrrrDrgrhrBrCndimrcrdr6encoder_hid_projupdate enumeraterEis_grad_enabledrH_gradient_checkpointing_funcrrnpceilrFrGrrr )rQr1r2rKrLrMrNrOrcontrolnet_block_samplescontrolnet_single_block_samplesrPrQ lora_scalerrrUrSr index_blockblockinterval_controloutputs r9rFluxTransformer2DModel.forwardwsR " -%;%@%@%B "/33GSAJJ   d /%16L6P6PQXZ^6_6kr 6 ;;}223d:  {{=#6#67$>H   >%%h:LM !% 5 56K L <<1  NNU ajG <<1  NNU ajGii*2>>#. ! -2KOe2e&<&@&@AZ&[ ##445LM  " ) )+=?O*P Q"+D,C,C"D K$$&&4+F+F7;7X7X!)$* 84%}8="/*?%5+A 84%(3#&t'>'>#?#F^B_#_ #&rww/?'@#A +%sSkOlAl(mm"%2[\lMl4m$mM=#E@#,D,J,J"K K$$&&4+F+F7;7X7X!)$* 84%}8="/*?%5+A 84%/:#&t'E'E#FMlIm#m #&rww/?'@#A -P[_oPo0p p 1#L4 m: }-    19 'v66r;) rCrHrrGr9r6rrFrBrErD) rXrN&iiF)8rm) NNNNNNNNNTF)rPrqrrrsr _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patterns_repeated_blocksr rrrrrRrcrt LongTensorrrrrr rrurrs@r9r5r5s>(,$/1MN(3V'<$.0LM&*!#"%#%#'%( %/;7,7,7,sm 7,  7,  7, 7,!7,!7, #7,7,c3m,7,7,x/3+/%) $ $!%;?!%(, ).^7||^7 %||^7"LL ^7 "" ^7  ^7^7,,^7!)c3h 8^7^7#'^7 u||55 6^7^7r;r5r')r0r%)Drtypingrrrrrrnumpyr^rctorch.nnrtorch.nn.functional functionalrMconfiguration_utilsr r loadersr r rutilsrrrrrutils.import_utilsrutils.torch_utilsr attentionrrrattention_dispatchr cache_utilsr embeddingsrrrrmodeling_outputsr modeling_utilsr! normalizationr"r#r$ get_loggerrPrr:rDrGrIModulerwr%rrrr5ror;r9rs.:: B^^aa85II6$ 8'\\   H % HHH :!:!z!!DMvEHHOO%9Mv`>4>4>4BQ4299Q4Q4h$299$>E7!E7r;