+hbSSKJrJrJrJrJr SSKrSSKJs J r SSKJr SSK J r J r SSKJrJr SSKJrJrJrJr SS KJrJrJr SS KJrJrJrJr SS KJ r SS K!J"r" SS K#J$r$J%r% \RL"\'5r("SS\RR5r*"SS\RR5r+"SS\RR5r,"SS5r-"SS\RR5r."SS\"\ \\5r/g))AnyDictOptionalTupleUnionN)nn) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)USE_PEFT_BACKENDloggingscale_lora_layersunscale_lora_layers) AttentionAttentionProcessorSanaLinearAttnProcessor2_0) PatchEmbedPixArtAlphaTextProjectionTimestepEmbedding Timesteps)Transformer2DModelOutput) ModelMixin)AdaLayerNormSingleRMSNormc ^\rSrSrS S\S\S\S\\S\SS4 U4S jjjr S \ RS\ R4S jr S r U=r$) GLUMBConv&N in_channels out_channels expand_ratio norm_typeresidual_connectionreturnc >[TU]5 [X1-5nX@lXPl[ R "5Ul[ R"XS-SSS5Ul [ R"US-US-SSSUS-S9Ul [ R"XbSSSSS9Ul SUl US:Xa[US S S S 9Ul gg) Nrrr )groupsFbiasrms_normh㈵>T)epselementwise_affiner+)super__init__intr$r%rSiLU nonlinearityConv2d conv_inverted conv_depth conv_pointnormr)selfr!r"r#r$r%hidden_channels __class__s h/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/transformers/sana_transformer.pyr1GLUMBConv.__init__'s l89"#6 GGIYY{a4GAqQ))Oa$719LaQRTU^mpq^qr))O1aQVW  " $4VZ[DI # hidden_statescUR(aUnURU5nURU5nURU5n[R "USSS9upXRU5-nUR U5nURS:Xa1URURSS55RSS5nUR(aUW-nU$)Nrr(dimr,) r%r6r4r7torchchunkr8r$r9movedim)r:r@residualgates r=forwardGLUMBConv.forward>s  # #$H**=9 ))-8  6 #kk-B %(9(9$(??  6 >>Z ' IIm&;&;Ar&BCKKBPQRM  # #)H4Mr?)r7r6r8r4r9r$r%)NT)__name__ __module__ __qualname____firstlineno__r2floatrstrboolr1rETensorrJ__static_attributes__ __classcell__r<s@r=rr&s~  #'$( \\\ \ C= \ " \ \\.U\\ellr?rc^\rSrSrS S\S\S\4U4SjjjrS\RS\RS\RS \R4S jr S r U=r $) SanaModulatedNormUrCr/r.cV>[TU]5 [R"XUS9Ulg)Nr/r.)r0r1r LayerNormr9)r:rCr/r.r<s r=r1SanaModulatedNorm.__init__Vs! LLQTU r?r@tembscale_shift_tabler&cURU5nUSUSS2S4RUR5-RSSS9upEUSU--U-nU$)Nrr(rB)r9todevicerF)r:r@r_r`shiftscales r=rJSanaModulatedNorm.forwardZsh -0 )$/$q$w-2B2BCTC[C[2\\ccdeklcm %U3e; r?)r9)Fư>) rMrNrOrPr2rSrQr1rErTrJrUrVrWs@r=rYrYUscVCVTVVV"\\16RWR^R^ r?rYc~^\rSrSrU4SjrSS\R S\R S\R4SjjrSr U=r $) &SanaCombinedTimestepGuidanceEmbeddingscc>[TU]5 [SSSS9Ul[ SUS9Ul[SSSS9Ul[ SUS9Ul[R"5Ul [R"USU-SS9Ul g)NTr) num_channelsflip_sin_to_cosdownscale_freq_shift)r!time_embed_dimr*) r0r1r time_projrtimestep_embedderguidance_condition_projguidance_embedderrr3siluLinearlinear)r: embedding_dimr<s r=r1/SanaCombinedTimestepGuidanceEmbeddings.__init__dsx "T`ab!2sS`!a'0cSWno'p$!2sS`!aGGI ii q=/@tL r?timestepguidance hidden_dtypec URU5nURURUS95nURU5nUR URUS95nXW-nUR UR U55U4$)N)dtype)rrrsrbrtrurxrv) r:r{r|r}timesteps_proj timesteps_emb guidance_proj guidance_emb conditionings r=rJ.SanaCombinedTimestepGuidanceEmbeddings.forwardos1..~/@/@|/@/TU 44X> --m.>.>\.>.RS $3 {{499\23\AAr?)rtrurxrvrrrsNN) rMrNrOrPr1rErTrrJrUrVrWs@r=rirics< MB B B[`[f[fBBr?ric \rSrSrSrSrS S\S\RS\ \RS\ \RS \R4 S jjr S r g) SanaAttnProcessor2_0zzk Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). cD[[S5(d [S5eg)Nscaled_dot_product_attentionzTSanaAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.)hasattrF ImportError)r:s r=r1SanaAttnProcessor2_0.__init__s!q899tu u:r?Nattnr@encoder_hidden_statesattention_maskr&c *Uc URO URupVnUb<URXFU5nURXQRSURS5nUR U5nUcUnUR U5n UR U5n URbURU5nURbURU 5n U RSn XR-n URUSURU 5RSS5nU RUSURU 5RSS5n U RUSURU 5RSS5n [R"XXSSS9nURSS5RUSURU -5nURUR5nURS"U5nURS"U5nX!R - nU$)NrDr(rF) attn_mask dropout_p is_causalr)shapeprepare_attention_maskviewheadsto_qto_kto_vnorm_qnorm_k transposerrreshaperbrto_outrescale_output_factor) r:rr@rr batch_sizesequence_length_querykeyvalue inner_dimhead_dims r=__call__SanaAttnProcessor2_0.__call__s$9#@M  F[FaFa ' Q  %!88ZdeN,00ZZ^MaMabdMefN -( ($1 !ii-. /0 ;; "KK&E ;; "++c"CIIbM  * :r4::x@JJ1aPhhz2tzz8<FFq!L :r4::x@JJ1aP66 3RW &//15==j"djj[cNcd %((5  A}5  A}5 %(B(BB r?r) rMrNrOrP__doc__r1rrErTrrrUrr?r=rrzsiv9=15 66||6 ( 5 6 !. 6  66r?rch^\rSrSrSrSS\S\S\S\S\\S \\S \\S \S \S \S\S\S\\ SS4U4Sjjjr SS\ RS\\ RS\\ RS\\ RS\\ RS\S\S\ R4SjjrSrU=r$)SanaTransformerBlockzS Transformer block introduced in [Sana](https://huggingface.co/papers/2410.10629). NrCnum_attention_headsattention_head_dimdropoutnum_cross_attention_headscross_attention_head_dimcross_attention_dimattention_biasnorm_elementwise_affinenorm_epsattention_out_bias mlp_ratioqk_normr&c >[TU]5 [R"USU S9Ul[ UUUU bUOSU UUS[ 5S9 UlUb>[R"XU S9Ul[ UU U bUOSUUUUSU [5S9 Ul [XU SSS9Ul [R"[R"SU5US-- 5Ulg) NFr\) query_dimrdim_headkv_headsrrr+r processorT) rrrrrrrr+out_biasr)r$r%rq?)r0r1rr]norm1rrattn1norm2rattn2rff ParameterrErandnr`)r:rCrrrrrrrrrrrrr<s r=r1SanaTransformerBlock.__init__s \\#%XN %',3,?(T $02     *c[cdDJ"6=6I2t$7/1+.0 DJCi4UZ[!#ekk!S.ACH.L!Mr?r@rrencoder_attention_maskr{heightwidthcjURSnURSURUSS5-RSSS9upppUR U5nUSU --U -nUR UR 5nURU5nXU--nURbURUUUS9nUU-nURU5nUSU --U -nURSXg45RSSSS5nURU5nURSS5RSSS5nXU--nU$) NrrqrDr(rB)rrr r)rr`rrFrrbrrrr unflattenpermuterflatten)r:r@rrrr{rrr shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlpnorm_hidden_states attn_output ff_outputs r=rJSanaTransformerBlock.forwardso#((+   " "4 (8+;+;J2+N N %q%/ G h9 "ZZ 6/1y=AIM/22=3F3FGjj!34 %;(>>  :: !**&;5%K (-7M"ZZ 6/1y=AIM/99!f_MUUVWYZ\]_`aGG./ %%a+33Aq!< %9(<< r?)rrrrrr`) F rprTFrgT@N)NNNNNN)rMrNrOrPrr2rQrrSrRr1rErT LongTensorrJrUrVrWs@r=rrs #%"$3525-1#(-#'!%3N 3N!3N 3N  3N $,C= 3N#+3-3N&c]3N3N"&3N3N!3N3N#3N 3N3Np268<9=/3+||+!.+ ( 5 + !) 6 + 5++, +++ ++r?rc.`^\rSrSrSrSr/SQrSS/r\S.S\ S \ \ S \ S \ S \ S \ \ S\ \ S\ \ S\ S\ S\ S\ S\ S\ S\ S\ S\ \ S\ S\ S\ \ S\ SS4,U4Sjjj5r\S\\ \44Sj5rS \\\\ \444S!jrS/S"\R,S#\R,S$\R,S%\ \R,S&\ \R,S'\ \R,S(\ \\ \4S)\ \\R,S*\ S\\\R,S+4\44S,jjrS-rU=r$)0SanaTransformer2DModeli$a A 2D Transformer model introduced in [Sana](https://huggingface.co/papers/2410.10629) family of models. Args: in_channels (`int`, defaults to `32`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `32`): The number of channels in the output. num_attention_heads (`int`, defaults to `70`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `32`): The number of channels in each head. num_layers (`int`, defaults to `20`): The number of layers of Transformer blocks to use. num_cross_attention_heads (`int`, *optional*, defaults to `20`): The number of heads to use for cross-attention. cross_attention_head_dim (`int`, *optional*, defaults to `112`): The number of channels in each head for cross-attention. cross_attention_dim (`int`, *optional*, defaults to `2240`): The number of channels in the cross-attention output. caption_channels (`int`, defaults to `2304`): The number of channels in the caption embeddings. mlp_ratio (`float`, defaults to `2.5`): The expansion ratio to use in the GLUMBConv layer. dropout (`float`, defaults to `0.0`): The dropout probability. attention_bias (`bool`, defaults to `False`): Whether to use bias in the attention layer. sample_size (`int`, defaults to `32`): The base size of the input latent. patch_size (`int`, defaults to `1`): The size of the patches to use in the patch embedding layer. norm_elementwise_affine (`bool`, defaults to `False`): Whether to use elementwise affinity in the normalization layer. norm_eps (`float`, defaults to `1e-6`): The epsilon value for the normalization layer. qk_norm (`str`, *optional*, defaults to `None`): The normalization to use for the query and key. timestep_scale (`float`, defaults to `1.0`): The scale to use for the timesteps. T)rrrY patch_embedr9Nr!r"rr num_layersrrrcaption_channelsrrr sample_size patch_sizerrinterpolation_scaleguidance_embedsguidance_embeds_scalertimestep_scaler&ct>[TU]5 U=(d UnX4-n[U U UUUUUbSOSS9UlU(a[ U5UlO[ U5Ul[U US9Ul[USSS9Ul [R"[U5Vs/sHn[UUUU UUUU UUU US9 PM sn5Ul[R "["R$"SU5US -- 5Ul[)US S S 9Ul[R,"UX-U-5UlS Ulgs snf) Nsincos)rrrr! embed_dimrpos_embed_type) in_features hidden_sizer-T)r.r/) rrrrrrrrrrrFrgr\)r0r1rrri time_embedrrcaption_projectionr caption_normr ModuleListrangertransformer_blocksrrErr`rYnorm_outrwproj_outgradient_checkpointing)r:r!r"rrrrrrrrrrrrrrrrrrrrrr<s r=r1SanaTransformer2DModel.__init__SsN2 #2{ '< &!# 3':'F8D  DYODO0;DO";HXfo"p#I4DQ#%-- z* +A%'&#.G-E(;#1,C%'# + # *"$ekk!Y.G)UX..X!Y))SWX  )Z-D|-ST &+#1 sD5c^0nS[S[RRS[[[ 44U4SjjmUR 5Hup#T"X#U5 M U$)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsc>[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processor.)rrnamed_children)rrrsub_namechildfn_recursive_add_processorss r=r KSanaTransformer2DModel.attn_processors..fn_recursive_add_processorssZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; r?)rRrErModulerrr )r:rrrr s @r=attn_processors&SanaTransformer2DModel.attn_processorssb  c 588?? X\]`bt]tXu !//1LD 'j A2r?rc d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrc >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorrr )r isinstancedictrpopr )rrrr r fn_recursive_attn_processors r=rNSanaTransformer2DModel.set_attn_processor..fn_recursive_attn_processorssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r?N) lenrkeysrr ValueErrorrRrErrr )r:rcountrrrs @r=set_attn_processor)SanaTransformer2DModel.set_attn_processorsD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r?r@rr{r|rrattention_kwargscontrolnet_block_samples return_dict.c Ub#UR5nURSS5n OSn [(a [X 5 O+Ub(UR SS5b[ R S5 UbBURS:Xa2SURUR5- S-nURS5nUbBURS:Xa2SURUR5- S-nURS5nURuppURRnX-X-nnURU5nUbURX4URS9unnOURX;URS9unnUR!U5nUR#U S URS 5nUR%U5n[&R("5(atUR*(ac[-UR.5HIunnUR1UUUUUUUU5nUcM#S Us=:a[3U5::dM;O M?XUS- -nMK OX[-UR.5H?unnU"UUUUUUU5nUcMS Us=:a[3U5::dM1O M5XUS- -nMA UR5UUUR65nUR9U5nUR;U UUURRURRS 5nUR=S S SS SS 5nUR;U S UU-UU-5n[(a [?X 5 U (dU4$[AUS9$)Nre?zVPassing `scale` via `attention_kwargs` when not using the PEFT backend is ineffective.rr(g)r|r})rr}rDrr rL)sample)!copyrrrgetloggerwarningndimrbr unsqueezerconfigrrrrrrrEis_grad_enabledr enumerater_gradient_checkpointing_funcrrr`rrrrr)r:r@rr{r|rrr r!r" lora_scalerrmrrppost_patch_heightpost_patch_widthembedded_timestep index_blockblockoutputs r=rJSanaTransformer2DModel.forwards  '/446 )--gs;JJ   d /+0@0D0DWd0S0_l  %.*=*=*B  ."3"3M4G4G"HHHTN+55a8N " -2H2M2MQR2R&'*@*C*CMDWDW*X&X\d%d "%;%E%Ea%H "3@2E2E/ & KK " ".4k5:+((7  *.//-:M:M+:+ 'H'+///m>Q>Q+:+ 'H'!% 7 78M N 5 : ::r=K^K^_aKb c $ 1 12G H  " "t'B'B&/0G0G&H" U $ A A!")*%$ ! ,7A $&(#%"$3525-1 $$(--1 %'*!% #-K,K,smK,! K,  K,  K,$,C=K,#+3-K,&c]K,K,K,K,K,K,K, "&!K,"#K,$&c]%K,&'K,( %)K,*#+K,,-K,. /K,K,Zc+=&=!>0 AE2Dd3PbKbFc2c,d AN,09=1559BF x7||x7 %||x7,, x7 5<<( x7 !) 6 x7!.x7#4S>2x7#+5+>"?x7x7 uU\\3&')AA Bx7x7r?r)0typingrrrrrrEtorch.nn.functionalr functionalrconfiguration_utilsr r loadersr r utilsrrrrattention_processorrrr embeddingsrrrrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerrMr)rrrYrirrrrr?r=rJs54 B?VV ]\7'7   H %, ,^  BRYYB.??De299ePq7Z6FH^q7r?