+hSSKJrJrJrJrJrJrJr SSKrSSK J r SSK J s J r SSKJrJr SSKJrJrJr SSKJr SSKJrJrJrJrJrJr SS KJrJ r J!r! SS K"J#r# SS K$J%r%J&r&J'r'J(r(J)r) \"5(aSSK*r+OSr+\RX"\-5r."S S 5r/"SS5r0S\ RbS\RdS\3S\34Sjr4\"SS\ Rb55r5\"SS\ Rb55r6\"SS\ Rb55r7"SS\ Rb5r8\"SS\ Rb55r9"SS \ Rb5r:\"S!S"\ Rb55r;"S#S$\ Rb5r[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processor.)hasattrr*named_children)r&r'r(sub_namechildfn_recursive_add_processorss T/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/attention.pyr1CAttentionMixin.attn_processors..fn_recursive_add_processors2sZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; )strtorchnnModulerrr.)selfr(r&r'r1s @r2attn_processorsAttentionMixin.attn_processors(sb  c 588?? X\]`bt]tXu !//1LD 'j A2r4 processorc d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.r&r'c >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorr+r,)r- isinstancedictr?popr.)r&r'r<r/r0fn_recursive_attn_processors r2rCFAttentionMixin.set_attn_processor..fn_recursive_attn_processorUssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r4N) lenr:keysr@rA ValueErrorr5r6r7r8r.)r9r<countr&r'rCs @r2set_attn_processor!AttentionMixin.set_attn_processor@sD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r4c"URR5H3upS[URR5;dM*[ S5e UR 5H*n[U[5(dMUR5 M, g)z Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. AddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.N) r:itemsr5 __class____name__rGmodulesr@AttentionModuleMixinfuse_projections)r9_attn_processorr's r2fuse_qkv_projections#AttentionMixin.fuse_qkv_projectionsbsq "&!5!5!;!;!= A#n66??@@ !tuu">llnF&"677'')%r4cUR5H*n[U[5(dMUR5 M, g)ulDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rPr@rQunfuse_projections)r9r's r2unfuse_qkv_projections%AttentionMixin.unfuse_qkv_projectionsos.llnF&"677))+%r4N)rO __module__ __qualname____firstlineno__propertyrr5rr:r rIrUrY__static_attributes__r[r4r2r"r"'sU c+=&=!>. AE2Dd3PbKbFc2c,d AD * ,r4r"c \rSrSrSr/rSrS\SS4SjrS'S\ SS4S jjr S \ 4S jr S \ SS4S jr S(S\ S\\\\ S4SS4SjjrS)S\ S\\SS4Sjjr\R("5S5r\R("5S5rS\SS4SjrS\R2S\R24SjrS*S\R2S\S\R24SjjrS)S\R2S\R2S\\R2S\R24S jjrS*S\R2S!\S"\S\S\R24 S#jjrS$\R2S\R24S%jrS&rg)+rQ}NFr<r$cj[US5(a[UR[RR 5(ai[U[RR 5(d@[ RSURSU35 URRS5 Xlg)zu Set the attention processor to use. Args: processor (`AttnProcessor`): The attention processor to use. r<z-You are removing possibly trained weights of z with N) r-r@r<r6r7r8loggerinfo_modulesrB)r9r<s r2r?"AttentionModuleMixin.set_processorsx D+ & &4>>588??;;y%((//:: KKGGWW]^g]hi j MM  k *"r4return_deprecated_lorarc*U(d UR$g)z Get the attention processor in use. Args: return_deprecated_lora (`bool`, *optional*, defaults to `False`): Set to `True` to return the deprecated LoRA attention processor. Returns: "AttentionProcessor": The attention processor in use. N)r<)r9rhs r2r*"AttentionModuleMixin.get_processors&>> !&r4backendcSSKJn URR5Vs1sHo3RiM nnX;a"[ SU<S3SR U5-5eU"UR55nXRl gs snf)Nr)AttentionBackendNamez `backend=z ` must be one of the following: z, ) attention_dispatchrm __members__valuesvaluerGjoinlowerr<_attention_backend)r9rkrmxavailable_backendss r2set_attention_backend*AttentionModuleMixin.set_attention_backendsw?? ""=1r4use_xla_flash_attentionpartition_spec.chU(a[5(d [S5eURS5 g)a Set whether to use XLA flash attention from `torch_xla` or not. Args: use_xla_flash_attention (`bool`): Whether to use pallas flash attention kernel from `torch_xla` or not. partition_spec (`Tuple[]`, *optional*): Specify the partition specification if using SPMD. Otherwise None. is_flux (`bool`, *optional*, defaults to `False`): Whether the model is a Flux model. ztorch_xla is not available _native_xlaN)rr|rw)r9rris_fluxs r2set_use_xla_flash_attention0AttentionModuleMixin.set_use_xla_flash_attentions*" #)++!">?? ""=1r4'use_memory_efficient_attention_xformers attention_opcU(a[5(d [SSS9e[RR 5(d [ S5e[5(aCSnUbUupEUR tp6[R"SSUS9n[R"XwU5nURS5 gg![anUeSnAff=f) ax Set whether to use memory efficient attention from `xformers` or not. Args: use_memory_efficient_attention_xformers (`bool`): Whether to use memory efficient attention from `xformers` or not. attention_op (`Callable`, *optional*): The attention operation to use. Defaults to `None` which uses the default attention operation from `xformers`. zeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformersxformers)r&zvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU N)rr (cudadevicedtype) rModuleNotFoundErrorr6r is_availablerGSUPPORTED_DTYPESrandnxopsmemory_efficient_attention Exceptionrw) r9rrrop_fwop_bwrSqes r2+set_use_memory_efficient_attention_xformers@AttentionModuleMixin.set_use_memory_efficient_attention_xformerss 3(**){#ZZ,,.. / ,.. $'3+7LE(-(>(>IE!KK 6O ;;A!D**:61 3*!GsAB55 C?CCcf [USS5(agURRRRnURRRR n[ US5(GakUR(GaY[R"URRRURRR/5nURSnURSn[R"XEURXS9UlUR RR#U5 [ US5(aUR(ay[R"URR$RURR$R/5nUR R$R#U5 GO[R"URRRURRRURRR/5nURSnURSn[R"XEURXS9UlUR&RR#U5 [ US5(aUR(a[R"URR$RURR$RURR$R/5nUR&R$R#U5 [US S5Gb[US S5Gb[US S5Gb[R"UR(RRUR*RRUR,RR/5nURSnURSn[R"XEUR.XS9UlUR0RR#U5 UR.(a[R"UR(R$RUR*R$RUR,R$R/5nUR0R$R#U5 S Ulg) zU Fuse the query, key, and value projections into a single projection for efficiency. fused_projectionsFNis_cross_attentionrr)biasrruse_bias add_q_proj add_k_proj add_v_projT)getattrto_qweightdatarrr-rr6catto_kto_vshaper7Linearrto_kvcopy_rto_qkvrrradded_proj_bias to_added_qkvr)r9rrconcatenated_weights in_features out_featuresconcatenated_biass r2rR%AttentionModuleMixin.fuse_projectionss 4,e 4 4 !!&&--   %%++ 4- . .43J3J3J#(99dii.>.>.C.CTYYEUEUEZEZ-[#\ .44Q7K/55a8L;4==Y_mDJ JJ   # #$8 9tZ((T]]$)IItyy~~/B/BDIINNDWDW.X$Y! %%&78$)99dii.>.>.C.CTYYEUEUEZEZ\`\e\e\l\l\q\q-r#s .44Q7K/55a8L))KDMMZ`nDK KK   $ $%9 :tZ((T]]$)IItyy~~/B/BDIINNDWDWY]YbYbYgYgYlYl.m$n!   &&'89 D, - 9lD1=lD1=#(99'',,doo.D.D.I.I4??KaKaKfKfg$ /44Q7K/55a8L " 0D0DV!D     $ $ * *+? @##$)II__))..0D0D0I0I4??K_K_KdKde%!!!&&,,->?!%r4c[USS5(dg[US5(a [US5 [US5(a [US5 [US5(a [US5 SUlg)zL Unfuse the query, key, and value projections back to separate projections. rFNrrr)rr-delattrr)r9s r2rX'AttentionModuleMixin.unfuse_projections5sf t0%88  4 " " D( # 4 ! ! D' " 4 ( ( D. )!&r4 slice_sizec[US5(a.Ub+XR:a[SUSURS35eSnUbURS5nUcUR 5nUR U5 g)z Set the slice size for attention computation. Args: slice_size (`int`): The slice size for attention computation. sliceable_head_dimNz slice_size z has to be smaller or equal to r,sliced)r-rrG_get_compatible_processordefault_processor_clsr?)r9rr<s r2set_attention_slice(AttentionModuleMixin.set_attention_sliceJs 4- . .:3Ij[r[rNr{:,6UVZVmVmUnnopq q   !66x@I  224I 9%r4tensorcURnURup4nURX2-X$U5nURSSSS5RX2-XEU-5nU$)z Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size // heads, seq_len, dim * heads]`. Args: tensor (`torch.Tensor`): The tensor to reshape. Returns: `torch.Tensor`: The reshaped tensor. rr r)headsrreshapepermute)r9r head_size batch_sizeseq_lendims r2batch_to_head_dim&AttentionModuleMixin.batch_to_head_dimas_JJ #)<< S 7SQ1a+33J4KW\eVef r4out_dimcURnURS:XaURupEnSnOURupGpVURXEU-X6U-5nUR SSSS5nUS:XaURXC-XW-Xc-5nU$)z Reshape the tensor for multi-head attention processing. Args: tensor (`torch.Tensor`): The tensor to reshape. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the tensor. Returns: `torch.Tensor`: The reshaped tensor. rrrr )rndimrrr)r9rrrrrr extra_dims r2head_to_batch_dim&AttentionModuleMixin.head_to_batch_dimqsJJ ;;! '-|| $JI28,, /J7 i,?S\L\]1a+ a<^^J$:G