+h'SSKrSSKJrJrJrJr SSKrSSKJr SSK J r J r J r J r SSKJr \R "\5r"SS5rg)N)DictListTupleUnion) AttentionAttentionProcessor"PAGCFGIdentitySelfAttnProcessor2_0PAGIdentitySelfAttnProcessor2_0)loggingc\rSrSrSrSrSrSSjrSr\ "5\ "544S\ \ \ \ 4S\\\44S jjr\S \4S j5r\S \4S j5r\S \4S j5r\S \4Sj5r\S \\ \44Sj5rSrg)PAGMixin!zZMixin class for [Pertubed Attention Guidance](https://huggingface.co/papers/2403.17377v1).cJURnUc [S5eU(aUSOUSn[US5(a URnO URnS[ R S[4SjnS nUHn/n UR5HeupU"U 5(dM[R"X5cM-U"X5(aM<[RS U 35 U RU 5 Mg [U 5S:Xa[S U35eU H n XKlM M g) z1 Set the attention processor for the PAG layers. NzNo PAG attention processors have been set. Set the attention processors by calling `set_pag_applied_layers` and passing the relevant parameters.runetmodulereturncR[U[5=(a UR(+$)zA Check if the module is self-attention module based on its name. ) isinstanceris_cross_attention)rs [/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/pipelines/pag/pag_utils.py is_self_attn6PAGMixin._set_pag_attn_processor..is_self_attn5sfi0R9R9R5R RcURS5SnURS5SnUR5=(a UR5=(a X:H$)N.)split isnumeric)layer_idnames ris_fake_integral_match@PAGMixin._set_pag_attn_processor..is_fake_integral_match;sJ~~c*2.H::c?2&D%%'QDNN,<QAQ QrzApplying PAG to layer: z6Cannot find PAG layer to set attention processor for: )_pag_attn_processors ValueErrorhasattrr transformernnModulebool named_modulesresearchloggerdebugappendlen processor) selfpag_applied_layersdo_classifier_free_guidancepag_attn_processors pag_attn_procmodelrr#r!target_modulesr"rs r_set_pag_attn_processor PAGMixin._set_pag_attn_processor$s&#77  &c 3N+A.SfghSi 4 #yyE#//E S St S  R +HN % 3 3 5 !(( (1=28BBLL#:4&!AB"))&1!6>"a' #YZbYc!dee(#0 )++rcUR(a)URURSU- -- nUS:aSnU$UR$)zL Get the scale factor for the perturbed attention guidance at timestep `t`. ir)do_pag_adaptive_scaling pag_scalepag_adaptive_scale)r4t signal_scales r_get_pag_scalePAGMixin._get_pag_scaleXsG  ' '>>D,C,Ctax,PPLa  >> !rcURU5nU(a'URS5upxn UX8U- --XhU - --nOURS5upXX- --nU(aX4$U$)a> Apply perturbed attention guidance to the noise prediction. Args: noise_pred (torch.Tensor): The noise prediction tensor. do_classifier_free_guidance (bool): Whether to apply classifier-free guidance. guidance_scale (float): The scale factor for the guidance term. t (int): The current time step. return_pred_text (bool): Whether to return the text noise prediction. Returns: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]: The updated noise prediction tensor after applying perturbed attention guidance and the text noise prediction. r)rCchunk) r4 noise_predr6guidance_scalerAreturn_pred_textr?noise_pred_uncondnoise_pred_textnoise_pred_perturbs r#_apply_perturbed_attention_guidance,PAGMixin._apply_perturbed_attention_guidancees"''* &EOEUEUVWEX B 0B! 6G$GHI1CCDE  3=2B2B12E /O(8\+]]J . .rcr[R"U/S-SS9nU(a[R"X!/SS9nU$)aj Prepares the perturbed attention guidance for the PAG model. Args: cond (torch.Tensor): The conditional input tensor. uncond (torch.Tensor): The unconditional input tensor. do_classifier_free_guidance (bool): Flag indicating whether to perform classifier-free guidance. Returns: torch.Tensor: The prepared perturbed attention guidance tensor. rFr)dim)torchcat)r4conduncondr6s r%_prepare_perturbed_attention_guidance.PAGMixin._prepare_perturbed_attention_guidances4yy$!+ &99f^3D rr5r7cv[US5(dSUl[U[5(dU/nUb/[U[5(a[ U5S:wa [ S5e[[ U55H3n[X[5(aM[ S[X535e Xl X lg)a Set the self-attention layers to apply PAG. Raise ValueError if the input is invalid. Args: pag_applied_layers (`str` or `List[str]`): One or more strings identifying the layer names, or a simple regex for matching multiple layers, where PAG is to be applied. A few ways of expected usage are as follows: - Single layers specified as - "blocks.{layer_index}" - Multiple layers as a list - ["blocks.{layers_index_1}", "blocks.{layer_index_2}", ...] - Multiple layers as a block name - "mid" - Multiple layers as regex - "blocks.({layer_index_1}|{layer_index_2})" pag_attn_processors: (`Tuple[AttentionProcessor, AttentionProcessor]`, defaults to `(PAGCFGIdentitySelfAttnProcessor2_0(), PAGIdentitySelfAttnProcessor2_0())`): A tuple of two attention processors. The first attention processor is for PAG with Classifier-free guidance enabled (conditional and unconditional). The second attention processor is for PAG with CFG disabled (unconditional only). r%NrFz,Expected a tuple of two attention processorsz:Expected either a string or a list of string but got type ) r'r%rlisttupler2r&rangestrtyper5)r4r5r7is rset_pag_applied_layersPAGMixin.set_pag_applied_layerss4t344(,D %,d33"4!5   *1599SAT=UYZ=Z !OPPs-./A03S99 PQUVhVkQlPmn0 #5$7!rrcUR$)z:Get the scale factor for the perturbed attention guidance.) _pag_scaler4s rr?PAGMixin.pag_scalesrcUR$)zCGet the adaptive scale factor for the perturbed attention guidance.)_pag_adaptive_scalercs rr@PAGMixin.pag_adaptive_scales'''rcURS:=(a. URS:=(a [UR5S:$)zNCheck if the adaptive scaling is enabled for the perturbed attention guidance.r)rfrbr2r5rcs rr> PAGMixin.do_pag_adaptive_scalings9''!+h!0ChDLcLcHdghHhhrc^URS:=(a [UR5S:$)z5Check if the perturbed attention guidance is enabled.r)rbr2r5rcs rdo_perturbed_attention_guidance(PAGMixin.do_perturbed_attention_guidances("Gs4+B+B'Ca'GGrc|URc0$URVs1sHoRiM nn0n[US5(a URnO)[US5(a URnO [ S5eUR R5HupVURU;dMXcU'M U$s snf)z Returns: `dict` of PAG attention processors: A dictionary contains all PAG attention processors used in the model with the key as the name of the layer. rr(zNo denoiser module found.)r% __class__r'rr(r&attn_processorsitems)r4xvalid_attn_processors processorsdenoiser_moduler"procs rr7PAGMixin.pag_attn_processorss  $ $ ,I6:6O6O P6O6O P  4 "iiO T= ) )"..O89 9)99??AJD~~!66#'4 B!!QsB9)r%r5N)F)__name__ __module__ __qualname____firstlineno____doc__r;rCrNrVr r rr\rrr r_propertyfloatr?r@r+r>rkrr7__static_attributes__rrrr!se21h "\a@. / 0 + -N *8!#tCy.1*8##57I#IJ*8X5(E((iiiHHHT#/A*A%Brr)r-typingrrrrrRtorch.nnr)models.attention_processorrr r r utilsr get_loggerrwr/rrrrrsC ++     H %RRr