+hOpSSKrSSKrSSKJr SSKrSSKJr SS\ 4Sjjr \RRRSS4S\ S\ 4Sjjr"SS \R5r"S S \R5r"S S \R5r"SS\R5r"SS\R5rg)Nkey_chunk_sizec^^^^^^^^^TRSSunmmTRSm[TU5mT[R"T5- m[R "[ RSS9U4Sj5mUUUUUUUU4Sjn[ RRU[R"SUT5S 9upxn [R"U SS S 9n [R"X- 5n U[R"U SS 9-nX-nURSS 9n [R"US5RSS 9n X- $) zBMulti-head dot product attention with a limited number of queries.NF) prevent_csecF>[R"SXTS9n[R"USSS9n[RR U5n[R "X4- 5n[R"SX%TS9n[R"SU5nXeRSS9U4$) Nz...qhd,...khd->...qhk) precisionrTaxiskeepdimsz...vhf,...qhv->...qhfz ...qhk->...qhr )jnpeinsummaxjaxlax stop_gradientexpsum)querykeyvalue attn_weights max_score exp_weights exp_valuesr s Y/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/models/attention_flax.pysummarize_chunk/_query_chunk_attention..summarize_chunkszz"95QZ[ GGLrDA GG)))4 ggl67 ZZ 7W`a JJ : OOO4i@@c ^>[RRTS/TRS- -USS/-[ TR SS5TTT/-S9n[RRT S/T RS- -USS/-[ T R SS5TTT /-S9nT"TX5$)Nrroperand start_indices slice_sizes)rr dynamic_slicendimlistshape) chunk_idx key_chunk value_chunk k_featuresrr num_headsrr v_featuresrs r chunk_scanner-_query_chunk_attention..chunk_scanner+sGG))#A.)Q1BBSYYs^, :/VV* gg++#a0Iq!3DDU[["-..)Z1XX, ui==r!r)fxsTr r)r+minrsqrt functoolspartialr checkpointrmaparangerr expand_dimsr)rrrr rnum_kvr2 chunk_values chunk_weights chunk_max global_max max_diffs all_values all_weightsr/r0rr1s````` @@@@r_query_chunk_attentionrFs$'IIbcN!FIzRJ0N CHHZ( (Es~~59 A: A>>".1WW[[=SZZXY[acqMr[-s*LT:J ./ICOOIB77LM!!q!)J//-488a8@K  ##r!iquery_chunk_sizec ^^^^^^^ ^ ^ TRSSum m m UUU U UU UUU4 Sjn[RRUSS[R "T T- 5S9upx[ R"USS9$)a Flax Memory-efficient multi-head dot product attention. https://huggingface.co/papers/2112.05682v2 https://github.com/AminRezaei0x443/memory-efficient-attention Args: query (`jnp.ndarray`): (batch..., query_length, head, query_key_depth_per_head) key (`jnp.ndarray`): (batch..., key_value_length, head, query_key_depth_per_head) value (`jnp.ndarray`): (batch..., key_value_length, head, value_depth_per_head) precision (`jax.lax.Precision`, *optional*, defaults to `jax.lax.Precision.HIGHEST`): numerical precision for computation query_chunk_size (`int`, *optional*, defaults to 1024): chunk size to divide query array value must divide query_length equally without remainder key_chunk_size (`int`, *optional*, defaults to 4096): chunk size to divide key and value array value must divide key_value_length equally without remainder Returns: (`jnp.ndarray`) with shape of (batch..., query_length, head, value_depth_per_head) rNc > [RRT S/T RS- -USS/-[ T R SS5[ T T5TT/-S9nUT -[UTT TTS94$)Nrr#rr$)rrrr r)rrr(r)r*r+r6rF) r,_ query_chunkrrr0num_qr q_featuresrrGrs rr25jax_memory_efficient_attention..chunk_scannerasgg++3%**q.1iA5FFU[["-.#6F2NPY[e1ff,  ( ( "!s%9]k   r!r)r4initr5lengthr)r+rrscanmathceilr concatenate) rrrr rGrr2rJresr0rLrMs `````` @@@rjax_memory_efficient_attentionrVJsp*$);;rs#3 E9j   WW\\   yy!112 FA ??3R ((r!c\rSrSr%Sr\\S'Sr\\S'Sr\\S'Sr \ \S 'S r \ \S 'S r \ \S '\Rr\R \S 'SrSrSrSSjrSrg) FlaxAttentionza A Flax multi-head attention module as described in: https://huggingface.co/papers/1706.03762 Parameters: query_dim (:obj:`int`): Input hidden states dimension heads (:obj:`int`, *optional*, defaults to 8): Number of heads dim_head (:obj:`int`, *optional*, defaults to 64): Hidden states dimension inside each head dropout (:obj:`float`, *optional*, defaults to 0.0): Dropout rate use_memory_efficient_attention (`bool`, *optional*, defaults to `False`): enable memory efficient attention https://huggingface.co/papers/2112.05682 split_head_dim (`bool`, *optional*, defaults to `False`): Whether to split the head dimension into a new axis for the self-attention computation. In most cases, enabling this flag should speed up the computation for Stable Diffusion 2.x and Stable Diffusion XL. dtype (:obj:`jnp.dtype`, *optional*, defaults to jnp.float32): Parameters `dtype` query_dimheads@dim_headdropoutFuse_memory_efficient_attentionsplit_head_dimdtypecURUR-nURS-Ul[R"USUR SS9Ul[R"USUR SS9Ul[R"USUR SS9Ul[R"URUR SS9Ul [R"URS 9Ul g) NgFto_q)use_biasrcnameto_kto_vto_out_0)rcrgrate)r^r\scalennDensercrrrrZ proj_attnDropoutr` dropout_layerself inner_dims rsetupFlaxAttention.setupsMMDJJ. ]]D( XXi%tzzPVW 88ITZZfUXXi%tzzPVW $.. TZZT\\:r!cURup#nURnURX#XTU-5n[R"US5nURX%-X4U-5nU$N)rr#r+r\reshaper transposerttensor batch_sizeseq_lendim head_sizes rreshape_heads_to_batch_dim(FlaxAttention.reshape_heads_to_batch_dims[#)<< SJJ  Yy@PQv|4 6 AQR r!cURup#nURnURX%-XSU5n[R"US5nURX%-X4U-5nU$ryr|rs rreshape_batch_dim_to_heads(FlaxAttention.reshape_batch_dim_to_headssZ#)<< SJJ  7SQv|4 7 /R r!Nc6UcUOUnURU5nURU5nURU5nUR(aURSn[ R "XGSURUR45n[ R "XWSURUR45n [ R "XgSURUR45n O3URU5nURU5n URU5n UR(aURSSS5nU RSSS5n U RSSS5n URSn U S-S:Xa[U S- 5n O;U S-S:Xa[U S- 5n O#U S-S:Xa[U S- 5n O [U 5n [XXS S 9nURSSS5nURU5nGOUR(a[ R"S X5n O[ R"S X5n XR -n ["R$"XR(aSOSS 9nUR(aW[ R"SX5nURSn[ R "XSURUR-45nO([ R"SX5nURU5nUR'U5nUR)XS9$)Nrrr{rzrr]i@)rGrzb t n h, b f n h -> b n f tzb i d, b j d->b i jrzb n f t, b t n h -> b f n hzb i j, b j d -> b i d deterministic)rrrrbr+rr}r\r^rrar~intrVrrrmrnsoftmaxrprr)rt hidden_statescontextr query_projkey_proj value_projb query_states key_states value_statesflatten_latent_dimrGattention_scoresattention_probss r__call__FlaxAttention.__call__s#*?-ZZ . 88G$ZZ(   ##A&A;;zr4::t}}3UVLX2tzz4==/QRJ;;zr4::t}}3UVL:::FL88BJ:::FL  . .'11!Q:L#--aA6J'11!Q:L ".!3!3B!7 !B&!+#&'9B'>#? #b(A-#&'9B'>#? #a'1,#&'9A'=#> #&'9#: :,jrM*33Aq!