oih$~SrSSKJr SSKJr SSKJrJrJr SSK J r SSK J r SSK 7 SS KJrJr \(aSS KJrJr \S LrS \ R=(d S ;rSrSrSrSr\(a\R8R:OS r\"SS55r\"SS55r SSSjjr!SSjr"/SQr#g )z=Shared helpers for attention backend selection and execution.) annotations) dataclass)AnyOptionalTuple)Tensor)scaled_dot_product_attention)*) build_sdpa_packed_attention_mask build_xformers_block_causal_mask)flash_attn_funcflash_attn_varlen_funcN enable_gqa flash_varlen flash_densexformerssdpacp\rSrSr%SrS\S'S\S'S\S'SrS \S 'SrS \S 'SrS \S 'Sr S \S 'Sr g)AttentionConfig0a Per-layer attention metadata. NOTE(djsaunde): I had originally intended this to be populated once per layer, but we're currently constructing it on every forward pass since it can possibly be invalid from one forward pass to the next (e.g., switching from training to inference). For now, I'm keeping separate from AttentionContext for the sake of better grouping of params. strbackendint n_kv_headsn_groupsNzOptional[dict[str, Any]]flash_dense_kwargsflash_varlen_kwargs sdpa_kwargsxformers_kwargs) __name__ __module__ __qualname____firstlineno____doc____annotations__rrr r!__static_attributes__r"Z/home/james-whalen/.local/lib/python3.13/site-packages/unsloth/utils/attention_dispatch.pyrr0sFLOM37074818,0K)004O-4r*rc\rSrSr%SrS\S'S\S'S\S'S\S'S\S'S \S 'S \S 'S \S'S\S'SrS\S'Srg)AttentionContextEz(Per-call info required to run attention.rbszq_len kv_seq_lenn_headshead_dimbool requires_gradz$Optional[Tuple[Tensor, Tensor, int]]seq_infozOptional[Tensor]attention_maskz Optional[Any] causal_maskNz Optional[int]sliding_windowr")r#r$r%r&r'r(r9r)r"r*r+r-r-Es@2 H JO LM22$$$(NM(r*r-cl[(aU(a[$[$[(a[$[ $)z@Return attention backend based on availability / priority order.)HAS_FLASH_ATTENTION FLASH_VARLEN FLASH_DENSE HAS_XFORMERSXFORMERSSDPA) use_varlens r+select_attention_backendrBUs(   | Kr*c n URnU[:Xa$URc[(a[O[ nUR =(d 0nUR=(d 0nUR=(d 0nUR=(d 0n URn URn URn URn URnURnUR nU[:XaUR#SS5R%X-X5nUR#SS5R%X-UR&U 5nUR#SS5R%X-UR&U 5nURunnn[)UUUUUUU40UD6R+XX5$U[:XaTUR#SS5nUR#SS5nUR#SS5n[-UUU40UD6R%XX5$U[.:XGa[1URUUR2S9nUR#SS5nUR#SS5nUR#SS5nUnUnUnUR4S:waUR+XUR&SU 5nUR+XUR&SU 5nUR7XUR&UR4U 5nUR7XUR&UR4U 5nU(a%UR%XX5nUR%XX5nO(UR+XUR&UR4U 5n[8SL=(a [;U[85nUR4S:waU(aU(dUR+SX-UR&UR4U 5nUR+SX-UR&UR4U 5nUR+SX-UR&UR4U 5nO?UR+SX-X5nUR+SX-X5nUR+SX-X5n[=UUU4SU0U D6nUR4S:waCU(d<UR+XUR&UR4U 5nUR%XX5nU$UR+XX5nU$UR>n Sn!URb.U c+[AURURBURDUS9n O.URFSn"URFSn#U SL=(a U"U#:Hn![IU5n$U$RKS U 5 U$RKS U!5 [L(a>U$RKS UR4S:g5 [OX#U40U$D6nUR#SS5$UnUnUR4S:waUSS2SS2SSS2SS24R7XR&UR4X5nUSS2SS2SSS2SS24R7XR&UR4X5nUR%XX5nUR%XX5n[OURQ5URQ5URQ540U$D6nUR#SS5RQ5$) a  Run attention using config / context info. Backend choice is prioritized for speed: FlashAttention when installed (`flash_varlen` for packed/variable-length inputs with `seq_info`, otherwise dense flash), then xFormers if flash is unavailable, with PyTorch SDPA as the final fallback (e.g., CPU or no fused kernels). Varlen flash is preferred when packing metadata is present because it avoids padding and keeps peak memory low. xFormers and SDPA can also handle packed batches (we pass a block-diagonal mask into each). Nr )r9 base_mask attn_biasF)dtypedevicer9 attn_mask is_causalr))rr<r6r;r=r@rrr r!r/r2r0r3r1r5r9 transposereshaperrviewrr?r r8rexpandXFORMERS_BLOCK_DIAG_CLS isinstancexformers_attentionr7r rGrHshapedict setdefault SDPA_HAS_GQAr contiguous)%configcontextQKVrrrr r!r/r2r0r3r1r5r9Q_fK_fV_f_ cu_seqlens max_seqlenQ_tK_tV_trFK_modV_modQ_mod has_blockout local_maskis_causal_local q_len_local k_len_localkwargss% r+ run_attentionrpbs@*nnG,7#3#3#;!4!4+$228b 44:$$*K,,2O ++CooG MMEH##J))M++N,kk!Q'' WGkk!Q'' V5F5FQkk!Q'' V5F5FQ$+$4$4!:z%         "   $s7 - . K kk!Qkk!Qkk!QsCC0BCKK    H 4   +++ kk!Qkk!Qkk!Q ??a HHSf.?.?HMEHHSf.?.?HMELL!2!2FOOXELL!2!2FOOXE cwI cwI 1 16??H,47 J .=  ??a I  s{F$5$5v s'):):FOOX s'):):FOOX 1ck7E 1c&6J 1c&6J    "     ??a  ((3v'8'8&//8TC++c'9  !/ J''"+K''"+K(D0O[K5OOk"+z2+7 <   lFOOq,@ A.qQA&AC==A& & ??a aD!Q&'..&&EaD!Q&'..&&EMM# EEMM# EE* LLN            }}Q"--//r*)rr-rBrp)F)rAr4returnr) rXrrYr-rZrr[rr\rrqr)$r' __future__r dataclassesrtypingrrrtorchrtorch.nn.functionalr models._utils utils.packingr r r; flash_attnrrrr>rVr<r=r?r@rFBlockDiagonalCausalMaskrPrr-rBrp__all__r"r*r+r|s D"!''< Bt#  < D D JK    3?H..D  55 5(  ) )  ) h0 h0h0 h0  h0  h0 h0V r*