΅i|*!tSrSSKrSSKJr SSKJrJr SSKr\R"\ 5r SS/r \"SS9S \ S \ 4S j5r"S S\5r\R R#S 0S9S,S\R$S\R$S\R$S\R$S\R$S\ S\ S\ S\S-S \\R$\R$\R$44Sjj5r\R,S,S\R$S\R$S\R$S\R$S\R$S\ S\ S\ S\S-S \\R$\R$\R$44Sjj5rS-S\R$S\R$S\R$S\R$S\R$S\ S\ S\ S\S-S\S-S \R$\\R$\R$4-4SjjrS\S\\S4S\S S4S jr\R R#S!0S9S.S"\R$S\R$S\R$S\R$S#\R$S$\R$S\R$S\R$S\ S\ S\ S%\R$S\S-S \\R$\R$\R$44S&jj5r\R,S.S"\R$S\R$S\R$S\R$S#\R$S$\R$S\R$S\R$S\ S\ S\ S%\R$S\S-S \\R$\R$\R$44S'jj5rS\S"\R$S(\R$S)\R$S \\R$S-S44 S*jr\R;\\S+9 g)/z Variable-length attention implementation using Flash Attention. This module provides a high-level Python interface for variable-length attention that calls into the optimized Flash Attention kernels. N) lru_cache)Any NamedTuple varlen_attn AuxRequest)maxsize device_indexreturncg)z;Cache device capability check to avoid repeated CUDA calls.F)r s S/home/james-whalen/.local/lib/python3.13/site-packages/torch/nn/attention/varlen.py_should_use_cudnnrs c(\rSrSr%SrSr\\S'Srg)rz Request which auxiliary outputs to compute from varlen_attn. Each field is a boolean indicating whether that auxiliary output should be computed. Flser N) __name__ __module__ __qualname____firstlineno____doc__rbool__annotations____static_attributes__r rrrrs Crztorch_attn::_varlen_attn) mutates_argsquerykeyvaluecu_seq_qcu_seq_kmax_qmax_k is_causalscalec UR=(a [URR5n U (aW[R S5 [ RRRUUUSUUUUSSUSUS9 n U SU SU S pn OK[R S 5 [ RRRUUUUUUUSUSUS 9 upn n[ R"S [ RURS 9nXU4$)z Private custom op for variable-length attention. This is the internal implementation. Users should use the public varlen_attn function instead. #Using cuDNN backend for varlen_attnNTFr%r-Using Flash Attention backend for varlen_attn)return_debug_maskr%dtypedevice) is_cudarr2indexloginfotorchopsaten_cudnn_attention_forward_flash_attention_forwardzerosuint64)rrrr r!r"r#r$r% use_cudnnresultoutput softmax_lse rng_state_ rng_state_s r _varlen_attnrE$s$ G"3ELL4F4F"GI 6788            9  *0F1IvayYY @A/4yy~~/V/V         #0W 0 ,Y1 ELLJ  **rc [R"U5n URS5n URS5n [RR(aEURS5S- n [R "XU4[R URS9n O/[R "X4[R URS9n [R "S[RURS9nXU4$)z Fake implementation for meta tensor computation and tracing. Based on the 3D varlen path from meta__flash_attention_forward: - query shape: (total, num_heads, head_dim) - logsumexp shape: (num_heads, total_q) rr*r0r.) r7 empty_likesizeversionhipemptyfloatr2r=)rrrr r!r"r#r$r%r@total_q num_heads batch_size logsumexprBs r_varlen_attn_fakerQas(  e $FjjmG 1 I }}]]1%) KK E *%++ell KK   ELL  D U\\JI i ''r return_auxc [RRRXX#XEXgU 5 upn UbUR(aX4$U $)a Compute variable-length attention using Flash Attention. This function is similar to scaled_dot_product_attention but optimized for variable-length sequences using cumulative sequence position tensors. Args: query (Tensor): Query tensor; shape :math:`(T_q, H, D)` key (Tensor): Key tensor; shape :math:`(T_k, H, D)` value (Tensor): Value tensor; shape :math:`(T_k, H, D)` cu_seq_q (Tensor): Cumulative sequence positions for queries; shape :math:`(N+1,)` cu_seq_k (Tensor): Cumulative sequence positions for keys/values; shape :math:`(N+1,)` max_q (int): Maximum query sequence length in the batch. max_k (int): Maximum key/value sequence length in the batch. is_causal (bool, optional): If set to True, applies causal masking (default: False). return_aux (Optional[AuxRequest]): If not None and ``return_aux.lse`` is True, also returns the logsumexp tensor. scale (float, optional): Scaling factor for attention scores Returns: output (Tensor): Output tensor from attention computation; shape :math:`(T_q, H, D)`. If ``return_aux`` is not None and ``return_aux.lse`` is True: lse (Tensor): Log-sum-exp of attention scores; shape :math:`(T_q, H)`. Shape legend: - :math:`N`: Batch size - :math:`T_q`: Total number of query tokens in the batch (sum of all query sequence lengths) - :math:`T_k`: Total number of key/value tokens in the batch (sum of all key/value sequence lengths) - :math:`H`: Number of attention heads - :math:`D`: Head dimension Example:: >>> # xdoctest: +REQUIRES(env:TORCH_DOCTEST_CUDA) >>> batch_size, max_seq_len, embed_dim, num_heads = 2, 512, 1024, 16 >>> head_dim = embed_dim // num_heads >>> seq_lengths = [] >>> for _ in range(batch_size): ... length = torch.randint(1, max_seq_len // 64 + 1, (1,)).item() * 64 ... seq_lengths.append(min(length, max_seq_len)) >>> seq_lengths = torch.tensor(seq_lengths, device="cuda") >>> total_tokens = seq_lengths.sum().item() >>> >>> # Create packed query, key, value tensors >>> query = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> key = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> value = torch.randn( ... total_tokens, num_heads, head_dim, dtype=torch.float16, device="cuda" ... ) >>> >>> # Build cumulative sequence tensor >>> cu_seq = torch.zeros(batch_size + 1, device="cuda", dtype=torch.int32) >>> cu_seq[1:] = seq_lengths.cumsum(0) >>> max_len = seq_lengths.max().item() >>> >>> # Call varlen_attn >>> output = varlen_attn( ... query, key, value, cu_seq, cu_seq, max_len, max_len, is_causal=False ... ) )r7r8 torch_attnrEr) rrrr r!r"r#r$rRr%outrrCs rrrsHV))&&33 EXeKCa*..x Jrctxinputs.r@c vUu p4pVpxpn UupnURX4XVX|X5 XlXlXlXlgN)save_for_backwardr"r#r$r%)rVrWr@rrrr r!r"r#r$r%rUrrBs r_setup_contextr[sDLRIE%E Ci%excUIIMIrz!torch_attn::_varlen_attn_backwardgrad_outrUrrBc [R"SURS9n UR=(a [ URR 5nU(aO[ RS5 [RRRUUUUUUUUUU SU U U U S9unnnON[ RS5 [RRRUUUUUUUUUU SU U U U S9unnnUUU4$)Nr)r2r'r(r)r,) r7rKr2r3rr4r5r6r8r9_cudnn_attention_backward_flash_attention_backward)r\rrrrUrr r!r"r#r$rBr%unusedr>dqdkdvs r_varlen_attn_backwardrds [[5<< 0F G"3ELL4F4F"GI 67YY^^==              > B$ @AYY^^==              > B" r2:rc [R"U5n [R"U5n[R"U5nXU4$)z> Fake implementation for meta tensor computation and tracing. )r7rG)r\rrrrUrr r!r"r#r$rBr% grad_querygrad_key grad_values r_varlen_attn_backward_fakeri%s?(!!%(J$H!!%(J  ++rgrad_lsegrad_rngc URupEpgppURn URn URnURn[ R RRUUUUU U UUU U UU U5 unnnUUUSSSSSS4 $rY) saved_tensorsr"r#r$r%r7r8rTrd)rVr\rjrkrrrr r!rUrrBr"r#r$r%rarbrcs r _backwardrn@sBEARAR>E IIE IIE I IIE%%;;        JBB r2tT4tT 99r) setup_context)FN)FNNrY)rlogging functoolsrtypingrrr7 getLoggerrr5__all__intrrrlibrary custom_opTensorrLtuplerE register_fakerQrr[rdrirnregister_autogradr rrr|s" ! , ' 1CD 3"E9+ <<9+ 9+ <<9+ll 9+ ll 9+  9+ 9+9+ 4<9+ 5<<u|| 349+F9+x%( <<%( %( <<%(ll %( ll %(  %( %(%( 4<%( 5<<u|| 34%(%(`$(P <<P P <<Pll P ll P  P PPT!P 4<P \\E%,, 455Pf  U38_ c d <2N8ll8 <<8 8 << 8  8  8ll8ll8 8 88||8 4<8 5<<u|| 348O8v$$,ll, <<, , << ,  ,  ,ll,ll, , ,,||, 4<, 5<<u|| 34,%,4: : :05 :HM : 5<<$  #$:8 yGr