bCiSrSSKJrJrJr SSKrSSKrSSKJr SSK Js J r SSK J r Jr SSKJr SSKJrJr SSKJr SS KJr SS KJrJrJr SS KJrJr S S KJ r J!r! S SK"J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r* SSK+J,r,J-r-J.r.J/r/J0r0 \Rb"\25r3SDS\44Sjjr5SES\RlS\4S\4S\44Sjjr7S\RlS\RpS\4S\9S\4S\Rl4 Sjr:SS\Rv4S\RlS \4S!\4S"\\Rl\Rl44S%jjr?S&\RlS'\\4S\Rl4S(jr@"S)S*\'5rA"S+S,\(5rB"S-S.\ 5rC"S/S0\%5rD"S1S2\)5rE"S3S4\&5rF\"S5S6\$55rG"S7S8\G5rH"S9S:\G5rI"S;S<\G5rJ"S=S>\G5rK"S?S@\G5rL"SASB\#5rM/SCQrNg)Fz>> tokens = torch.tensor([[1, 2, 3], [4, 5, 6]]) >>> hashes = rolling_polynomial_hash(tokens, prime=31) >>> # hash[0] = 1*31^0 + 2*31^1 + 3*31^2 >>> # hash[1] = 4*31^0 + 5*31^1 + 6*31^2 dtypedevicer(dim)torchtensorint64r(arangeshapesum) token_tensorr$ prime_tensorpowers prime_powerss ]/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/blt/modular_blt.pyrolling_polynomial_hashr89s].<<U[[ATATUL \\,,,R09L9L MF'L 99\0b 99 token_ids group_sizemax_hashcT[R"5 URupE[R"XAS- [RUR S9n[R "X`/SS9nURSUS5n[X5n X-n SSS5 U $!,(df  W $=f)z1Hash token groups and map to range [0, max_hash].rr&r+N) r-no_gradr1zerosr/r(catunfoldr8) r:r;r$r< batch_sizeseq_lenpadding padded_tokenswindowshashes hash_valuess r7byte_group_hash_functionrIVs 'oo ++jq. T]TdTde 7"6A>  &&q*a8(8'     s A8B B'local_encoder_tokensencoder_hash_tok_embedding$encoder_hash_byte_group_nb_functionsencoder_hash_byte_group_sizeencoder_hash_byte_group_vocabreturnc/SQnURU5nSn[U5H>n Xi[U5-n UH%n [X X5n XU--n Xr"U 5- nUS- nM' M@ U$)z=Compute token embeddings enhanced with hash-based embeddings.) ʚ;l21Ai oYlvtl.l}glAul 0lTlAK l|rr) embed_tokensrangelenrI)rJ local_encoderrKrLrMrNprimes embeddings embedding_idxfunc_nbr$r;hash_idsoffset_hash_idss r7compute_hash_embeddingsr\hs F++,@AJM=>V,-6J/0DRWwH&9V)VVO 4_E EJ Q M 7? r9F patch_ids num_patchessequence_lengthpatches_as_queries cross_attn_kr'clURupgURnU(ajX-n Un [R"XS9R S5R S5R XaU5n UR S5R XaU5n OiUn X-n UR S5R XgU5n [R"XS9R S5R S5R XgU5n X:Hn U(aSOSnU R XNS9n XiU 4nU RU:wa[SU RSU35eU R S5n SU RU5- nURUR[R5[R"U5R5n U $) a Prepare cross-attention mask for patch-based attention, following mllama's robust approach. This function creates masks that control which patches can attend to which other patches, with support for query/key role swapping and cross-attention multipliers. Args: patch_ids (torch.Tensor): Tensor of shape [batch_size, seq_len] containing patch ids. num_patches (int): Total number of patches. sequence_length (int): Length of the sequence. patches_as_queries (bool): If True, patches are used as queries, otherwise as keys. cross_attn_k (int): Cross-attention multiplier for repeating patches. dtype (torch.dtype): Data type for the output mask. Returns: Tuple[torch.Tensor, torch.Tensor]: - cross_attention_mask: 4D tensor [batch_size, 1, q_len, kv_len] r*rr)rr+zCross attention mask shape z doesn't match expected g?) r1r(r-r0 unsqueezeexpandrepeat_interleave ValueErrorto masked_fillboolfinfomin)r]r^r_r`rar'rBrCr(q_lenkv_len q_patch_ids kv_patch_idscross_attention_mask repeat_dimexpected_shapeinverted_cross_attn_masks r7#_prepare_patch_cross_attention_maskrts4$//J   F*  LL 4 Yq\ Yr] VJW 5  !**1-44ZgV +))"-44Z+V LL 4 > >q A K KA N U UV`kv w  '6)bJ/AA,A_!0N!!^3)*>*D*D)EE]^l]m n  099!< #%9%<%UR U5 MQ UR U5 Mr [ SU55n [ R"X)4URURS9n [U5HFupU(dM[ R"XPRURS9XS[U524'MH U S:gRSS9R5U RS:aJU S:gRSS9R!5R 5R5S-n U SS2SU 24n U $)a Splits patch lengths into smaller segments if they exceed `max_patch_length`. Pads the result to uniform length across the batch. Args: patch_lengths (torch.Tensor): [batch_size, num_patches] tensor of patch lengths. max_patch_length (int, optional): Maximum allowed length per patch. Returns: torch.Tensor: [batch_size, max_len] tensor of split and padded patch lengths. Nrc38# UHn[U5v M g7fN)rT).0splitss r7 (process_patch_lengths..s6I&#f++Isr&r+r)sizeitemdivmodextendappendmaxr-r?r'r( enumerater.rTanyr2r1nonzero) rurvrB processedseqr{length full_chunks remaindermax_lenpaddedi last_nonzeros r7process_patch_lengthsrs##A&JI'lF[[]F%+F%E "K MM+,{: ;y i( #  6I66G [[*.m6I6IR_RfRf gFy)  6',||FBUBU^k^r^r'sFmF m# $* ! Q##% Q7! ((Q(/779==?DDFJ =L=() Mr9c\rSrSrSrg)BltMLPiN__name__ __module__ __qualname____firstlineno____static_attributes__rr9r7rrr9rc\rSrSrSrg) BltRMSNormi rNrrr9r7rr rr9rc\rSrSrSrg)BltRotaryEmbeddingirNrrr9r7rrrr9rc0^\rSrSrS\4U4SjjrSrU=r$)BltTransformerLayeri layer_idxc>[TU]5 [XS9Ul[ U5Ul[ URURS9Ul [ URURS9Ul g)N)configreps) super__init__BltSelfAttention self_attnrmlpr hidden_size rms_norm_epsinput_layernormpost_attention_layernormselfrr __class__s r7rBltTransformerLayer.__init__s[ )M&>)&*<*<&BUBUV(263E3E6K^K^(_%r9)rrrr)rrrrintrr __classcell__rs@r7rrs`#``r9rc ^\rSrSrS\S\4U4SjjrS S\RS\RS\RS\ 4U4S jjjr S r U=r $) rirrc2>[TU]X5 SUlg)NT)rr is_causalrs r7rBltSelfAttention.__init__s +r9 hidden_statesattention_maskposition_embeddings use_cachec 2>[TU]"SUUUUUUS.UD6$)N)rrrrpast_key_valuescache_positionr)rforward) rrrrrrrkwargsrs r7rBltSelfAttention.forward#s7w ') 3+)    r9)r)FNN) rrrrrrrr-Tensorrirrrrs@r7rrs]yS  ||   #\\     r9rc^\rSrSrSrSS\S\S\\4U4SjjjrSS\ RS\\ RS \\ S \\ RS \\ RS \ \4 S jjrSrU=r$)BltCrossAttentioni8z[TU]5 SUl[URUR S9Ul[URUR S9Ulg)NFr)rrrrrrq_normk_norm)rrrrrs r7rBltCrossAttention.__init__;sI  !1!1v7J7JK  !1!1v7J7JK r9rcross_attention_statesrrrrc UR5upxn URU5n URU 5n U RXxURUR 5R SS5n UbURU5nURU5n URU5n U RUSURUR 5R SS5n U RUSURUR 5R SS5n Ub!URXURSU05upOZUSS:waFURURRURURRpO [!S5e["n UR$R&S:wa[(UR$R&n U "UU U U U4UR*(dSO UR,UR.S .UD6upUR1XxS5R35nUR5U5nX-nX4$) Nrrr)rrz^Cross attention layer can't find neither `cross_attn_states` nor cached values for key/values!eagerg)dropoutscaling)r~rq_projview num_headshead_dim transposerk_projv_projnum_key_value_headsupdaterlayerskeysvaluesrfrr_attn_implementationr trainingrrreshape contiguouso_proj)rrrrrrrbszrl_ query_states key_states value_statesattention_interface attn_output attn_weightss r7rBltCrossAttention.forwardAs!&**, A{{=1 {{<0 #((T^^T]]S]]^_abc ! -%)[[1G%H "%;L#b$2J2JDMMZddefhijJ',,S"d6N6NPTP]P]^hhijlmnL*+:+A+Adnn?OQ_>`,( A ! #&&t~~6;;&&t~~6==% p )@ ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,LL %  % ! "))#b9DDF kk+. !1 ((r9)rrrryNNNN)rrrr__doc__rrrrr-rr LongTensorr rrrrrs@r7rr8sFLyLSLxPS}LL:>+/1559 3)||3)!) 63)"% 3) !. 3) !!1!12 3)+,3)3)r9rcf\rSrSr%\\S'SrSrSrS/r \ "\ SSS9\ "\ SSS9S .r S rS rS rS rg)BltPreTrainedModeliwrFrr local_decoderindex layer_namer)r attentionsc[S5eNzNo need to inherit it!AttributeErrorrmodules r7 _init_weights BltPreTrainedModel._init_weights566r9c[S5errrs r7_update_causal_mask&BltPreTrainedModel._update_causal_maskrr9c[S5errrs r75_prepare_4d_causal_attention_mask_with_cache_positionHBltPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positionrr9rN)rrrrr__annotations___supports_attention_backend_supports_flash_attn_supports_flex_attn_no_split_modulesrrr_can_record_outputsrrrrrr9r7rrwsU "' ./'(;1Q`a$%5Q?[ 777r9rc^\rSrSr%\\S'S\"\SSS90rS\4U4Sjjr SS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \S\ \ RS\ \ RS\ \S\ \ RS\\4SjjrSrSrU=r$)BltLocalEncoderirencoder_attentionsrrUrc >[TU]U5 SUlXl[R "[ UR5Vs/sHn[X5PM sn5Ul [US9Ul [R"URURUR-SS9Ul[R "UR"UR5Ul[R "5UlUR((a UROSn[ U5H0nUR&R+[-XURS95 M2 UR/5 gs snf)NFr in_features out_featuresbiasrrrr)rrgradient_checkpointingrnn ModuleListrSnum_hidden_layersrrr rotary_embLinearrrapatch_embedding_projection Embedding vocab_sizerRcross_attn_layerscross_attn_all_layersrr post_initrrr layers_to_addrs r7rBltLocalEncoder.__init__s  &+# mmEJ6KcKcEd eEd  3Ed e  -F;*,))**++f.A.AA+ ' LL):):F>|L +33 2 21 5 8P8P PRVR]R]RiRi  $(;;#D#DC! ,0,B,B9,M-".+8#9- -)& ,.DD -1. ,222r9c:URSnURSnURS5RSSURS5n[R"XBU4UR UR S9nURUSUSSS9nUSS2SU2SS24nU$) a> Reduce variable length patches to single embedding per patch Note: this works with variable number of patches for different sequences in the batch It handles variable length patches by assuming that patch_lengths will be 0 for any extra patches on the *right*. Since there can be a variable number of patches this function also return the number of patches for each sequence in the batch. Any embeddings on the right that are not allocated to a patch (i.e. if the sum(patch_lengths[i]) < seq_len for any i) will be sent to a dummy patch, which is trimmed before returning. rr)r&ramaxF)srcr,rreduce include_selfN)r1rcrdr-r?r'r(scatter_reduce)rrmax_num_patchesr]rB embedding_dimreduced_embeddingss r7r-BltLocalEncoder.patch_reduces#((+ %++B/ ''+222r=;N;Nr;RS "[[ - 8 @S@S\i\p\p 0>> ? 03CO3CQ0FG!!r9)rrrRrrrr NNNNNNNNNN)rrrrr"rrrrrrr-rrrrr rrr-rrrs@r7r r s& !!n-=QSbc421504/31537+/599=%),043E,,-43  -43u||, 43 !. 43 u//0 43"%43!!1!1243!) 643c]43ELL)43+,43l""r9r cT^\rSrSr%\\S'S\4U4Sjjr\"5SS\\ RS\\ RS\\ RS\\ RS\\ RS \\ S \\ RS \\ RS \ \4S jj5rSrU=r$)BltLocalDecoderirc >[TU]U5 SUlXlSUl[ R "[UR5Vs/sHn[X5PM sn5Ul [US9Ul [ R"URURUR -SS9Ul[%URUR&S9Ul[ R "5UlUR,(a UROSn[U5H0nUR*R/[1XURS95 M2 UR35 gs snf)NFTr rrrr)rrrrcross_attn_decoderrrrSrrrrrrhidden_size_globalrrarrrnormrrrrrrs r7rBltLocalDecoder.__init__s  &+# "&mmEJ6KcKcEd eEd  3Ed e  -F;*,))11++f.A.AA+ ' v11v7J7JK !#4:4P4P00VW }-I  " " ) )!RXRdRde . ! fs E#r"r#r$rr%rrr&rc .URSn Un URU5nURXRSURR-URR 5nUbUR (dX-n UcK[R"URSURS9RS5RU S5nURX5n [R"XRRURS9n [!UR"5HTupU S:XdURR$(a UR&U "SU UUS.U D6unnX-n U"U 4U UUUS.U D6n MV UR)U 5nU$) Nrrr*r)r(r+r*r)r1rrrrarrAr-r0r(rcrdrr,rrrrrrrC)rr"r#r$rr%rrr&rrBrrrr/r0rlogitss r7rBltLocalDecoder.forwards#((+ % 66|D #++ **1- 0H0HH$++JaJa   #D,C,C)8M   ]003M[TU]U5 Xl[R"5Ul[ UR5H'nUR R[X55 M) [US9Ul [USS5b0[R"URURSS9UlO[R""5UlUR%5 g)Nr encoder_cross_output_sizeFr)rrrrrrrSrrrrrgetattrrrNrtoken_embedding_projectionIdentityrrs r7rBltGlobalTransformer.__init__Us   mmo v778I KK  26E F9,F; 66 = I.0ii00&2D2D5/D +/1kkmD + r9 input_embedsrr%rrrc URupxn URU5n [R"XRRUR S9n UcK[ R"URSURS9RS5RUS5nURX5n [UR5HupU "U 4U UUUS.UD6n M U $)Nr(rr*rr)r*)r1rQr,rrrr-r0r(rcrdrrr)rrTrr%rrrrBrCrrrrr/s r7rBltGlobalTransformer.forwardgs".!3!3 Q77 E  -;;3F3FQUQ^Q^_   \//2<;N;NOYYZ[\ccdnprs #oomJ!$++.HA!$7- /-   M/r9)rrrrQr)rrrrr rrrrrr-rrrrr rrrrrs@r7rJrJOs &&^,[TU]U5 [URS9Ul[ R "5Ul[URR5H2nURR[URU55 M4 [ R"URRURR5Ul[!URRURR"S9Ul[ R&"URRURRSS9Ulg)Nr rFrO)rrrrrrrrrSrrrrrrrRrrrCrlm_headrs r7rBltPatcher.__init__s  ,DKK@mmo t{{<<=I KK  24;; J K>LL)?)?AXAXYt{{66DKK>f>MUUW&3&9&9"1&=# O  ! ==. /%# >M"JJ_-]5H5HQ^QeQeM.m=MN #]F::r9c`URSn[R"SS/[RURS9R S5R US5nURSnUSS2SS24nX:nURSn[R"XRS9R S5RUS5n [R"X5n [R"X/SS9n [R"Xw)/SS9n XRXH5n URSS9R5nU SS2SU24n [R"X]U-4SS9n[R"USS2SS24US- 5n[R"USS2SS24S- U4SS9nUU- S-nU$)z Computes patch lengths from token entropies. Depending on whether a threshold is provided, the function uses either: - Thresholding the entropy values (when `threshold` is set). rrr&Nr*r)r+)r1r-r.longr(rcrepeatr0rd full_liker@rr2r)rar_r\r]rB init_tokensoffset patch_maskrC token_indicessentinelpadded_indices padded_mask patch_startsmax_valid_patchespatch_start_ids last_token patch_endsrus r7rf'BltPatcher.patch_lengths_from_entropiess__Q'  LL!Quzz):J:J K U UVW X _ _`jlm n ""1%ae$ * ""1% W5E5EFPPQRSZZ[egij ??=:M#?;!!1!12?;SM?;E??;#3-?;+,?;B 33r9rXc^\rSrSrS\4U4Sjjr\"5SS\\RS\\RS\\RS\\RS\\ S \\RS \\ S \\RS \\S \4Sjj5rSrSrS\RS\S \R4SjrSrU=r$)BltModeli rc>[TU]U5 SUlXl[ UR 5Ul[UR5Ul [UR5Ul UR[UR5-nUR U-n["R$"X1R R&5UlURR*(a][-UR.5UlUR0R35 UR0R55H nSUlM OSUlUR95 g)NF)rrrrr encoder_configrUrJ global_configrLr?decoder_configrrLrTrMrNrrrrKpatch_in_forwardrXpatcher_configpatchereval parameters requires_gradr)rrnum_embeddingstotal_vocab_sizeparamrs r7rBltModel.__init__s  &+# ,V-B-BC"6v7K7K"L,V-B-BCDDs6KnKnGoo!??.P*,,,7GI^I^IjIj*k' ;; ' '%f&;&;
~ !!1!12~ +,~  !~ ~ @/0 iU\\ iC iTYT`T` i ir9rc^\rSrSr%\\S'SrSrS/rS\4U4Sjjr SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \\ R\ R4S \ \\\\ R$4S\ \ R$S\ \ RS\ \S\ \ RS\\\ R4S\\S\\\44SjjrSrU=r$)BltForCausalLMirFmodelzlm_head.weightc>[TU]U5 URUl[U5Ul[ R "URRURSS9Ul UR5 g)NFrO) rrrrrrrrrrZr)rrrs r7rBltForCausalLM.__init__sZ   ++f% yy!6!6!B!BFDUDU\ab  r9r"rr%rrpfull_text_row_masked_out_maskrr#labelsrrlogits_to_keeprrOc UR"SUUUUUUUU U S. U D6nURn[U [5(a [ U *S5OU nUR USS2USS245R 5nSnU bUR"UXR40U D6n[UUURURURS9$)N) r"rr%rprrr#rr)lossrFrrrr) rr isinstancerslicerZr~ loss_functionrr rrr)rr"rr%rrprrr#rrrrroutputsr slice_indicesrFrs r7rBltForCausalLM.forwards"**  )%!5*G+')     11 8B>SV8W8W~ot4]k mA}a,?@AGGI  %%ffooPPD%#33!//))   r9)rZrr) NNNNNNNNNNNr)rrrrrr_can_compile_fullgraphbase_model_prefix_tied_weights_keysrrr-rrtuplerrlistr}rirr rr rrrrs@r7rrs "*+y151537=A;?UYKO59-1$(5934, E,,-, !., u//0 , !))9)9 : , 'u'7'78 , (0ellELL6P0Q'R, "%tE4E4E/F(F"GH,   1 12, ))*, D>, !!1!12, c5<</0, +,,  u,, -, , r9r)rrrXr)rQ)rrQi0u)Ortypingrrrr-torch.distributionstorch.nnrtorch.nn.functional functionalr, cache_utilsrr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrr utils.genericrrcohere2.modeling_cohere2rrmllama.modeling_mllamarrrrrrrrconfiguration_bltrr r!r"r# get_loggerrloggerrr8rrIrrr\float32rir'rrtrrrrrrrrr r?rJrXrr__all__rr9r7rseC,, ./O5&@@?      H %::<\a||),9<UX$#,,#!# #+. # #' # $' # \\#T % K ||K K K  K  K ;; K  5<< %&K \))RU)[`[g[g)X ]  "  / `9` . 4<)0<)~7.77*p"(p"fL(L^2-2jF#FRfi!fiR9 &9 x r9