cCiPSSKrSSKJr SSKrSSKJr SSKJrJr SSKJ r J r SSK J r SSK Jr SS KJr S S KJr S S KJrJrJrJrJrJrJrJr S S KJr SSKJr \R@"\!5r"Sr#Sr$"SS\5r%Sr&"SS\RN5r("SS\(5r)"SS\(5r*\(\)\*S.r+"SS\5r,"SS\5r-"S S!\5r."S"S#\5r/"S$S%\5r0"S&S'\5r1"S(S)\5r2/S*Qr3g)+N)Optional)nn)Cache StaticCache)_flash_attention_forward!flash_attn_supports_top_left_mask)PreTrainedModel)logging)deprecate_kwarg)GemmaForCausalLM)LlamaDecoderLayerLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaModelLlamaPreTrainedModelapply_rotary_pos_emb repeat_kv) MistralMLP)DiffLlamaConfigzkajuma/DiffLlama-0.3B-handcutrc\rSrSrSrg) DiffLlamaMLP2N__name__ __module__ __qualname____firstlineno____static_attributes__ri/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/diffllama/modular_diffllama.pyrr2r$rc@SS[R"SU-5-- $)Ng?g333333?g333333ӿ)mathexp) layer_idxs r%lambda_init_fnr+6s txxy 011 11r$c^\rSrSrSrSS\S\\4U4Sjjjr\ "SSSS 9SS \ RS \ \ R\ R4S \\ RS \\ RS\\S\S\\ RS\ \ R\\ R\\ \ R44Sjj5rSrU=r$)DiffLlamaAttention:z=Multi-headed attention from 'Attention Is All You Need' paperconfigr*c>[TU]5 XlX lUc-[R SUR RS35 URUlURUl URUl [USURUR-5Ul URUlURUR-UlUR UlUR"UlSUl[&R("URURUR-UR*S9Ul[&R("URURUR-UR*S9Ul[&R("URURUR-UR*S9Ul[&R("URUR-URUR*S9Ul[5U5Ul[&R8"[:R<"SUR>UR4S95Ul [&R8"[:R<"SUR>UR4S95Ul![&R8"[:R<"SUR>UR4S95Ul"[&R8"[:R<"SUR>UR4S95Ul#[&RH"SUR-URJS S 9Ul&g) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.head_dimT)biasr)sizer F)epselementwise_affine)'super__init__r/r*logger warning_once __class__rattention_dropout hidden_sizenum_attention_heads num_headsgetattrr1num_key_value_headsnum_key_value_groupsmax_position_embeddings rope_theta is_causalrLinearattention_biasq_projk_projv_projo_projr+ lambda_init Parametertorchnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormselfr/r*r:s r%r7DiffLlamaAttention.__init__=sz  "      !8!8 9:,,  "(!9!9!--33 D4D4D4VW #)#=#= $(NNd6N6N$N!'-'E'E$ ++ii 0 0$..4==2PW]WlWlm ii 0 0$2J2JT]]2Zagavavw ii 0 0$2J2JT]]2Zagavavw ii >@P@PW]WlWlm ))4ell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdA $56;N;Nchir$past_key_valuepast_key_values4.58new_nameversion hidden_statesposition_embeddingsattention_mask position_ids use_cachecache_positionreturnc UR5upn U n URU5n URU5nURU5nU R XUR UR 5RSS5n UR XURUR 5RSS5nUR XURUR 5RSS5nUunn[XUU5upUb%UUUS.nURXURU5up[XR5n[XR5n[R"[R "USSS9SS9nUR#SSSS5n[R$"XRSS55[&R("UR 5- nUb#USS2SS2SS2SUR*S24nUU-n[,R.R1US[R2S9R5U R65n[,R.R9UUR:UR<S 9n[R>"[R@"URBURD-S[R2S95R5U R65n[R>"[R@"URFURH-S[R2S95R5U R65nUU- URJ-n[R$"UU5n[R "USSS9unnUUU-- nSURJ- URMU5-nURSS5RO5nURQXS5nURSU5nUU4$) Nrr sincosredimrrldtype)ptraining)*r3rGrHrIviewr>r1 transposer@rupdater*rrArMcatchunkrepeatmatmulr(sqrtshaper functionalsoftmaxfloat32torpdropoutr;rrr)sumrPrQrRrSrKrV contiguousreshaperJ)rXr`rarbrcr[rdrekwargsbsz target_len_q_len query_states key_states value_statesrjri cache_kwargs attn_weights causal_masklambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2s r%forwardDiffLlamaAttention.forward_sd+//1{{=1 [[/ {{=1 #((T^^T]]S]]^_abc __S1I1I4==Yccdeghi #((T5M5Mt}}]gghiklm &S#7 RUWZ#[  &#&snUL'6'='=jX\XfXfht'u $Jz+D+DE  /H/HI yy\1!!D"M #**1aA6 ||L2F2Fq!2LMPTPYPYZ^ZgZgPhh  %(Aq2HJ4D4DR4H2H)HIK'+5L}},,\r,WZZ[g[m[mn }},,\T=S=S^b^k^k,l 99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< ll<> %*[[aQ%G" l"[<%?? 4+++t~~k/JJ !++Aq1<<> !))#b9 kk+. L((r$)r;r/rVr1r<rDrHrKrQrSrPrRr*rBr>rAr@rJrGrCrINNNNFN)rr r!r"__doc__rrintr7r rMTensortuple LongTensorrboolrr# __classcell__r:s@r%r-r-:sG j j8C= j jD%0A6R 2637+/59<)||<)#5<<#=><)!. <) u//0 <) "% <)<)!!1!12<) u||Xell3XeELL>Q5RR S<)S<)r$r-c2^\rSrSrSrU4Sjr\"SSSS9SS \RS \ \R\R4S \ \RS \ \RS\ \ S \ S\ \RS\ \RS44Sjj5rSrU=r$)DiffLlamaFlashAttention2a> DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cD>[TU]"U0UD6 [5Ulgr)r6r7r _flash_attn_uses_top_left_mask)rXargsrr:s r%r7!DiffLlamaFlashAttention2.__init__s# $)&) /P.Q+r$rZr[r\r]Nr`rarbrcrdrerfcH [U[5(a [S5eUR5upn UR U5n UR U5n UR U5n U RXURUR5RSS5n U RXURUR5RSS5n U RXURUR5RSS5n Uc)[RS5 URX5upOUup[XX5upUb$XUS.nUR!XUR"U5upU RSS5n U RSS5n U RSS5n UR$(a UR&OSnU R(nU R*R,S:waU R*R,OSnU[.R0:Xa[.R2"5(aA[5[.S 5(a[.R6"U5O[.R8"5nOR[5UR:S 5(aUR:R<nO URR>R(n[RS US 35 U RAU5n U RAU5n U RAU5n [.RB"U SSS 9unnURESSSS5nURESSSS5n[GU U UUU UU[IUSS5URJURLS9 n[GU U UUU UU[IUSS5URJURLS9 n[.RN"UU/SS 9n[.RB"USSS 9unn[.RP"[.RR"URTURV-S[.R0S95RAU R(5n[.RP"[.RR"URXURZ-S[.R0S95RAU R(5nUU- UR\-nUUU-- nSUR\- UR_U5-nURaXS5Rc5nUReU5nUS4$)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersrr aYThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be removed and `position_embeddings` will be mandatory.rhmpscpuget_autocast_dtype_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rksliding_window)rcrruse_top_left_maskrDrmro)3 isinstancer ValueErrorr3rGrHrIrsr>r1rtr@r8r9 rotary_embrrur*rrr;rpdevicetyperMr~is_autocast_enabledhasattrrget_autocast_gpu_dtyper/rweightrrwrxrr?rrDrvr)rrPrQrRrSrKrVrrrJ)rXr`rarbrcr[rdrerrrrrrrjrir dropout_rate input_dtype device_type target_dtype value_states1 value_states2rrrrrrs r%r DiffLlamaFlashAttention2.forwards o{ 3 3}  &**, A{{=1 [[/ {{=1 $((T^^T]]S]]^_abc __S1I1I4==Yccdeghi #((T5M5Mt}}]gghiklm  &   G  |BHC*HC#7 RU#[  &#&nUL'6'='=jX\XfXfht'u $J$--a3 ))!Q/ #--a3 15t--C #(( 2>2E2E2J2Je2Sl))..Y^ %-- '((**u&:;;,,[9557 &?@@#{{BB #{{1177    >$  (??<8L#|4J'??<8L',{{<'J$ }%,,Q1a8 %,,Q1a8 /     % "4)94@"AAnn  0     % "4)94@"AAnn  ii| <"E %*[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !))#b9DDF kk+. D  r$)rr)rr r!r"rr7r rMrrrrrrrr#rrs@r%rrs R%0A6R 6:37+/59B!||B!#5<<#=>B!!!1!12 B! u//0 B! "% B!B!!!1!12B! u||T! "B!SB!r$rcZ\rSrSrSr\"SSSS9SS\RS \\R\R4S \ \RS \ \RS\ \ S \ S \ \RS\\R\ \R\ \\R44Sjj5r Srg)DiffLlamaSdpaAttentioni4z DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. rZr[r\r]Nr`rarbrcrdrerfc UR5upn URU5n URU5n URU5nU R XUR UR 5RSS5n U R XURUR 5RSS5n UR XURUR 5RSS5nUunn[XUU5upUb$UXS.nURXURU5up[XR5n [XR5n[R"[R "USSS9SS9nUR#SSSS5nUnUbUSS2SS2SS2SU R$S24nU R&R(S:Xa3Ub0U R+5n U R+5n UR+5nUSL=(a U S:n[R,R.R1U U UUUR2(a UR4OSUS 9n[R "USSS9unn[R6"[R8"UR:UR<-S[R>S 95RAU RB5n[R6"[R8"URDURF-S[R>S 95RAU RB5nUU- URH-nUUU-- nSURH- URKU5-nURSS5R+5nUR XS5nURMU5nUS4$) Nrr rhrkrmrncudar) attn_mask dropout_prDro)'r3rGrHrIrsr>r1rtr@rrur*rrArMrvrwrxr{rrrrr|scaled_dot_product_attentionrrr;r)rrPrQr~rrprRrSrKrVrJ)rXr`rarbrcr[rdrerrrrrrrrjrirrrDrrrrrrs r%rDiffLlamaSdpaAttention.forward<sU&**, A{{=1 [[/ {{=1 #((T^^T]]S]]^_abc __S1I1I4==Yccdeghi #((T5M5Mt}}]gghiklm &S#7 RUWZ#[  &#&sUL'6'='=jX\XfXfht'u $Jz+D+DE  /H/HI yy\1!!D"M #**1aA6 $  %%aA/E1A1A"1E/E&EFK    # #v -+2I'224L#..0J'224L 4'5EAI hh))FF   !04 d,,3 G &+[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !++Aq1<<> !&&s26 kk+. D  r$rr)rr r!r"rr rMrrrrrrrr#rr$r%rr4s%0A6R 2637+/59I!||I!#5<<#=>I!!. I! u//0 I! "% I!I!!!1!12I! u||Xell3XeELL>Q5RR SI!SI!r$r)eagerflash_attention_2sdpac4^\rSrSrS\S\4U4SjjrSrU=r$)DiffLlamaDecoderLayerir/r*c\>[TU]X5 [UR"XS9Ulg)N)r/r*)r6r7DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnrWs r%r7DiffLlamaDecoderLayer.__init__s( +4V5P5PQY_ur$)r) rr r!r"rrr7r#rrs@r%rrsvv3vvr$rc"\rSrSrSrSrSrSrg)DiffLlamaPreTrainedModeliFc,[R"X5 [U[5(aURR R SURR5 URR R SURR5 URR R SURR5 URR R SURR5 gg)Nr) r _init_weightsrr-rPdatanormal_r/rOrQrRrS)rXmodules r%r&DiffLlamaPreTrainedModel._init_weightss%%d3 f0 1 1    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H    ! ! ) )!T[[-G-G H 2r$rN)rr r!r"_supports_flex_attn_supports_attention_backendrr#rr$r%rrs"'Ir$rc\rSrSrSrg)DiffLlamaModelirNrrr$r%rrr&r$rc\rSrSrSrg)DiffLlamaForCausalLMirNrrr$r%rrr&r$rc\rSrSrSrg)"DiffLlamaForSequenceClassificationirNrrr$r%rrr&r$rc\rSrSrSrg)DiffLlamaForQuestionAnsweringirNrrr$r%rrr&r$rc\rSrSrSrg)DiffLlamaForTokenClassificationirNrrr$r%rrr&r$r)rrrrrr)4r(typingrrMr cache_utilsrrmodeling_flash_attention_utilsrr modeling_utilsr utilsr utils.deprecationr gemma.modeling_gemmarllama.modeling_llamarrrrrrrrmistral.modeling_mistralrconfiguration_diffllamar get_loggerrr8_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCrr+Moduler-rrrrrrrrrr__all__rr$r%rs$  -i-03   24   H %5# : 2b)b)JR!1R!jR!/R!l 1 "v-v I3 I Z  +  )G  $=  &A  r$