cCiSrSSKrSSKJrJr SSKrSSKJs Jr SSKJr SSK J r J r J r SSKJr SSKJr SS KJrJrJrJrJrJrJr SS KJr SS KJrJrJrJ r J!r! SS K"J#r# S SK$J%r% \ RL"\'5r(Sr)Sr*"SS\RV5r,"SS\RV5r-"SS\RV5r."SS\RV5r/"SS\RV5r0"SS\RV5r1"SS\RV5r2"SS \RV5r3"S!S"\RV5r4"S#S$\RV5r5"S%S&\RV5r6"S'S(\RV5r7"S)S*\RV5r8"S+S,\5r9S-r:S.r;\"S/\:5"S0S1\955r<\"S2\:5"S3S4\955r=\"S5\:5"S6S7\955r>\"S8\:5"S9S:\955r?\"S;\:5"S<S=\955r@\"S>\:5"S?S@\955rA"SASB\RV5rB\"SC\:5"SDSE\955rC/SFQrDg)GzPyTorch MEGA model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings)deprecate_kwarg) MegaConfigzmnaylor/mega-base-wikitextrc>^\rSrSrSrS\4U4SjjrSSjrSrU=r $)MegaEmbeddings6z Mega's basic implementation does not incorporate token type embeddings, so this is a stripped-down version of RoBERTa's embeddings which optionally includes token types configc>[TU]5 [R"URUR UR S9UlURUl UR(a{[R"URUR 5Ul URS[R"UR[R S9R#S5SS9 UR Ulg)N) padding_idxtoken_type_idsdtype)rF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsadd_token_type_embeddingsuse_token_typestype_vocab_sizetoken_type_embeddingsregister_buffertorchzeros max_positionslongexpandr!selfr __class__s k/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/deprecated/mega/modeling_mega.pyr(MegaEmbeddings.__init__<s !||F,=,=v?Q?Q_e_r_rs%??   )+f6L6LfN`N`)aD &   %++f.B.B%**"U"\"\]d"erw ! "..cUcUc [S5eUb.UR5nURnURU5nOUR5SSnURnUR(aUch[ US5(a3UR SS2SUS24nURUSUS5nUnO$[R"U[RUS9nURU5nX8-n U $Un U $)Nz.Must provide one of input_ids or inputs_embedsr%r"rrr$device) ValueErrorsizer@r-r/hasattrr"r7r3r4r6r1) r9 input_idsr" inputs_embeds input_shaper@buffered_token_type_ids buffered_token_type_ids_expandedr1 embeddingss r;forwardMegaEmbeddings.forwardJs  M$9MN N  "#..*K%%F!00;M',,.s3K"))F   %4!122.2.A.A!EU{ST~EUBU.V+7N7U7UVabcVdfqrsft7u4%EN%*[[EJJW]%^N%)$>$>~$N !&>J'Jr=)r!r1r/r-NNN __name__ __module__ __qualname____firstlineno____doc__rr(rJ__static_attributes__ __classcell__r:s@r;rr6s /z /r=rc:^\rSrSrSrS\4U4SjjrSrSrU=r $) MegaSimpleRelativePositionalBiaslzk Simple relative positional embeddings copied from the Mega repo; renamed variables for better readability rcB>[TU]5 XlURRS:aURROURRUl[ R "[R"SUR-S- 55Ul g)Nrr) r'r(r chunk_sizer5r Parameterr3Tensor rel_pos_biasr8s r;r()MegaSimpleRelativePositionalBias.__init__qsq  :>++:P:PST:TT[[66Z^ZeZeZpZpLLa&:N:N6NQR6R)STr=cXR:a[SUSUR35eURURU- URU-S- n[R"USU45n[ R "X145nUSU*nURUSU-S- 5nSU-S- S-nURS5U- nUSS2XE24nU$)NzSequence length z going beyond max length rrrZ) r5rAr^Fpadr3tileviewrB)r9seq_lenbiasrdstartends r;rJ(MegaSimpleRelativePositionalBias.forwardws '' '/y8QRVRdRdQefg g  $"4"4w">4CUCUX_C_bcCceuuTAw<(zz$ +IgXyy!g+/2WqQ&iilU"AuyL! r=)rr5r^rMrUs@r;rWrWls UzU r=rWc\^\rSrSrSrS\4U4Sjjr\S\S\4Sj5r Sr S r S r U=r $) MegaRotaryRelativePositionalBiasa\ Rotary relative bias for positional information; similar in concept to RoPE (i.e. RoFormer) but taken from the Mega repo due to differences in implementation. When initialized, produces a positional bias which ranges from position 0 to config.max_positions, but can extrapolate to longer sequences. Can be indexed according to input position IDs rc>[TU]5 URS-S:wa [S5eXlUR UlURRS:aURROURRUl[RURUR 5uUl Ul [R"[R "SUR 55Ul[R"[R "SUR 55UlUR'S[R("S/55 g)NrZrzCRotary positional bias requires `hidden_size` to be a multiple of 2r _float_tensor)r'r(r+ RuntimeErrorrshared_representation_size embed_dimr[r5rlget_sinusoid_embeddingssinecosinerr\r3r]alphab_paramr2 FloatTensorr8s r;r()MegaRotaryRelativePositionalBias.__init__s     !Q &de e :::>++:P:PST:TT[[66Z^ZeZeZpZp!A!Y!Y  $.."  4; \\%,,q$.."AB ||ELLDNN$CD  _e.?.?.FGr=r5 embedding_dimcUS-n[R"S5U- n[R"[R"U[R S9R 5U*-5n[R"U[R S9RS5URS5-n[R"U5[R"U54$)NrZi'r#rr) mathlogr3exparangeint64float unsqueezesincos)r5r{half_dimembs r;rt8MegaRotaryRelativePositionalBias.get_sinusoid_embeddingss A%hhuo(ii XU[[AGGISDPQll= <FFqICMMZ[L\\yy~uyy~--r=c*UR5up#[R"USSS9upEURbX RRS5:a([R X#5uUlUlX lURRUR5UlUR RUR5UlURSUnUR SUn[R"XG-XV-- XW-XF--/SS9$)NrZr%dimrr) rBr3chunkrurlrtrvr5torocat)r9inputrfrschunk_1chunk_2rrs r;rotary'MegaRotaryRelativePositionalBias.rotarys"ZZ\ ;;uaR8 99 ))..*; ;%E%]%]^e%q "DIt{!( IILL!3!34 kknnT%7%78 ii!kk(7#yy'-'-79VW]^__r=cURURRXR55nURURRXR55n[ R "SX#5nU$)Nz mk,nk->mn)rrwr7rsrxr3einsum)r9rf rotary_alpha rotary_betargs r;rJ(MegaRotaryRelativePositionalBias.forwardsY{{4::#4#4Wnn#MN kk$,,"5"5g~~"NO ||KC r=)rwrxrrvrsr5ru)rNrOrPrQrRrr( staticmethodintrtrrJrSrTrUs@r;rlrlsHHzH .s.3.. `r=rlcB^\rSrSrSrSU4SjjrSS\4SjjrSrU=r $) MegaDropoutaN A unified class for standard dropout functionality and featurewise dropout. The original fairseq Mega repo used 2 classes for these, which included some unnecessary handling of training logic and an unused `inplace` option. The original implementation used torch.nn.functional instead of submodules, which is retained here as well. c:>[TU]5 XlX lgN)r'r(dropout_probabilityis_featurewise)r9rrr:s r;r(MegaDropout.__init__s #6 ,r= batch_firstcUR(aU(aJ[R"URSS5URUR S9RSS5$UR 5S:wa [S5e[R"URSSS5URUR S9RSSS5$[R"XRUR S9$) Nr%)ptrainingrazzFeature dropout inputs must be exactly 3-dimensional if inputs are ordered [sequence length, batch size, hidden dimension]rrZr) rrb dropout2d transposerrrrApermutedropout)r9rrs r;rJMegaDropout.forwards   {{OOB+t/G/GRVR_R_)B#$99;!#$U {{5==Aq#9T=U=U`d`m`mnvvq!99U&>&>W Wr=)rr)F) rNrOrPrQrRr(boolrJrSrTrUs@r;rrs!- X$XXr=rc<^\rSrSrSrSU4SjjrSrSrSrU=r $) MegaRMSNormz RMSNorm used in Mega implementation. Differs from T5's RMSNorm by applying the weight prior to taking the square root (as opposed to after in T5) c>[TU]5 XlX lX0lU(a:[ R "[R"UR55Ul gURSS5 g)Nweight) r'r( num_featuresepsaffinerr\r3r]rregister_parameter)r9number_featuresrrr:s r;r(MegaRMSNorm.__init__sP + ,,u||D4E4E'FGDK  # #Hd 3r=c[R"[R"U5SSS9nURbXR-nU[R"X R -5- U$)Nr%Trkeepdim)r3meansquarerrsqrtr)r9r mean_squares r;rJMegaRMSNorm.forwardsPjje!4"dK ;; "KK'E  K((233 r=cRURSURSUR3$)Nz, eps=z , affine=)rrrr9s r; extra_reprMegaRMSNorm.extra_reprs(##$F488*Idkk]KKr=)rrrrgư>T) rNrOrPrQrRr(rJrrSrTrUs@r;rrs 4LLr=rc6^\rSrSrSrSU4SjjrSrSrU=r$) MegaScaleNormiz Scale normalization introduced in MEGA which is similar to RMSNorm, but uses a single parameter for scalar multiplication instead of a vector, and applies over a specified dimension c>[TU]5 XlX lX0lU(a0[ R "[R"S55Ul gURSS5 g)Nrscalar) r'r(rrrrr\r3r]rr)r9rrrr:s r;r(MegaScaleNorm.__init__sH  ,,u||A7DK  # #Hd 3r=c[R"[R"U5URSS9nURbURU-nU[R "X R -5-nU$)NTr)r3rrrrrr)r9rroutputs r;rJMegaScaleNorm.forwardsXjje!4$((DQ ;; "KK%'E[88%;<< r=)rrrrr rNrOrPrQrRr(rJrSrTrUs@r;rrs 4r=rc6^\rSrSrSrSU4SjjrSrSrU=r$)MegaSequenceNormiz A wrapper class for various layer normalization options used in Mega. Used to handle differences in expectations on input axis locations for different normalization methods. cZ>[TU]5 US:Xa[R"X#US9UlgUS:Xa[ SX4S9UlgUS:Xa[ X#US9UlgUS:Xa[R"X#US9UlgUS :Xa[R"X#US9Ulg[S U35e) N layernorm)elementwise_affine scalenormr%)rrrrmsnorm)rr batchnorm syncbatchnormzUnknown norm type: ) r'r(r LayerNormnormrr BatchNorm1d SyncBatchNormrA)r9 norm_typer{rrexportr:s r;r(MegaSequenceNorm.__init__ s   # ]FSDI + %%"#EDI ) ##M6JDI + %}fMDI / )((ODI29+>? ?r=cJ[UR[RRR 5(aVUR 5S:wa [S5eURSSS5nURU5nURSSS5$URU5$)Nraz.BatchNorm inputs must be exactly 3-dimensionalrrZr) isinstancerrmodulesr _BatchNormrrAr)r9rs r;rJMegaSequenceNorm.forward/s dii!5!5!@!@ A Ayy{a !QRRMM!Q*EIIe$E==Aq) )99U# #r=)r)gh㈵>TFrrUs@r;rrs @$$r=rc ^\rSrSrSrS\4U4SjjrSrS\4Sjr Sr S\4S jr S r SS jr SS jrSS \\R"S\\R"S\S\R"4SjjrSrU=r$)MegaMultiDimensionDampedEmai:a Mega's Exponential Moving Average layer, largely left unmodified from the original repo with the exception of variable names and moving away from the stateful representation of incremental decoding state. See "https://huggingface.co/papers/2209.10655" for more details. rc>[TU]5 XlURUlUR UlURUlURUl[R"SUR - 5Ul UR(aSUR-O URn[R"[R"X R S55Ul[R"[R"X R S55Ul[R"[R"X R S55Ul[R"[R"X R 55Ul[R"[R"UR55UlSUlSUlg)N?rZr)r'r(rr+rsema_projection_sizendim bidirectional truncationr}sqrtscalerr\r3r]damping_factor decay_factorema_expansion_matrixkernel_projection_matrixresidual_weight_kernel_coeffs)r9r kernel_dimr:s r;r($MegaMultiDimensionDampedEma.__init__As/  ++.. #11 ++YYsTYY/ /3/A/AQ+++vGYGY  ll5<< IIq+QRLLj))Q)OP%'LLj))UV1W$X!(* U\\*ii5X(Y%!||ELL9K9K,LM  r=cSUl[R"UR5n[R"UR5nSX-- nX4$)Nr)rr3sigmoidrr)r9rrprevious_timestep_weights r;_compute_ema_coefficients5MegaMultiDimensionDampedEma._compute_ema_coefficientsYsH t':':;}}T%6%67 #&)F#F 77r=lengthctSUlUR5up#[R"U5R U5R SSU5[R "U5-nX R-[R"U5-n[R"SXPRUR-5$)Nr dnl,dn->dl) rrr3rrrer~rrrrr)r9rrrvanderkernels r;_compute_efficient_ema_kernel9MegaMultiDimensionDampedEma._compute_efficient_ema_kernelas 373Q3Q3S0f%((8==aFKeiiXpNqq #<#<< &@QQ||L&2O2ORVR\R\2\]]r=cUR(aUR5$URcUR5UlUR$r)rrrrs r;get_ema_coefficients0MegaMultiDimensionDampedEma.get_ema_coefficientsms< ==113 3||##==? << r=c:URcUO[URU5nUR(aURU5$URbURR S5U:aURU5UlURSSU24$)Nr%.)rminrrrrB)r9r kernel_sizes r;get_ema_kernel*MegaMultiDimensionDampedEma.get_ema_kernelus $ 7fSRX=Y ==55kB B||#t||'8'8'<{'J#AA+N <<\k\ 12 2r=c[RRUR5SU-S9n[RRUR5SU-S9n[RR XE-SU-S9nU$)NrZ)n)r3fftrfftrirfft)r9inputsrr inputs_fft kernel_fftconvolved_sequences r;fft_convolution+MegaMultiDimensionDampedEma.fft_convolution~sfYY^^FLLNa&j^A YY^^FLLNa&j^A "YY__Z-DF _S!!r=c tUS:XaURXS9$UR5upE[R"US-5R U5R SSUS-5[R "U5-n[R"U5nUb^USS2SS2SS24URUR-RS5-n[R"SX75nUSS2SS2S4U-n OSnSn USS2SS2SS24nX@R-U-n [R"SXRUR-5n URXUS9SSU24n U RU5n UbX-n [R"S U[R"U S /S 95n U bX-n U R!S SS5U 4$) Nr past_stater%z bdn,dnl->bdlrr.rz bdl,dnl->bdnrZ)dims) one_ema_steprr3rrrer~rrrrrrrtype_asflipr)r9rrrrrr past_ema_projpast_ema_statepast_vandermonder kernel_proj ema_outputupdated_hidden_states r;ema_step$MegaMultiDimensionDampedEma.ema_steps Q;$$V$C C483L3L3N0fqj),,^<AA!QQR SV[V_V_ $W  6"  !#1a8,0M0MPTPZPZ0Z/e/efh/iiM"\\.*TN &aBh/*< !N# 1crc " #<#<<Fll<9V9VY]YcYc9cd ))&f)McSTU[S[m\ ''/  %#4J$||NFEJJv]^\_D`a  '#7#J !!!Q*,@@@r=cUR5up4X0R-RS5U-nUbXTRS5U--n[R"SXPR UR -5nURS5U4$)Nr%z bdn,dn->bdr)rrsqueezer3rrrr)r9rrrr updated_stateouts r;r(MegaMultiDimensionDampedEma.one_ema_steps373L3L3N0(*C*CCLLRPSYY  !),L,LR,PS],]]Mll<8U8UX\XbXb8bc}}Q..r=attention_mask prev_state use_cachereturncUR5upVnXpR:wa[SUSUR35eXR-nUR SSS5nUb"XR S5R U5-nUR(aU(a [S5eU(a-URXUS9up[R"X-5n X4$URU5n Un Sn U RS5nUR(a[R"XRUR/SS 9unn[R"XS- S45[R"UR!S 5SUS- 45-n [R"XS- S45nX-S- n SU-S- n UR#XU S 9S XU-24nUR U5n[R"UR SSS5U-5nUS4$) a{ Mega's exponential moving average (EMA) sub-layer applied prior to single-headed (traditional) self-attention Args: inputs (`torch.Tensor` of shape `(sequence_length, batch_size, hidden_size)`): Hidden state / embedding input to update via EMA based on FFT convolution attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indicates which inputs are to be ignored (mostly due to padding), where elements are either 1 for *not masked* or 0 for *masked* prev_state (`torch.Tensor` of shape `(batch_size, config.ndim)`, *optional*): The hidden state returned from the previous timestep during incremental decoding. use_cache (`bool`, default `False`): Whether to perform incremental decoding; uses `prev_state` as the prior timestep, and returns the updated EMA hidden state for use in the next step Returns: `tuple(torch.FloatTensor)` containing various elements depending on configuration ([`MegaConfig`]) and inputs: - **hidden_states** (`torch.FloatTensor` of shape `(sequence_length, batch_size, hidden_size)`) -- Hidden states updated by EMA, with same shapes as inputs - **updated_state** (*optional*, returned when `use_cache=True`) `torch.FloatTensor of shape `(batch_size, config.ndim)` -- The incremental EMA state for use in the next step of incremental decoding 2Unexpected embedding dimension received: input is z, model expects rrZrNz4Bidirectional EMA does not support incremental staterrr%r.)rBrsrArrrrrrqr$rbsilur r3splitrcrr)r9rr+r,r-rfbszrsresidualr)r(rfft_lens_indexrk1k2r"gated_ema_outputs r;rJ#MegaMultiDimensionDampedEma.forwards>#)++-i  &DYKO_`d`n`n_op  0001a(  %77:BB6JKF   )UV V !%v:!V C&&(C% %((1FGG ++a.K!!Vnndnn-MSTUBr!OQ#78155qR]`aRaNb;ccva';<!/!3k/A---fW-McSZgn]nSnNnoJ#++F3J vvj&8&8Aq&AH&LM #T) )r=) rrrrrrrrsrrrrrr)NNF)rNrOrPrQrRrr(rrrrr rr$rrr3r]rrJrSrTrUs@r;rr:s z08 ^C ^ 3S3"(AT /26-1 M*!.M*U\\* M*  M*  M*M*r=rc^\rSrSrSrS\4U4SjjrSrSrSS\ \ RS\ \ RS \ \ RS \ \ S \ S \ S \\ R\ \ R44SjjrSrU=r$)MegaGatedCrossAttentioni a, Gated Structured State Attention for use in encoder-decoder model. See Mega paper for more details. Only modifications from original implementation are variable names, removing the unnecessary `before_attn_fn` and `static_kv` arguments, and the stateful representation of incremental decoder state. rcn>[TU]5 Xl[URRUlURR UlUR S:XaURR S-OSUl[URRURRS9Ul [URRURRS9Ul [URRSS9UlURR Ul[%URR&URR(URR*S9Ul[.R0"URR(URR 5Ul[.R0"URR(URR(5Ul[.R0"URR(SURR(-URR -5Ul[.R0"URR(URR(5UlURR:S:Xa[=U5UlOMURR:S:Xa[AU5UlO"[CS URR:35e[.RD"S S 9Ul#g) Nsoftmax࿩rFrrZsimplerz unknown relative position bias: r%r)$r'r(rr activationattention_activationrrscalingr dropout_probuse_feature_dropoutrhidden_dropout_probhidden_dropoutattention_probs_dropout_probattention_dropoutnormalize_before_megaprenormrnormalization_typer+ norm_affinerrLineark_projv_projq_projh_projrelative_positional_biasrWr^rlrASoftmaxr>r8s r;r( MegaGatedCrossAttention.__init__s'   !7!78$(KK$D$D!GKG`G`dmGmt{{==tCsw "4;;#;#;DKKLkLkl ) KK + +DKK<[<[ "-T[[-U-Ufk!l{{88 $ KK * *DKK,C,CDKKLcLc  ii 7 79_9_` ii 7 79P9PQ ii KK # #Q)@)@%@4;;CiCi%i  ii 7 79P9PQ ;; / /8 ; @ HD  [[ 1 1X = @ HD ? @d@d?efg gzzb) r=cUR5upVnUcURS5OUS-nUb!URSS9RUSS5n OUn UR[ X55SS2SU24n Ub%URS5S:wa [ S5eXn OU SUn [ R"XRSS55U - U -n [UR"U 5RU 5n UbXRS5-n U $)Nrr%r9Position offset provided with queries longer than 1 tokenrZ) rBsumrer^maxrAr3bmmrr rDrr) r9querykeykey_padding_maskpidxr3src_len_tgt_lenlengthsrgqk attn_weightss r;element_attention)MegaGatedCrossAttention.element_attention6s((*a#'<%**Q-TAX  '&**r*277QBGG  W!678G8 D  zz!}! !\]]:D>DYYummAq1 2W r) r9r]r^r_r`r3rarbrcrgrerfs r;softmax_attention)MegaGatedCrossAttention.softmax_attentionTs ((*a#'<%**Q-TAX  W!678G8 D  zz!}! !\]]:D>D $ YYummAq1 2T 9  '%5!5 @ @ C F Fuzz RTYZ`TabB||B'//3 r=r^valuer_past_key_valuesoutput_attentionsr-r.cUR5upn XRR:wa%[SU SURR35eUb8US:wa[SU35eUSSupS=p#USn U RS5S-nOS=pU(aUS:XaSOSnUnUR(aUR U5nUR U5n[R"UURRURRURR/SS 9unnn[R"U5n[R"U5nUcUb [S 5eS=nnO1URU5nURURU55nUR!SS5nUbUR!SS5nUbUR!SS5nUbU nU nU(aUnUnURS5nUbUR#5S:XaSnUb@URS5U :wa [S 5eURS5U:wa [S 5eUR$S :XaUR'UUXN5nOUR)UUXN5nUR+USS9nUR-U5n[R."UU5R!SS5nURUR1UU-55nUR3U5n[R4"UUUU- 5nUR(dUR U5nU(aUU4OU4nU(aUWW4-nU$)a Gated cross-attention used in Mega Args: query (`torch.Tensor` of shape `(target_sequence_length, batch_size, hidden_size)`): The self (or target) sequence input used as query inputs for cross-attention key (`torch.Tensor` of shape `(source_sequence_length, batch_size, hidden_size)`): The cross (or source) sequence input with shape used as keys in cross-attention value (`torch.Tensor` of shape `(source_sequence_length, batch_size, hidden_size)`): The cross (or source) sequence input with shape used as values in cross-attention key_padding_mask (`torch.LongTensor` of shape `(batch_size, source_sequence_length)`, *optional*): Padding mask corresponding to the source sequence, where entries are 1 for *not masked* and 0 for *masked* tokens past_key_values (`tuple(torch.FloatTensor)`, *optional*): If provided, the hidden state returned from the previous timestep during incremental decoding; expects that prior cross-attention keys and values will be the last two items in the tuple output_attentions (`bool`, defaults to `False`): Whether or not to return the cross-attention weights. use_cache (`bool`, defaults to `False`): Whether to perform incremental decoding; uses `prev_state` as the prior timestep, and returns the updated EMA hidden state for use in the next step Returns: `tuple(torch.FloatTensor)` containing various elements depending on configuration ([`MegaConfig`]) and inputs: - **hidden_states** (`torch.FloatTensor` of shape `(target_sequence_length, batch_size, hidden_size)`) -- Hidden states from target sequence updated by gated cross-attention - **attn_weights** (*optional*, returned when `output_attentions=True`) `torch.FloatTensor` of shape `(batch_size, source_sequence_length, target_sequence_length)` -- The pairwise cross-attention weights corresponding to each token in the source and target sequences - **cross_key** (*optional*, returned when `use_cache=True`) `torch.FloatTensor` of shape `(batch_size, source_sequence_length, config.shared_representation_size)` -- The cross-attention key state for use in the next step of incremental decoding - **cross_value** (*optional*, returned when `use_cache=True`) `torch.FloatTensor` of shape `(batch_size, source_sequence_length, config.hidden_size)` -- The cross-attention value state for use in the next step of incremental decoding r0z but expected Nrz>Incremental decoding requested with self-sequence length > 1: rrr%rz+Key and value must be `None` simultaneouslyz6Key padding mask does not align on the batch dimensionz@Key padding mask does not align on the sequence length dimensionr>Tr)rBrr+rArMrrSr3r2rrrrbr1rQrCrRrrrDrlrgrIrKr\rTraddcmul)r9r]r^rnr_rorpr-rfr3rsprev_cross_keyprev_cross_value prev_self_keynum_incremental_steps full_queryquery_projectedr target_gateattention_query projected_keyprojected_valueupdated_cross_keyupdated_cross_valuectx_lenrfrweighted_targetsr)outputss r;rJMegaGatedCrossAttention.forwardns~`#(**,i  // /DYK~^b^i^i^u^u]vw   &!| #abiaj!kll/>rs/C ,N C,A.M$1$6$6q$9A$= !04 4N)21 A4 ! <<:.J++j1 9>  [[ $ $dkk&=&=t{{?e?e f9 5o --8ff[) ;  !NOO.2 2MO!KK,M"oodkk#.>?O*33Aq9  $)33Aq9M  &-771=O  &*M.O  - "1 $$Q'  ',<,@,@,Ba,G#   '$$Q'3. !YZZ$$Q'72 !cdd  $ $ 1110@L 110@L--o4-P'' 5!99V_=GG1M??4;;7G+7U+VW<<(89mmE?4Du4LM||))C.C):3 % !24G HHGr=)rCrDrKrrrTrIrQrrMrSr^rEr>rR)NNFF)rNrOrPrQrRrr(rgrlrr3r]r rtuplerJrSrTrUs@r;r<r< s "*z"*H<>48+/"'\ell #\ % \ #5<<0 \ "% \ \\ u||Xell33 4\\r=r<c^\rSrSrSrS\4U4SjjrSrSrS S\ \ RS\ \ RS \ \ 4S jjr S rU=r$) MegaMovingAverageGatedAttentioni an Pure PyTorch implementation of Mega block; see https://huggingface.co/papers/2209.10655 and original fairseq implementation at https://github.com/facebookresearch/mega (copyright Meta Research, licensed under MIT License) Differences from original implementation include hidden state refactor and fixed inconsistency with additive / multiplicative attention masks rcf>[TU]5 Xl[URRUlURR S:XaURR S-OSUl[URRURRS9Ul [URRURRS9Ul [URRSS9Ul[!URR"URR$URR&S9Ul[+U5Ul[.R0"URR$URR25Ul[.R0"URR$URR URR2-SURR$--5Ul[.R0"URR2URR$5Ul[.R:"[<R>"SURR 55Ul [.R:"[<R>"SURR 55Ul!URRDS:Xa[GU5Ul$OMURRDS:Xa[KU5Ul$O"[MS URRD35e[.RN"S S 9Ul(URR S:XaURRUl+gURTUl+g) Nr>r?r@FrArZrBrz"Unknown relative positional bias: r%r),r'r(rr rCrDrrrErrFrGrrHrIrJrKrrNr+rOrrema_gaterrPintermediate_sizerRmx_projrTr\r3r] qk_weightqk_biasrUrWr^rlrArVr>rlrgattention_functionr8s r;r((MegaMovingAverageGatedAttention.__init__s   !7!78<@KK<\<\`i>?CKKBO  #'*@*@*CCL  "'5Lr=cURS5nURU5nXQRS5:wa%URS5S:wa [S5eUSSnXR-n[R "XR SS55U-nUbN[R"XGRS9nURSU- R5[S55nXx-nUbsSU- nURSS S 9n [R"X9)5nURURS5R[R5[S55nUR!U5R#U5n U $) zEStandard softmax self-attention, as in the original Transformer paperrZrz2Size mismatch between Q and K in softmax attentionr%Nrar#rjTr)rBr^rArEr3rr zeros_liker$rkrrall logical_andrrr>r) r9r]r^rrrfrgreadditive_causal_maskpadding_mask_allrfs r;rl1MegaMovingAverageGatedAttention.softmax_attentionfsW((1+  ) jjm #zz!}! !UVV9D $\\%q!!4 5 <  "#(#3#3Kxx#P #7#C#CQ_DZDZD\^cdj^k#l *B  # |+L+//B/E  ,,\;LML 6 6q 9 <