cCiy ^SrSSKJr SSKrSSKrSSKrSSKJ r J r J r SSK J r JrJrJrJr SSKJrJrJrJrJrJrJrJrJr SSKJrJr SS KJ r S S K!J"r" \ RF"\$5r%S r&S r'Sr(Sr)Sr*Sr+"SS\RXRZ5r."SS\RXRZ5r/"SS\RXRZ5r0"SS\RXRZ5r1"SS\RXRZ5r2"SS\RXRZ5r3"SS\RXRZ5r4"S S!\RXRZ5r5"S"S#\RXRZ5r6"S$S%\RXRZ5r7"S&S'\RXRZ5r8"S(S)\5r9S*r:S+r;\ "S,\:5"S-S.\RXRZ55r<\ "S,\:5"S/S0\955r=\ "S1\:5"S2S3\9\55r>"S4S5\RXRZ5r?\ "S6\:5"S7S8\9\55r@\ "S9\:5"S:S;\9\55rA"S<S=\RXRZ5rBS@S>jrC/S?QrDg)AzPyTorch ESM model.) annotationsN)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward)+TFBaseModelOutputWithPastAndCrossAttentions.TFBaseModelOutputWithPoolingAndCrossAttentionsTFMaskedLMOutputTFSequenceClassifierOutputTFTokenClassifierOutput) TFMaskedLanguageModelingLossTFModelInputTypeTFPreTrainedModelTFSequenceClassificationLossTFTokenClassificationLossget_initializerkeras shape_list unpack_inputs)check_embeddings_within_boundsstable_softmax)logging) EsmConfigzfacebook/esm2_t6_8M_UR50Drcb[R"USSS9up[R"U*U4SS9$)Naxis)tfsplitconcat)xx1x2s a/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/esm/modeling_tf_esm.py rotate_halfr'6s- XXa $FB 99rc2YR ((cUSS2SS2S[R"U5S2SS24nUSS2SS2S[R"U5S2SS24nX-[U5U--$)N)r shaper')r#cossins r&apply_rotary_pos_embr.;sb a%bhhqk"o%q( )C a%bhhqk"o%q( )C G A, --r(cFU[RRU5-$)zJMake layer symmetric in final two dimensions, used for contact prediction.)r linalgmatrix_transpose)r#s r& symmetrizer2Bs ryy))!, ,,r(c[R"USSS9n[R"USSS9n[R"USSS9nX-nXC- nX- nU$)z=Perform average product correct, used for contact prediction.rT)keepdimsr*)rr*)r reduce_sum)r#a1a2a12avg normalizeds r&average_product_correctr;GsS q"t ,B q"t ,B --8d 3C 'C )CJ r(cT^\rSrSrSrSS U4SjjjrU4SjrS SjrS SjrSr U=r $) TFRotaryEmbeddingSz Rotary position embeddings based on those in [RoFormer](https://huggingface.co/docs/transformers/model_doc/roformer). Query and keys are transformed by rotation matrices which depend on their relative positions. c,>[TU]US9 Xlg)Nname)super__init__dim)selfrDrA __class__s r&rCTFRotaryEmbedding.__init__Zs d#r(c T>[TU]U5 URSURS-4[R [ S5SS9UlURRSS[R"SURS[R S9UR- -- 5 g) Ninv_freqr?F)r+dtype initializer trainablei'r)startlimitdeltarK) rBbuild add_weightrDr float32rrIassignrange)rE input_shaperFs r&rQTFRotaryEmbedding.buildds  k" txx1}.bjjo^aNbns(    5RXXATXXQbjjY\`\d\dde f r(c`[R"U5Un[R"X0RRS9n[R "SX@R5n[R "XU4SS9SSSS2SS24n[R"U5[R"U54$)NrKz i, j -> ijrr) r r+rUrIrKeinsumr"r,r-)rEr# seq_dimensionseq_lentfreqsembs r&_compute_cos_sin"TFRotaryEmbedding._compute_cos_sinms{((1+m, HHWMM$7$7 8 ,==9iiR0tQ1ABvvc{BFF3K''r(cVURUSS9up4[XU5[X#U54$)Nr*)r[)r`r.)rEqkcos_embsin_embs r&callTFRotaryEmbedding.callvs:00"0E !W 5 W 5  r()rDrIN)rDint)r)rc tf.Tensorrdrkreturnztuple[tf.Tensor, tf.Tensor]) __name__ __module__ __qualname____firstlineno____doc__rCrQr`rg__static_attributes__ __classcell__rFs@r&r=r=Ss&  (  r(r=cP^\rSrSrSrSSU4SjjjrS SjrSrSrU=r $) TFEsmContactPredictionHeadzWPerforms symmetrization, apc, and computes a logistic regression on the output featuresc>[TU]US9 X0lXl[R R SUSSS9Ulg)Nr@rsigmoid regression)use_bias activationrA)rBrCeos_idx in_featuresrlayersDenserz)rEr~biasr}rArFs r&rC#TFEsmContactPredictionHead.__init__s@ d# &,,,,Q)Zf,gr(c*UR(agSUl[USS5b[[R"URR 5 URR SUR45 SSS5 gg!,(df  g=f)NTrz)builtgetattrr name_scoperzrArQr~rErVs r&rQ TFEsmContactPredictionHead.buildsi ::  4t , 8t334%%tT-=-=&>?54 944s (B Bc[R"XR:gUR5n[R"US5[R"US5-nX#SS2SSSS2SS24-nUSSS2SS24nUSSS2SS24n[ U5upEpgn[R "X$XV-Xw45n[[U55n[R"USS9n[R"URU5S5$)Nrr.r)rrrrpermr) r castr}rK expand_dimsrreshaper;r2 transposesqueezerz) rEtokens attentionseos_mask batch_sizerheadsseqlen_s r&rgTFEsmContactPredictionHead.calls776\\1:3C3CD>>(A.!1LL1dD!Q+>"?? SbS#2# . QR , /9*/E, E1ZZ ,XY -Z -CD \\*<@ zz$//*5q99r()rr}r~rz)TrN)r~rjr}rjri) rmrnrorprqrCrQrgrrrsrts@r&rvrvsAa   h h h h@::r(rvcL^\rSrSrSrSU4SjjrS SjrSrSSjrSr U=r $) TFEsmEmbeddingszN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c>[TU]US9 [RR UR UR [UR5SS9Ul [RR URUR [UR5SS9Ul UR(a.[RRURSS9UlOSUl[!USS5Ul[$R&"UR5SSS24UlUR*UlUR.UlUR0UlXlg) Nr@word_embeddings)embeddings_initializerrAposition_embeddings layer_normepsilonrAposition_embedding_typeabsolute)rBrCrr Embedding vocab_size hidden_sizerinitializer_rangermax_position_embeddingsremb_layer_norm_beforeLayerNormalizationlayer_norm_epsrrrr rU position_ids pad_token_id padding_idx token_dropout mask_token_idconfigrErrArFs r&rCTFEsmEmbeddings.__init__s+ d#$||55      #263K3K#L" 6  $)<<#9#9  * *   #263K3K#L& $:$    ' '#ll==fF[F[bn=oDO"DO(/v7PR\']$HHV%C%CDT1WM!..#11#11 r(cdUc+Ub[XRU5nOURU5nUc0[XRR 5 UR U5nUnUR(a[R"XR:HSS2SS2S4SU5nSn[R"[R"USS9[R5nXR:Hn [RRU [RSS9U- n USU- -SU - SS2SS4- nUR S:XaUR#U5n Xk- nUR$bUR%U5nUb9U[R"[R&"US5UR(5-nU$)NggQ?rr)rKrrr)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrrrrrr whererrr5rSmath count_nonzerorrrrrK) rE input_idsattention_maskr inputs_embedspast_key_values_length embeddingsmask_ratio_train src_lengths masked_tokensmask_ratio_observedrs r&rgTFEsmEmbeddings.calls  $A)M]M]_uv #JJ=Y  *9kk6L6L M 00;M#    90B0B#BAq$J"OQTV`aJ) ''"--R"H"**UK%););;M"$''"7"7 RZZ^`"7"ado"o #q+;';<DW@WYZ\`bfYf?ggJ  ' ': 5"&":":<"H   -J ?? &4J  %#bggbnn^R.PR\RbRb&ccJr(c[U5SSnUSn[R"URS-X0R-S-[RS9n[R "[R "US5U5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: tf.Tensor Returns: tf.Tensor Nrr)rNrOrKr)rr rUrint64 broadcast_tor)rErrVsequence_lengthrs r&r6TFEsmEmbeddings.create_position_ids_from_inputs_embedssr!/4 %a.xx""Q&o@P@P.PST.T\^\d\d r~~lA> LLr(cUR(agSUl[USS5bN[R"URR 5 URR S5 SSS5 [USS5bN[R"URR 5 URR S5 SSS5 [USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  N=f!,(df  N=f!,(df  g=f)NTrrr) rrr rrrArQrrrrrs r&rQTFEsmEmbeddings.builds ::  4*D 1 =t33889$$**40: 4. 5 At77<<=((..t4> 4t , 8t334%%tT4;;3J3J&KL54 9 :9>=54$E.E 3E) E E&) E7) rrrrrrrrrrri)NNNNr) rmrnrorprqrCrgrrQrrrsrts@r&rrs,>rs+ZM" M Mr(rc~^\rSrSrSU4SjjrSSjrS S SjjrS SjrSrU=r $) TFEsmSelfAttentionic >[TU]US9 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[RRUR[UR5SS9Ul[RRUR[UR5S S9Ul[RRUR[UR5S S9Ul[RR#UR$5UlU=(d [)US S 5UlSUlUR*S :XdUR*S:XacUR.Ul[RR1SUR.-S- UR[UR5S9UlO)UR*S:Xa[5URSS9UlUR6UlXlg)Nr@rembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()querykernel_initializerrAkeyvaluerr relative_keyrelative_key_queryrr)rrotaryrotary_embeddings)rDrA)rBrCrnum_attention_headshasattr ValueErrorrjattention_head_size all_head_sizerrrrrrrrDropoutattention_probs_dropout_probdropoutrrrrrdistance_embeddingr= is_decoderr)rErrrArFs r&rCTFEsmSelfAttention.__init__sA d#    : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $"&  ' '> 9T=Y=Y]q=q+1+I+ID (&+ll&<&<F222Q6(('6v7O7O'P'='D #  ) )X 5%64;S;SZm%nD " ++ r(c[U5SSURUR/-n[R"X5n[R "USS9$)Nrrrrrr)rrrr rr)rEr# new_x_shapes r&transpose_for_scores'TFEsmSelfAttention.transpose_for_scores@sG mCR(D,D,DdF^F^+__ JJq &||AL11r(c .URU5n USLn U (aUbUSn USn UnGOU (aCURURU55n URURU55n UnOUbuURURU55n URURU55n [R "USU /SS9n [R "USU /SS9n O@URURU55n URURU55n URU 5n XR S--n UR(aX4nURS:XaURX5up[R"XSS9nURS :XdURS :XGa;[U5Sn[R"[R"U[RS 9S 5n[R"[R"U[RS 9S5nUU- nURUUR -S- 5n[R""UU R$5nURS :Xa[R&"S U U5nUU-nOHURS :Xa8[R&"S U U5n[R&"SU U5nUU-U-nUbX-n[)US S9nUR+UUS9nUbUU-nUU -n[R,"USS9n[U5SSUR./-n[R0"UU5nU(aUU4OU4nUR(aUU4-nU$)NrrrrgrT transpose_brrrYrzbhld,lrd->bhlrzbhrd,lrd->bhlrtrainingrrr*)rrrrr r"rrrrmatmulrrrUrrrrrKrZrrrrr)rE hidden_statesr head_maskencoder_hidden_statesencoder_attention_maskpast_key_valueoutput_attentionsrmixed_query_layeris_cross_attention key_layer value_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapeoutputss r&rgTFEsmSelfAttention.callEsh!JJ}5 3$> ."<&q)I(+K3N 11$((;P2QRI33DJJ?T4UVK3N  '11$((=2IJI33DJJ}4MNK >!#4i"@qII))^A%6 $D1MK11$((=2IJI33DJJ}4MNK//0AB "$<$ 9T=Y=Y]q=q#M215J^^BHHZrxx,PRTUN^^BHHZrxx,PRSTN%6H#'#:#:8dFbFb;bef;f#g #%77+?ARAR#S ++~=+-995E{Th+i(#36N#N --1EE13;K[Zn1o./1yy9I9Vj/k,#36T#TWs#s  %/@ ))9C,,,J  - 9O'+5  ]F ",]";CR"@DDVDVCW"W =2IJ 6G=/2mM] ?? 11Gr(c,UR(agSUl[USS5be[R"URR 5 URR SSURR/5 SSS5 [USS5be[R"URR 5 URR SSURR/5 SSS5 [USS5be[R"URR 5 URR SSURR/5 SSS5 [USS5bO[R"URR 5 URR S5 SSS5 gg!,(df  GNS=f!,(df  N=f!,(df  N=f!,(df  g=f)NTrrrr) rrr rrrArQrrrrrrs r&rQTFEsmSelfAttention.builds` ::  4$ ' 3tzz/   $dkk.E.E!FG0 4 % 1txx}}-dDKK,C,CDE. 4$ ' 3tzz/   $dkk.E.E!FG0 4,d 3 ?t55::;&&,,T2<; @0/.-0/<;s03G3G#83G4+H G # G14 H H)rrrrrrrrrrrrrrNN)r#rkrlrkNNNNNFF)rrkrtf.Tensor | Nonerrrrrrrztuple[tuple[tf.Tensor]] | Noner bool | Nonerboolrlztuple[tf.Tensor]ri) rmrnrorprCrrgrQrrrsrts@r&rrs&P2,0&*26379=).e e)e$ e 0 e !1 e7e'ee eN33r(rc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$)TFEsmSelfOutputic >[TU]US9 [RR UR [ UR5SS9Ul[RRUR5Ul Xl gNr@denser rBrCrrrrrrrrhidden_dropout_probrrrs r&rCTFEsmSelfOutput.__init__j d#\\''   ?6C[C[3\cj(  ||++F,F,FG  r(cNURU5nURXS9nX- nU$NrrrrEr input_tensorrs r&rgTFEsmSelfOutput.call. =1  ] F % r(c@UR(agSUl[USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  g=fNTr rrr rrrArQrrrs r&rQTFEsmSelfOutput.buildm ::  4$ ' 3tzz/   $dkk.E.E!FG0/ 4// 3B BrrrrriF rmrnrorprCrgrQrrrsrts@r&rrs HHr(rcT^\rSrSrSU4SjjrSrSSjrSSjrSrU=r $) TFEsmAttentionic>[TU]US9 [USS9Ul[ USS9Ul[ 5Ul[RRURSS9Ul Xl g)Nr@rEoutput LayerNormr)rBrCrrEr output_layerset pruned_headsrrrrr.rrs r&rCTFEsmAttention.__init__sa d#&vF; +FBE88AVAV]h8i r(c[eriNotImplementedError)rErs r& prune_headsTFEsmAttention.prune_heads!!r(c URU5n URU UUUUUUU5n URU SU5n U 4U SS-n U $)Nrr)r.rEr/) rErrrrrrrrhidden_states_ln self_outputsattention_outputrs r&rgTFEsmAttention.callsk >>-8yy    ! "      ,,\!_mL#% QR(88r(cUR(agSUl[USS5bN[R"URR 5 URR S5 SSS5 [USS5bN[R"URR 5 URR S5 SSS5 [USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  N=f!,(df  N=f!,(df  g=f)NTrEr/r.) rrr rrErArQr/r.rrrs r&rQTFEsmAttention.builds ::  4 & 2tyy~~. %/ 4 . :t00556!!''-7 4d + 7t~~223$$dD$++2I2I%JK43 8 /.7643r)r.rrr/r1rErir ) rmrnrorprCr6rgrQrrrsrts@r&r+r+s5" "#4 L Lr(r+c@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) TFEsmIntermediatei c >[TU]"S0UD6 [RR UR [ UR5SS9UlXl g)Nr)unitsrrA) rBrCrrrintermediate_sizerrrrrErkwargsrFs r&rCTFEsmIntermediate.__init__ sO "6"\\''**.v/G/GH(   r(cbURUS9n[RRU5nU$)Ninputs)rr nngelu)rErs r&rgTFEsmIntermediate.calls* - 8  =1 r(c@UR(agSUl[USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  g=fr"r#rs r&rQTFEsmIntermediate.buildr%r&rrrrrrrkrlrkrir)rts@r&rArA s HHr(rAc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) TFEsmOutputi$c >[TU]US9 [RR UR [ UR5SS9Ul[RRUR5Ul Xl grrrs r&rCTFEsmOutput.__init__%rr(cNURU5nURXS9nX- nU$rrrs r&rgTFEsmOutput.call-r r(c@UR(agSUl[USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  g=fr") rrr rrrArQrrErs r&rQTFEsmOutput.build3sm ::  4$ ' 3tzz/   $dkk.K.K!LM0/ 4//r&r'rir(r)rts@r&rUrU$s NNr(rUcN^\rSrSrSU4SjjrSSjrSSjrSrU=r$) TFEsmLayeri<c>[TU]US9 URUlSUl[ USS9UlUR UlURUlUR(a/UR (d[US35e[ U5Ul [USS9Ul [USS9Ul [RR!UR"SS9UlXlg) Nr@r attentionz> should be used as a decoder model if cross attention is added intermediater-r.r)rBrCchunk_size_feed_forward seq_len_dimr+r_radd_cross_attention RuntimeErrorcrossattentionrAr`rUr/rrrrr.rrs r&rCTFEsmLayer.__init__=s d#'-'E'E$'[A ++#)#=#=  # #??"dV+i#jkk"0"8D -f>J'X>88AVAV]h8i r(c UbUSSOSn URUUUUU US9n U Sn UR(a U SSn U Sn OU SSn SnUR(aZUbW[US5(d[SUS35eUbUS SOSnUR U UUUUUUUS 9nUSn U USS-n USnW U-n UR U 5nUR US 9nURUXS 9nU4U -n UR(aU W 4-n U $) Nr)rrrrrrrez'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r*rr)rrr)r_rrAttributeErrorrer.r`r/)rErrrrrrrrself_attn_past_key_valueself_attention_outputsr<rpresent_key_valuecross_attn_present_key_valuecross_attn_past_key_valuecross_attention_outputslayernorm_outputintermediate_output layer_outputs r&rgTFEsmLayer.callMs:H9S>"1#5Y] !%   /3 "0" 2!4 ??,Qr2G 6r : ,QR0G'+$ ??4@4!122$=dVD`` @N?Yrs(;_c %&*&9&9 %&)!!': ' # 7q9  7" ==G,C2+F ( 14P P >>*:;"//>N/O((-[TU]US9 Xl[UR5Vs/sHn[ USU3S9PM snUl[RRURSS9Ul gs snf)Nr@zlayer_._emb_layer_norm_afterr) rBrCrrUnum_hidden_layersr]layerrrrrry)rErrAirFs r&rCTFEsmEncoder.__init__sw d# GLVMeMeGfgGf!jn=Gfg $)LL$C$C))0F%D% !hsA9c xU (aSOSn U(aSOSn U(aURR(aSOSnU(aSOSn[UR5HunnU (aX4-n UbUUOSnUbUUOSnU"UUUUUUUU 5nUSnU(a UUS4- nU(dMSU US4-n URR(dMyUUS4-nM UR(aUR U5nU (aX4-n U (d[ SUUU U U455$[ UUU U US9$)NrDrrrrc30# UH nUcMUv M g7frirD).0vs r& $TFEsmEncoder.call..s" As )last_hidden_statepast_key_valuesrrcross_attentions)rrc enumerater{rytupler)rErrrrrr use_cacheroutput_hidden_states return_dictrall_hidden_statesall_self_attentionsall_cross_attentionsnext_decoder_cacher| layer_modulelayer_head_maskr layer_outputss r&rgTFEsmEncoder.callsz#7BD$5b4%64;;;Z;Zr`d#,R$(4OA|#$58H$H!.7.CilO3B3N_Q/TXN(%&! M*!,M"}R'8&::"  &9]1=M))//tT[[=T=T0UV? 4$ ' 3]]5::.KK%/.$ 4?>/.s3C* C;* C8; D )rrryr{ri) NNNNNNFFTFr)rts@r&rwrws9 "#"E N & &r(rwc@^\rSrSrSU4SjjrSSjrSSjrSrU=r$) TFEsmPooleric >[TU]"S0UD6 [RR UR [ UR5SSS9UlXl g)Ntanhr)rCrr|rArD) rBrCrrrrrrrrrFs r&rCTFEsmPooler.__init__sR "6"\\''$$.v/G/GH (   r(c6USS2S4nURUS9nU$)NrrJ)r)rErfirst_token_tensor pooled_outputs r&rgTFEsmPooler.calls*+1a40 *< = r(c@UR(agSUl[USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  g=fr"r#rs r&rQTFEsmPooler.buildr%r&rQrRrSrir)rts@r&rrs HHr(rc \rSrSrSr\rSrSrg)TFEsmPreTrainedModeli zz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. esmrDN) rmrnrorprqr config_classbase_model_prefixrrrDr(r&rr s Lr(ra2 This model inherits from [`TFPreTrainedModel`]. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.) This model is also a Keras [Model](https://www.tensorflow.org/api_docs/python/tf/keras/Model) subclass. Use it as a regular Keras model and refer to the TF/Keras documentation for all matters related to general usage and behavior. Parameters: config ([`EsmConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~TFPreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_ids (`tf.Tensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`tf.Tensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) position_ids (`tf.Tensor` of shape `({0})`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) head_mask (`tf.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. inputs_embeds (`tf.Tensor` of shape `({0}, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. z]The bare ESM Model transformer outputting raw hidden-states without any specific head on top.c^\rSrSrSrS/rS U4SjjrS SjrSrSSjr Sr SSS jjr S r S r U=r$)TFEsmMainLayerica The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. rc :>[TU]"S SU0UD6 XlURUl[ USS9Ul[ USS9UlU(a [USS9OSUl [URRURR-SSS9Ul g) NrArr@encoderpoolerT contact_head)r~rrArD)rBrCrrrrrwrrrrvrzrr)rEradd_pooling_layerrArGrFs r&rCTFEsmMainLayer.__init__vs -d-f-  ++)&|D#F; cd d  "$Y/K  &$]3CR8KTU U!, J  "%& "#fs4<<+=+='>>O%/0B10E%Fr%J "  !WW:zDZ7Z*[cdeN??)%'#9 +  *.9$'== ??hh/G--dA ._a0PQa &K''+^5I5IJK&1N1dA:4N&N ##-.E#F &(jj'*>q*A1FZ[\F]_stu_v)w' #q!-*A!Q VWBW*X'&(jj!5a!8!Q@TUV@W X' ##%''*AIYI_I_"`++c)9)?)?@;;x7G7M7MN"$++bkk'CZ.[]m"n ??5A&(WW-CKbKhKh%i ".1*=S2T.U +.!32HDRSUV2W/.!32HDRVXYIY2Z/035T/TX`.` +.2 +  % %!>!>>I,,*2"7#B+/!5#'  *!,FJkkF] / Bcg  #$ $ >-'+;;)77&11,==   r(cU"XSSS9Rn[R"USS9n[R"X#R5nX2SS2SSS4-nX2SS2SSSS2S4-nUR X5$)NT)rrrrr)rr stackrrKr)rErrattnss r&predict_contactsTFEsmMainLayer.predict_contacts8szV`deppQ' = 4t 344 4q$ 677  //r()rrrrrrr)TNri)rz tf.Variable NNNNNNNNNNNNF)rTFModelInputType | Nonernp.ndarray | tf.Tensor | Nonerrrrrrrrrrr+tuple[tuple[np.ndarray | tf.Tensor]] | NonerrrrrrrrrrrlATFBaseModelOutputWithPoolingAndCrossAttentions | tuple[tf.Tensor])rmrnrorprq_keys_to_ignore_on_load_missingrCrQrrrrgrrrrsrts@r&rrcs (7&7#  ."/:" .28<6:377;?C@DGK!%)-,0#'W *W 6W 4 W 1 W 5 W  =W !>W EW W 'W *W !W W  KW r 0 0r(rc^\rSrSrS S U4Sjjjr\\"\RS55\ "\ \ \ S9S S Sjj555r SrS SjrSrU=r$) TFEsmModeliEcL>[TU]"U/UQ70UD6 [XSS9Ulg)NrrrA)rBrCrr)rErrrKrGrFs r&rCTFEsmModel.__init__Js) 3&3F3!&TYZr(batch_size, sequence_length checkpoint output_typerc<URUUUUUUUUU U U U U S9 nU$)aK encoder_hidden_states (`tf.Tensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`tuple[tuple[tf.Tensor]]` of length `config.n_layers`) contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*, defaults to `True`): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). Set to `False` during training, `True` during generation ) rrrrrrrrrrrrr)r)rErrrrrrrrrrrrrrs r&rgTFEsmModel.callOsFV(()%'"7#9+/!5# r(c8URRX5$rirrrErrs r&rTFEsmModel.predict_contactsxx((@@r(cUR(agSUl[USS5bO[R"URR 5 URR S5 SSS5 gg!,(df  g=f)NTr)rrr rrrArQrs r&rQTFEsmModel.buildsZ ::  4 % 1txx}}-t$.- 2--s A88 B)rr)TrRr)rrrrrrrrrrrrrrrrrrrrrrrrrrrlrri)rmrnrorprCrrESM_INPUTS_DOCSTRINGformatr_CHECKPOINT_FOR_DOCr _CONFIG_FOR_DOCrgrrQrrrsrts@r&rrEs [[ *+?+F+FGd+ef&B$.28<6:377;?C@DGK!%)-,0#' %3*3634 3 1 3 5 3 =3!>3E33'3*3!33 K3 g3jA%%r(rz1ESM Model with a `language modeling` head on top.c ^\rSrSrS/rS/rU4SjrSrSrSr \ \ "\ RS55\"\\\S S 9SSS jj555rS rSS jrSrU=r$)TFEsmForMaskedLMirrcd>[TU]U5 UR(a[R S5 [ USSS9Ul[USS9UlUR(a[R"[RRUR5SSS55 UR R R"R%S 5 SSS5 UR R R"R&S URlgg!,(df  NL=f) NzjIf you want to use `EsmForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrlm_headr@rrr r)rBrCrloggerwarningrr TFEsmLMHeadrtie_word_embeddingsr rospathjoin _name_scoperrrQweightsdecoderrErrFs r&rCTFEsmForMaskedLM.__init__s     NN1  "&EN"6 :  % %rww||D,<,<,>|Ufgh##3399,Gi#'88#6#6#F#F#N#Nq#QDLL &hhs +0D!! D/c.URR$rirrrs r&get_output_embeddings&TFEsmForMaskedLM.get_output_embeddingss||###r(c$XRlgrir)rEnew_embeddingss r&set_output_embeddings&TFEsmForMaskedLM.set_output_embeddingss - r(cUR$ri)rrs r& get_lm_headTFEsmForMaskedLM.get_lm_heads ||r(rz)rrrmaskc 4U bU OURRn URUUUUUUUU U U U S9 n U SnURU5nSnUbUR XS9nU (dU4U SS-nUbU4U-$U$[ UUU R U RS9$)a labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` kwargs (`dict[str, any]`, *optional*, defaults to `{}`): Used to hide legacy arguments that have been deprecated. N) rrrrrrrrrrr)labelslogitsrlossrrr)ruse_return_dictrrhf_compute_lossr rr)rErrrrrrrrrrrrrrprediction_scoresmasked_lm_lossr-s r&rgTFEsmForMaskedLM.calls>&1%6 .6 '6 *6 !6 6  -6 g6 pA ) )r(rcF^\rSrSrSrSU4SjjrSSjrSrSrSr U=r $) riz&ESM Head for masked language modeling.c>[TU]US9 [RR UR [ UR5SS9Ul[RRURSS9Ul UR(aSUl OB[RR UR[ UR5SSS9Ul Xlg) Nr@rrrrrF)rrAr{)rBrCrrrrrrrrrrrrrrrs r&rCTFEsmLMHead.__init__s d#\\''   ?6C[C[3\cj(   ,,99&BWBW^j9k  % %DL <<--!!#263K3K#L .DL  r(cUR(agSUlURSURR4SSS9Ul[ USS5be[ R"URR5 URRSSURR/5 SSS5 [ USS5be[ R"URR5 URRSSURR/5 SSS5 [ USS5bURR(df[ R"URR5 URRSSURR/5 SSS5 ggg!,(df  GN=f!,(df  N=f!,(df  g=f)NTrzeros)r+rLrMrrr)rrRrrrrr rrrArQrrrrrs r&rQTFEsmLMHead.buildsT ::  OOF4;;3I3I2KY`lpOq 4$ ' 3tzz/   $dkk.E.E!FG0 4t , 8t334%%tT4;;3J3J&KL5 4D ) 5dkk>]>]t||001 ""D$ 0G0G#HI21?^ 5 0/5421s$>3F=13G?3G = G  G G.cSUR0$)Nr)rrs r&get_biasTFEsmLMHead.get_bias+s ""r(cVURU5n[RRU5nUR U5nUR R (a.[R"X RSS9UR-nU$URU5UR-nU$)NTr) rr rLrMrrrrrr)rEfeaturesr#s r&rgTFEsmLMHead.call.s JJx  EEJJqM OOA  ;; * * !\\t[TU]U5 URUlXl[ USSS9Ul[ USS9UlgNFrr classifierr@)rBrC num_labelsrrrTFEsmClassificationHeadr0rs r&rC'TFEsmForSequenceClassification.__init__Es@   ++ !&EN1&|Lr(rrc 4U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UcSOUR Xm5nU (dU 4U SS-nUbU4U-$U$[ UU U R U RS9$)a^ labels (`tf.Tensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrrrrrrrrrr)rrrr0rr rrrErrrrrrrrrrrrrrr-s r&rg#TFEsmForSequenceClassification.callMs4&1%[TU]U5 URUl[USSS9Ul[ R RUR5Ul [ R RURSS9Ul Xl gr/) rBrCr1rrrrrrrrr0rrs r&rC$TFEsmForTokenClassification.__init__so   ++!&EN||++F,F,FG ,,,,V->->\,R r(rrc RU bU OURRn URUUUUUUUU U S9 n U Sn URXS9n UR U 5n UcSOUR Xm5nU (dU 4U SS-nUbU4U-$U$[ UU U RU RS9$)z labels (`tf.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr5rrrr) rrrrr0rr rrr6s r&rg TFEsmForTokenClassification.calls0&1%[TU]US9 [RR UR [ UR5SSS9Ul[RRUR5Ul [RR UR[ UR5SSS9Ul Xlg)Nr@rr)rr|rAlinearout_proj)rBrCrrrrrrrrrrr1rGrrs r&rC TFEsmClassificationHead.__init__s d#\\''   .v/G/GH (  ||++F,F,FG  **   .v/G/GH +   r(cUSS2SSS24nURX2S9nURU5nURX2S9nURU5nU$)Nrr)rrrG)rEr*rr#s r&rgTFEsmClassificationHead.callsR Q1W  LLL . JJqM LLL . MM! r(cHUR(agSUl[USS5be[R"URR 5 URR SSURR/5 SSS5 [USS5bf[R"URR 5 URR SSURR/5 SSS5 gg!,(df  N=f!,(df  g=f)NTrrG) rrr rrrArQrrrGrs r&rQTFEsmClassificationHead.builds ::  4$ ' 3tzz/   $dkk.E.E!FG0 4T * 6t}}112 ##T41H1H$IJ32 70/32s3D3D D D!)rrrrrGrir() rmrnrorprqrCrgrQrrrsrts@r&r2r2s7" K Kr(r2c[R"X:g[R5n[R"USS9U-U-nXA-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: tf.Tensor x: Returns: tf.Tensor rr)r rrcumsum)rrrrincremental_indicess r&rrs@ 779+RXX 6D99T25KKtS  ,,r()rr-r=rr)r)Erq __future__rrnumpynp tensorflowr file_utilsrrrmodeling_tf_outputsrr r r r modeling_tf_utilsr rrrrrrrrtf_utilsrrutilsrconfiguration_esmr get_loggerrmrrrr'r.r2r;rLayerr=rvrrrr+rArUr]rwrrESM_START_DOCSTRINGrrrrrr-r=r2r__all__rDr(r&r^s" qq   G(   H %1) .- )  **) X%:!3!3%:PmMell((mM`d3++d3NHell((H02LU\\''2LjH **H2N%,,$$N0fL##fLRZ&5<<%%Z&|H%,,$$H:, 'Tc[0U\\''[0 [0|cK%%K% K%\MObcj)+-Ij)dj)Z3%,,$$3l K,%9;WK,K,\ NM"68QNMNMb%Kell00%KP -  r(