cCi-(SrSSKrSSKrSSKJrJr SSKrSSKJr SSKJ r J r J r SSK J r SSKJrJrJr SS KJr SS KJr SS KJrJrJrJrJrJrJrJr SS KJ r SS K!J"r"J#r#J$r$ SSK%J&r&J'r' SSK(J)r) SSK*J+r+ \'RX"\-5r.Sr/"SS\R`5r1"SS\R`5r2"SS\R`5r3"SS\R`5r4"SS\R`5r5"SS\R`5r6"SS \R`5r7"S!S"\5r8"S#S$\R`5r9"S%S&\R`5r:"S'S(\R`5r;"S)S*\R`5r<\&"S+S,\ 55r=\&"S-S.9"S/S0\=55r>\&"S1S2\=55r?\&"S3S.9"S4S5\=\55r@\&"S6S.9"S7S8\=55rA\&"S9S:\=55rB\&"S;S<\=55rC\&"S=S>\=55rD/S?QrEg)@zPyTorch RemBERT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)deprecate_kwarg) RemBertConfigc ^SSKnSSKnSSKn[ RRU5n[RSU35 URRU5n/n/n UH{umn [U4SjS55(aM"[RSTSU 35 URRUT5n URT5 U RU 5 M} [X5GHumn TR!S S 5mTR#S 5m[S T55(a)[RS S R%T535 MjUn THn UR'SU 5(aUR#SU 5nOU /nUSS:Xd USS:Xa [)U S5n OZUSS:Xd USS:Xa [)U S5n O;USS:Xa [)U S5n O%USS:Xa [)U S5n O[)XS5n [/U5S:dM[1US5nXn M W SSS:Xa [)U S5n OU S:XaUR3U 5n U R4U R4:wa&[7SU R4SU R4S 35e[RS!T35 [<R>"U 5U l GM U$![a [R S5 ef=f![*a8 [RSR-S R%T555 GMf=f![8a1nU=R:U R4U R44- sleSnAff=f)"z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from c3,># UH oT;v M g7fN).0denynames f/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/rembert/modeling_rembert.py -load_tf_weights_in_rembert..GsX(Wt|(Ws)adam_vadam_moutput_embeddingclszLoading TF weight z with shape zbert/zrembert//c3,# UH nUS;v M g7f))r+r,AdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepNr$)r%ns r(r)r*Xs  n nsz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierz Skipping {}ri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )!renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variablesany load_variableappendzipreplacesplitjoin fullmatchgetattrAttributeErrorformatlenint transposeshape ValueErrorAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr@nptftf_path init_varsnamesarraysrZarraypointerm_name scope_namesnumer's @r(load_tf_weights_in_rembertrp1sL ggoo01G KK8 BC''0I E F e X(WX X X  (l5'BC&&w5 T e!5) e||GZ0zz#      KK)CHHTN#34 5 F||,f55 hhy&9 %h 1~)[^w-F!'84Q=0KNf4L!'62Q#33!'84Q7*!'<8%g1~>G;1$+a.)!,+, #$<= (gx0G x LL'E }} + >'--@QRWR]R]Q^^i!jkk,  078''. c*d LS   Q   n&KK 4 4SXXd^ DE  FFw}}ekk2 2F  s6 K:K<AM!K9<=L>=L> M< ,M77M<c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\\RS\ S \R4 S jjr S r U=r$) RemBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cv>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR#S[$R&"UR5R)S5SS9 g)N) padding_idxeps position_ids)rF) persistent)super__init__r Embedding vocab_sizeinput_embedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr^arangeexpandselfrb __class__s r(r|RemBertEmbeddings.__init__s !||   v::H[H[ $&<<0N0NPVPkPk#l %'\\&2H2H&JeJe%f"f&A&AvG\G\]zz&"<"<=   ELL)G)GHOOPWXej  input_idstoken_type_idsrx inputs_embedspast_key_values_lengthreturncUbUR5nOUR5SSnUSnUcURSS2XWU-24nUc8[R"U[RURR S9nUcUR U5nURU5nXH-n URU5n X- n URU 5n URU 5n U $)Nryrdtypedevice) sizerxr^zeroslongrrrrrr) rrrrxrr input_shape seq_lengthr embeddingsrs r(forwardRemBertEmbeddings.forwards  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  !"[[EJJtO`O`OgOghN   00;M $ : :> J": "66|D) ^^J/ \\*- r)rrrrr)NNNNr)__name__ __module__ __qualname____firstlineno____doc__r|rr^ LongTensor FloatTensorrXTensorr__static_attributes__ __classcell__rs@r(rrrrsQ (15593759&' E,,-!!1!12u//0    1 12  !$  rrrcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) RemBertPoolerc>[TU]5 [R"URUR5Ul[R "5Ulgr#)r{r|rLinear hidden_sizedenseTanh activationrs r(r|RemBertPooler.__init__s9 YYv1163E3EF '')r hidden_statesrc\USS2S4nURU5nURU5nU$)Nr)rr)rrfirst_token_tensor pooled_outputs r(rRemBertPooler.forwards6+1a40 #56  6 r)rr rrrrr|r^rrrrrs@r(rrs($ U\\ellrrc^\rSrSrSU4Sjjr\"SSSS9SS\RS\\RS \\RS \\RS\\ S \ S \\RS \ 4Sjj5r SrU=r$)RemBertSelfAttentionc>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlUR"UlX lg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r{r|rnum_attention_headshasattrr[rXattention_head_size all_head_sizerrquerykeyvaluerattention_probs_dropout_probr is_decoder layer_idxrrbrrs r(r|RemBertSelfAttention.__init__s1     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9FFrTdimrr )rZrviewrrrY isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater^matmulmathsqrtr functionalsoftmaxrpermute contiguousrr)rrrrrrrr batch_sizer_ query_layerris_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapes r(rRemBertSelfAttention.forwards%2$7$7!  JJ} % T*b$":":D  &/+>??,77;;DNNK %*9*O*O'*9*N*N'&5#2D.- /"=*+224>>BGGI-44T^^DKKK(j"d&>&>@X@XY1a   >*j"d&>&>@X@XY1a  *7It)<)C)C{DNN=M~<^*& ;&*_FY*Z*ZAEO..t~~>!<< Y5H5HR5PQ+dii8P8P.QQ  %/.@ --//0@b/I,,7  - 9O _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD o--r) rrrrrrrrrr#NNNNFN)rrrrr|rr^rrrr booltuplerrrrs@r(rrs#*%0A6R7;15=A+/"'15R.||R.!!2!23R.E--. R. ((9(9: R. "% R. R.!.R. R.SR.rrcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)RemBertSelfOutputi7c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNrv) r{r|rrrrrrrrrrs r(r|RemBertSelfOutput.__init__8s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rr input_tensorrcpURU5nURU5nURX-5nU$r#rrrrrrs r(rRemBertSelfOutput.forward>5 =1  ]3 }'CD rrrrrrs@r(rr76> U\\RWR^R^rrc^\rSrSrSU4SjjrSrSS\RS\\RS\\RS\\RS\\ S \\ S \\RS \ \R4S jjr S rU=r$)RemBertAttentioniEc|>[TU]5 [XS9Ul[ U5Ul[ 5Ulg)Nr)r{r|rrroutputset pruned_headsrs r(r|RemBertAttention.__init__Fs0 (E '/ Erc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)rWrrrrrrrrrrrrunion)rheadsindexs r( prune_headsRemBertAttention.prune_headsMs  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rrrrrrrrrc lURUUUUUUUS9nURUSU5n U 4USS-n U $)Nrrrrrrrr)rr) rrrrrrrr self_outputsattention_outputoutputss r(rRemBertAttention.forward`s\yy )"7+/)!  ;;|A F#% QR(88r)rrrr#r)rrrrr|rr^rrrr rrrrrrs@r(r r Es";,7;15=A+/,115||!!2!23E--.  ((9(9:  "% $D>!. u|| rr cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertIntermediateiyc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr#) r{r|rrrintermediate_sizerr hidden_actstrr intermediate_act_fnrs r(r|RemBertIntermediate.__init__zs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rrrcJURU5nURU5nU$r#rr&rrs r(rRemBertIntermediate.forwards& =1 00? rr)rrs@r(r!r!ys(9U\\ellrr!cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) RemBertOutputic(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r{r|rrr#rrrrrrrrs r(r|RemBertOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rrrrcpURU5nURU5nURX-5nU$r#rrs r(rRemBertOutput.forwardr rr rrs@r(r-r-r rr-c.^\rSrSrSU4SjjrSS\R S\\RS\\RS\\RS\\RS\\ S \\ S \\R S \ \R 4S jjr S r SrU=r$) RemBertLayericr>[TU]5 URUlSUl[ X5UlUR UlURUlUR(a-UR (d[US35e[ XS9Ul [U5Ul [U5Ul g)Nrz> should be used as a decoder model if cross attention is addedr)r{r|chunk_size_feed_forward seq_len_dimr  attentionradd_cross_attentionr[crossattentionr! intermediater-rrs r(r|RemBertLayer.__init__s '-'E'E$)&< ++#)#=#=  # #?? D6)g!hii"26"OD /7#F+ rrrrrencoder_attention_maskrrrrc PURUUUUUUS9n U Sn U SSn UR(aDUbA[US5(d[SUS35eUR U UUUUUUS9n U Sn XSS-n [ UR URURU 5n U 4U -n U $)N)rrrrrrrr9z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r7rrr[r9rfeed_forward_chunkr5r6)rrrrrr<rrrself_attention_outputsrrcross_attention_outputs layer_outputs r(rRemBertLayer.forwards"& )/+) "0" 2!4(, ??4@4!122 =dVDDD '+&9&9 5#&; /"3-':' # 7q9  ;;G0  # #T%A%A4CSCSUe  /G+rcJURU5nURX!5nU$r#)r:r)rrintermediate_outputrAs r(r>RemBertLayer.feed_forward_chunks)"//0@A{{#6I r)r8r7r5r9r:rrr6r#)NNNNNFN)rrrrr|r^rrrr rrrr>rrrs@r(r3r3s ,$7;15=A>B+/,115.||.!!2!23.E--. . ((9(9: . !)):): ; ."%.$D>.!.. u|| .brr3c&^\rSrSrU4SjrSS\R S\\RS\\RS\\RS\\RS\\ S \\ S \ S \ S \ S \\R S\ \ \ 44SjjrSrU=r$)RemBertEncoderic 2>[TU]5 Xl[R"UR UR 5Ul[R"[UR5Vs/sH n[XS9PM sn5Ul SUl gs snf)NrF)r{r|rbrrrrembedding_hidden_mapping_in ModuleListrangenum_hidden_layersr3layergradient_checkpointing)rrbirs r(r|RemBertEncoder.__init__ss  +-99V5P5PRXRdRd+e(]]uU[UmUmOn#oOn!L$EOn#op &+#$ps-Brrrrr<r use_cacheroutput_hidden_states return_dictrrc UR(a/UR(aU(a[RS5 SnU(a1Uc.[ [ UR S9[ UR S95nU(a@[U[5(a+[RS5 [R"U5nURU5nU (aSOSn U(aSOSn U(aUR R(aSOSn[UR5HfunnU (aX4-n UbX?OSnU"UUUUUUU5nUSnU(dM7U US4-n UR R(dM]UUS4-nMh U (aX4-n U (d[S UUU U U455$[UUU U US 9$) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rbzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r$rrr>c30# UH nUcMUv M g7fr#r$)r%vs r(r))RemBertEncoder.forward..%s" As )last_hidden_staterr attentionscross_attentions)rNtrainingrD warning_oncer r rbrrfrom_legacy_cacherIr8 enumeraterMr)rrrrrr<rrQrrRrSrall_hidden_statesall_self_attentionsall_cross_attentionsrO layer_modulelayer_head_mask layer_outputss r(rRemBertEncoder.forwards  & &4==##p" 01,dkk2RT`hlhshsTtuO OU;;   \  2CCOTO88G "6BD$5b4%64;;;Z;Zr`d(4OA|#$58H$H!.7.CilO(%&!M*!,M  &9]1=MB+/$("'%* 15O ||O !!2!23O E--. O ((9(9: O !)):): ; O "%O D>O  O #O O !.O  u?? @O O rrGcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertPredictionHeadTransformi:cp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) r{r|rrrrrr$r%r transform_act_fnrrrs r(r|'RemBertPredictionHeadTransform.__init__;s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrrrclURU5nURU5nURU5nU$r#)rrirr*s r(r&RemBertPredictionHeadTransform.forwardDs4 =1 --m< }5 r)rrrirrs@r(rgrg:s)UU\\ellrrgcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertLMPredictionHeadiKcn>[TU]5 [R"URUR 5Ul[R"UR UR5Ul[URUl [R"UR URS9Ul gr)r{r|rrroutput_embedding_sizerr~decoderr r$rrrrs r(r| RemBertLMPredictionHead.__init__Lsz YYv1163O3OP yy!=!=v?P?PQ  !2!23f&B&BH]H]^rrrcURU5nURU5nURU5nURU5nU$r#)rrrrqr*s r(rRemBertLMPredictionHead.forwardSs@ =1  6 }5  ]3 r)rrrqrrrs@r(rnrnKs)_U\\ellrrncb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)RemBertOnlyMLMHeadi\cB>[TU]5 [U5Ulgr#)r{r|rn predictionsrs r(r|RemBertOnlyMLMHead.__init__]s 26:rsequence_outputrc(URU5nU$r#rx)rrzprediction_scoress r(rRemBertOnlyMLMHead.forwardas ,,_=  rr|rrs@r(rvrv\s(;!u||! !!rrvc2\rSrSr%\\S'\rSrSr Sr Sr g)RemBertPreTrainedModelifrbrembertTc[U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 URb2URRURR5 gg[U[R5(aJURRR5 URRRS5 gg)zInitialize the weightsg)meanstdNg?)rrrr7r`normal_rbinitializer_ranger:zero_r}rurfill_)rmodules r( _init_weights$RemBertPreTrainedModel._init_weightsms fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).rr$N) rrrrr __annotations__rpload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrr$rr(rrfs 0O!&*#*rra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introc"^\rSrSrSU4SjjrSrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ S\ \S\ \S\ \S\ \S\ \ RS\\\44Sjj5rSrU=r$) RemBertModeli~c>[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r{r|rbrrrrGencoderrpooler post_init)rrbadd_pooling_layerrs r(r|RemBertModel.__init__sK   +F3%f- /@mF+d  rc.URR$r#rrrs r(get_input_embeddings!RemBertModel.get_input_embeddingss...rc$XRlgr#r)rrs r(set_input_embeddings!RemBertModel.set_input_embeddingss */'rcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrMr7r)rheads_to_prunerMrs r( _prune_headsRemBertModel._prune_headss< +002LE LL  u % / / ; ;E B3rrrrrxrrrr<rrQrrRrSrrclU bU OURRn U bU OURRn U bU OURRn URR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnSnU b:[U [5(dU SSRSOU R5nUc[R"UUU-4US9nUc$[R "U[R"US9nUR%X/5nURR(aEUbBUR5unnnUU4nUc[R"UUS9nUR'U5nOSnUR)XPRR*5nUR-UUUUUS 9nUR/UUUUUU U U U U US 9 nUSnUR0bUR1U5OSnU (d UU4US S-$[3UUUR4UR6UR8UR:S 9$) NFzDYou cannot specify both input_ids and inputs_embeds at the same timeryz5You have to specify either input_ids or inputs_embedsrr)rr)rrxrrr) rrrr<rrQrrRrSrr)rX pooler_outputrrrYrZ)rbrrRuse_return_dictrrQr[%warn_if_padding_and_no_attention_maskrrrr rZget_seq_lengthr^onesrrget_extended_attention_maskinvert_attention_mask get_head_maskrLrrrrrrrYrZ)rrrrrxrrrr<rrQrrRrSrrrrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsrzrs r(rRemBertModel.forwards$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T!"  &"/599 "1%++B/$335 #  !"ZZ*jCY6Y)ZdjkN  !"[[EJJvVN150P0PQ_0m ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +&&y++2O2OP ??%)'#9 + ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   r)rbrrr)TNNNNNNNNNNNNNN)rrrrr|rrrrrr^rrr rrrrrrrrrs@r(rr~s} /0C155959371559=A>B+/$(,0/3&*15m E,,-m !!1!12m !!1!12 m u//0 m E--. m   1 12m  ((9(9:m !)):): ;m "%m D>m $D>m 'tnm d^m !.m uBB C!m m rrc^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \ S\ \ S\ \ S\\\44Sjj5rSSjr\S\ 4Sj5rSrU=r$)RemBertForMaskedLMicls.predictions.decoder.weightc>[TU]U5 UR(a[R S5 [ USS9Ul[U5UlUR5 g)NznIf you want to use `RemBertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Fr r{r|rrDwarningrrrvr.rrs r(r|RemBertForMaskedLM.__init__sR     NN1  $FeD %f- rcBURRR$r#r.rxrqrs r(get_output_embeddings(RemBertForMaskedLM.get_output_embeddings-xx##+++rc8XRRlgr#rrnew_embeddingss r(set_output_embeddings(RemBertForMaskedLM.set_output_embeddings0'5$rrrrrxrrrr<labelsrrRrSrc U bU OURRn URUUUUUUUUU U U S9 n U SnURU5nSnU bF[ 5nU"UR SURR 5U R S55nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. N) rrrxrrrr<rrRrSrryr>losslogitsrrY) rbrrr.rrr~rrrY)rrrrrxrrrr<rrrRrSrrzr}masked_lm_lossloss_fctrs r(rRemBertForMaskedLM.forward3s,&1% &IIrcg)z Legacy correction: RemBertForMaskedLM can't call `generate()` from `GenerationMixin`, even though it has a `prepare_inputs_for_generation` method. Fr$)r.s r( can_generateRemBertForMaskedLM.can_generatezs rr.r) NNNNNNNNNNNNr#)rrrr_tied_weights_keysr|rrrrr^rrrrrrrr classmethodrrrrs@r(rrsp:; ,6155959371559=A>B-1,0/3&*6 E,,-6 !!1!126 !!1!12 6 u//0 6 E--. 6   1 126  ((9(9:6 !)):): ;6 ))*6 $D>6 'tn6 d^6  un$ %6 6 p JTrrzS RemBERT Model with a `language modeling` head on top for CLM fine-tuning. c"^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ S\ \ RS\ \S\ \S\ \S\ \S\\\44Sjj5rSrU=r$)RemBertForCausalLMirc>[TU]U5 UR(d[R S5 [ USS9Ul[U5UlUR5 g)NzOIf you want to use `RemBertForCausalLM` as a standalone, add `is_decoder=True.`Frrrs r(r|RemBertForCausalLM.__init__sL    NNl m#FeD %f- rcBURRR$r#rrs r(r(RemBertForCausalLM.get_output_embeddingsrrc8XRRlgr#rrs r(r(RemBertForCausalLM.set_output_embeddingsrrrrrrxrrrr<rrrQrrRrSrc UbUOURRnURUUUUUUUUU U U U US9 nUSnURU5nSnU b*UR"UU 4SURR 0UD6nU(dU4USS-nUbU4U-$U$[ UUURURURURS9$)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, RemBertForCausalLM, RemBertConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/rembert") >>> config = RemBertConfig.from_pretrained("google/rembert") >>> config.is_decoder = True >>> model = RemBertForCausalLM.from_pretrained("google/rembert", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```N) rrrxrrrr<rrQrrRrSrr~r>)rrrrrYrZ) rbrrr. loss_functionr~rrrrYrZ)rrrrrxrrrr<rrrQrrRrSkwargsrrzr}lm_lossrs r(rRemBertForCausalLM.forwardsR&1%B+/-1$(,0/3&*Q E,,-Q !!1!12Q !!1!12 Q u//0 Q E--. Q   1 12Q  ((9(9:Q !)):): ;Q "%Q ))*Q D>Q $D>Q 'tnQ d^Q " u77 8#Q Q rrz RemBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$) RemBertForSequenceClassificationic0>[TU]U5 URUl[U5Ul[ R "UR5Ul[ R"URUR5Ul UR5 gr# r{r| num_labelsrrrrclassifier_dropout_probrrrr=rrs r(r|)RemBertForSequenceClassification.__init__si   ++#F+ zz&"@"@A ))F$6$68I8IJ rrrrrxrrrrrRrSrc RU bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUGbURR cUR S:XaSURlOoUR S:aNUR[R:XdUR[R:XaSURlOSURlURR S:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURR S:Xa=[5nU"U RSUR 5URS55nO,URR S:Xa[5nU"X5nU (dU 4U SS-nUbU4U-$U$[!UU U R"U R$S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrxrrrrRrSr regressionsingle_label_classificationmulti_label_classificationryr>r)rbrrrr= problem_typerrr^rrXrsqueezerrrrrrY)rrrrrxrrrrrRrSrrrrrrs r(r(RemBertForSequenceClassification.forwards(&1%E 'tnE d^E  u.. /E E rrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)RemBertForMultipleChoiceiLc>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) r{r|rrrrrrrrr=rrs r(r|!RemBertForMultipleChoice.__init__NsV  #F+ zz&"@"@A ))F$6$6: rrrrrxrrrrrRrSrc ZU bU OURRn UbURSOURSn Ub!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR UUUUUUUU U S9 n U Sn UR U 5n URU 5nURSU 5nSnUb[5nU"X5nU (dU4U SS-nUbU4U-$U$[UUU RU RS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrryrrr>r) rbrrZrrrrr=rrrrY)rrrrrxrrrrrRrS num_choicesrrrreshaped_logitsrrrs r(r RemBertForMultipleChoice.forwardXsX&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  ,, ))%'/!5#     ]3 / ++b+6  ')HO4D%''!"+5F)-)9TGf$ Ev E("!//))   r)r=rrr)rrrrr|rrr^rrrrrrrrrrs@r(rrLs266:59481559-1,0/3&*X E--.X !!2!23X !!1!12 X u001 X E--. X   1 12X ))*X $D>X 'tnX d^X  u// 0X X rrcR^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ 44Sjj5rSrU=r$)RemBertForTokenClassificationic.>[TU]U5 URUl[USS9Ul[ R "UR5Ul[ R"URUR5Ul UR5 gNFrrrs r(r|&RemBertForTokenClassification.__init__sk   ++#FeD zz&"@"@A ))F$6$68I8IJ rrrrrxrrrrrRrSrc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrryr>r) rbrrrr=rrrrrrY)rrrrrxrrrrrRrSrrzrrrrs r(r%RemBertForTokenClassification.forwards$&1%2 'tn2 d^2  u++ ,2 2 rr cr^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \ 44Sjj5rSrU=r$)RemBertForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 gr ) r{r|rrrrrr qa_outputsrrs r(r|$RemBertForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ rrrrrxrrstart_positions end_positionsrrRrSrc U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5nUR S5nSnUbUb[ UR55S:aUR S5n[ UR55S:aUR S5nURS5nURSU5 URSU5 [US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrryr) ignore_indexr>)r start_logits end_logitsrrY) rbrrrrQrrWrclamp_rrrrY)rrrrrxrrrrrrRrSrrzrrr total_loss ignored_indexr start_lossend_lossrs r(r#RemBertForQuestionAnswering.forwards&1% E--.> !!2!23> !!1!12 > u001 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  u22 3> > rr) rrrrrr r3rrrp)FrrrFtypingrrr^rtorch.nnrrr activationsr cache_utilsr r r generationrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrutils.deprecationrconfiguration_rembertr get_loggerrrDrpModulerrrrrr r!r-r3rGrgrnrvrrrrrrr r__all__r$rr(r/sQ " AA!CC)9   .ll,00   H %Pf3 3n BII i.299i.Z  0ryy0h "))  BII D-DNX RYYX xRYY" bii "!!*_**.  M )M  M `e/eeP g /g  g T Q '=Q  Q hd 5d d N? $:? ? DK "8K K \ r