cCi>SrSSKrSSKJrJr SSKrSSKJr SSKJr SSK J r SSK J r J r Jr SS KJr SS KJrJr SS KJr SS KJrJr SS KJr SSKJrJr SSKJr SSK J!r! \RD"\#5r$"SS\RJ5r&"SS\RJ5r'"SS\RP5r)"SS\RP5r*"SS\5r+\"SS\55r,"SS\,5r-\"S S!9"S"S#\,55r.\"S$S!9"S%S&\,\55r/S&S/r0g)'z/PyTorch TrOCR decoder model (based on RoBERTa).N)OptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) TrOCRConfigc^\rSrSrSrS\S\4U4SjjrS S\RS\S\ \R4U4S jjjr S r U=r $) TrOCRLearnedPositionalEmbedding+zF This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcL>SUl[TU] XR-U5 g)N)offsetsuper__init__)selfrr __class__s b/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/trocr/modeling_trocr.pyr!(TrOCRLearnedPositionalEmbedding.__init__0s"  ++5}E input_idspast_key_values_length position_idsc>Uc]URSSupE[R"X"U-[RURR S9R US5nOURS5n[TU]%X0R-5$)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr,expand unsqueezer forwardr)r"r'r(r)bszseq_lenr#s r$r5'TrOCRLearnedPositionalEmbedding.forward6s  $??2A.LC <<&(HPUPZPZcgcncncucufS"o (11!4Lw|kk9::r&)r)rN) __name__ __module__ __qualname____firstlineno____doc__intr!r/Tensorrr5__static_attributes__ __classcell__r#s@r$rr+sWFsF3Fpt ; ;?B ;V^_d_k_kVl ; ;r&rc r^\rSrSrSrS S\S\S\S\\4U4SjjjrS\ R4U4S jjr S r U=r $) TrOCRScaledWordEmbeddingGzT This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scalec2>[TU]XU5 X@lgN)r r!rG)r"rrrFrGr#s r$r!!TrOCRScaledWordEmbedding.__init__Ls  D&r&r'c<>[TU]U5UR-$rI)r r5rG)r"r'r#s r$r5 TrOCRScaledWordEmbedding.forwardPswy)D,<,<<rfloatr!r/r?r5r@rArBs@r$rDrDGsJ's'3'S'_ghm_n''===r&rDc ^\rSrSrSrSS\S\S\\4U4Sjjjr\SS\S\S\\4Sjj5r \ R"5SS \ RS \4S jj5r SS \ RS\S \\4S jjrS rU=r$)"TrOCRSinusoidalPositionalEmbeddingTzDThis module produces sinusoidal positional embeddings of any length. num_positionsrrFc>[TU]5 SUlX lX0lUR XU5UlURS[R"S55 g)Nr _float_tensorr) r r!rrrF get_embeddingweightsregister_bufferr/ FloatTensor)r"rSrrFr#s r$r!+TrOCRSinusoidalPositionalEmbedding.__init__WsP  *&))- T  _e.?.?.BCr&rcUS-n[R"S5US- - n[R"[R"U[R S9R 5U*-5n[R"U[R S9R 5RS5URS5-n[R"[R"U5[R"U5/SS9RUS5nUS-S:Xa,[R"U[R"US5/SS9nUbSXBSS24'UR[R"55$) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r+rdimr-N)mathlogr/expr0int64rOr4catsincosviewzerostoget_default_dtype)rrrFhalf_dimembs r$rV0TrOCRSinusoidalPositionalEmbedding.get_embedding_s !A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#CQ vve--/00r&r'r(cJUR5up4URXRU5RUR5nURS-U-nUR bX`R RS5:a+UR X`RUR5UlUR RUR5UlUR RSURS55RX4S5R5nU$)Nrrr-) size"create_position_ids_from_input_idsrFrgr,rWrVrrU index_selectredetach)r"r'r(r6r7r)max_posxs r$r5*TrOCRSinusoidalPositionalEmbedding.forwardrs ~~' >>yJZJZ\rsvv    ""Q&0 << 7\\->->q-A#A--g7I7I4K[K[\DL||t'9'9: LL % %a):):2)> ? D DSSU V ] ] _r&cURU5R5n[R"USS9R U5U-U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rr\)ner>r/cumsumtype_asr1)r"r'rFr(maskincremental_indicess r$rnETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_idssW||K(,,.$||Da8@@FI__cgg"'')K77r&)rrrFrWrI)r)r9r:r;r<r=r>rr! staticmethodrVr/no_gradr?r5rnr@rArBs@r$rQrQTsNDcD#DHUXMDD1c1#1HUXM11$ ]]_s&bc 8 847 8QYZ]Q^ 8 8r&rQc^\rSrSrSrSS\S\S\\S\\S\\S\\S \\S \\S \\4U4S jjjr \ "S SSS9SS\ RS\\ RS\\ S\\ RS\\ RS\\S\\ RS\\ R\\ R\\\ R44Sjj5rSrU=r$)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper. embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc 2>[T U]5 X lUbUOUUlUbUOUUlX0lX`lX#-UlURU-UR:Xd[SURSUS35eURS-Ul Xpl Xl [R"URX(S9Ul[R"URX(S9Ul[R"X"US9Ul[R"X"US9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩r)r r!rrrrrhead_dim ValueErrorscalingrrrLineark_projv_projq_projout_proj) r"configrrrrrrrrrr#s r$r!TrOCRAttention.__init__s " ,D)  ,D) " !.  )T^^;MdnnM]^;b" }}d* $"ii 9@ ii 9@ ii 4@  )TB r&past_key_valuepast_key_values4.58new_nameversion hidden_stateskey_value_statesattention_masklayer_head_maskoutput_attentionscache_positionreturnc USLnUR5upn URU5UR-n Sn Ub][U[5(aFUR R UR5n U(a URnOURnOUnU(aUOUnU(aQUbNU (aGWRURRnURURRnOURU5nURU5nURU SUR UR"5R%SS5nURU SUR UR"5R%SS5nUbcU(dUOSnWR'UUURSU05unnU(a.[U[5(aSUR UR'XR -SUR"4nU RXUR UR"5R%SS5n U R("U6n UR("U6nUR("U6nURS5n[*R,"U UR%SS55nUR5XR -U U4:wa.[/SXR -U U4S UR535eUbzUR5U SU U4:wa#[/S U SU U4S UR535eURXR U U5U-nURXR -U U5n[0R2R5USS 9nUbUR5UR 4:wa*[/S UR 4S UR535eURSSSS5URXR U U5-nURXR -U U5nU(a=URXR U U5nURXR -U U5nOSn[0R2R7UUR6UR8S 9n[*R,"UU5nUR5XR -XR"4:wa5[/SXR XR"4S UR535eURXR XR"5nUR%SS5nUR)XU 5nUR;U5nUU4$)z#Input shape: Batch x Time x ChannelNFr-rrrTz$Attention weights should be of size z , but is z!Attention mask should be of size r\z/Head mask for a single layer should be of size ptrainingz `attn_output` should be of size )rmrr isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrrerr transposeupdatereshaper/bmmrr functionalsoftmaxrrr)r"rrrrrrrrr6tgt_lenr query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r$r5TrOCRAttention.forwards .T9"/"4"4"6i{{=1DLL@   &/+>??,77;;DNNK %*9*O*O'*9*N*N'&5#-?)] /"=*,33DNNCHHJ.55dnnELLL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*7It+>+E+E dnn?OQ_>`,( L&*_FY*Z*ZAEO..t~~>NN*B > #((t~~t}}U__`acde #++Z8 ''4 #++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S..'7SVddL',,S>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfmov?wwL',,S>>-A7GTL  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<rrOboolr!rr/r?r tupler5r@rArBs@r$r~r~sH#"#&%*#-2$(!C!C !C sm !C sm !C%!CTN!Ctn!C%TN!CD>!C!CF%0A6R48+/1526,115q2||q2#5<<0q2"% q2 !. q2 "%,,/ q2$D>q2!.q2 u||Xell3XeELL>Q5RR Sq2Sq2r&r~cT^\rSrSrSS\4U4Sjjjr\"SSSS9SS\RS \ \RS \ \RS \ \RS \ \RS \ \RS\ \ S\ \ S\ \ S\ \R4Sjj5r Sr U=r$)TrOCRDecoderLayeri-rc j>[TU]5 URUl[ UURUR UR SUS9UlURUl[URUl URUl [R"UR5UlUR (am[ UURUR UR"UR"UR SSUS9 Ul[R"UR5Ul[R("URUR*5Ul[R("UR*UR5Ul[R"UR5Ulg)NT)rrrrr)rrrrrrrr)r r! hidden_sizerr~decoder_attention_headsattention_dropout self_attnrractivation_function activation_fnactivation_dropoutr LayerNormself_attn_layer_normrcross_attention_hidden_size encoder_attnencoder_attn_layer_normrdecoder_ffn_dimfc1fc2final_layer_norm)r"rrr#s r$r!TrOCRDecoderLayer.__init__.s= ++' nn44,,  ~~ #F$>$>?"(";";$&LL$@!    ... 88777700#'# !D ,.<<+GD (99T^^V-C-CD99V33T^^D " T^^ <r&rrrrrrencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr use_cacherc Un URUUUUUU S9up[RRXRURS9nX-nUR U5nSn Ub`Un UR UUUUUUU S9up[RRXRURS9nX-nURU5nUn URURU55n[RRXRURS9nURU5n[RRXRURS9nX-nURU5nU4nU(aXU 4- nU$)au Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size *(decoder_attention_heads,)*. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrrrN)rrrrrrr) rrrrrrrrrrrrr)r"rrrrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r$r5TrOCRDecoderLayer.forwardRs@!,0>>'+)+/) ,:, (  --m||VZVcVc-d  0 11-@ " ,$H040A0A+!65 : /"3-1B1 -MMM11-< >Gr&) rrrrrrrrrrrrI) NNNNNNFTN)r9r:r;r<rr!rr/r?rr rr5r@rArBs@r$rr-s"={"="=H%0A6R268<9=26=A+/,1$(15Q||Q!.Q ( 5 Q !) 6 Q "%,,/ Q%-U\\$:Q"%Q$D>QD>Q!.QSQr&rc4\rSrSr%\\S'SrSrS/rSr Sr g) TrOCRPreTrainedModelirmodelTrcBURRn[U[R[R 45(aWUR RRSUS9 URb%URRR5 gg[U[R5(adUR RRSUS9 URb2UR RURR5 ggg)Nr)meanstd) rinit_stdrrrConv1dr2datanormal_rzero_ EmbeddingrF)r"modulers r$ _init_weights"TrOCRPreTrainedModel._init_weightsskk"" fryy"))4 5 5 MM   & &CS & 9{{&   &&('  - - MM   & &CS & 9!!- ""6#5#56<<>..r&N) r9r:r;r<r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesrr@rr&r$rrs" &*#,- ?r&rcX^\rSrSrSrS\4U4SjjrSSjrSrU=r $) TrOCRDecoderiz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rc |>[TU]U5 URUlURUlUR UlUR(a [R"UR5OSn[URURUR US9Ul UR(a&[UR UR5UlO@[%UR UR -S-URUR 5UlUR&(a&[(R*"UR5UlOSUl[(R,"[/UR05Vs/sH n[3XS9PM sn5UlSUlUR95 gs snf)NrNrMr)rF)r r!rdecoder_layerdrop layerdrop pad_token_idrFscale_embeddingr^sqrtrrD vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrQlayernorm_embeddingrr ModuleListrangedecoder_layersrrgradient_checkpointing post_init)r"rrGir#s r$r!TrOCRDecoder.__init__sT  ~~ 11!..7=7M7Mdii 2 23SV 4   v1143C3CQ\   1 1#B6CaCacicucu#vD #E..1A1AAAE""  $D  % %')||F4F4F'GD $'+D $mmUZ[a[p[pUq$rUqPQ%6v%KUq$rs &+#  %ssF9c:U bU OURRn U bU OURRn U bU OURRn U bU OURRn UbUb [ S5eUb"UnUR SURS5nO.Ub UR5SSnUSS2SS2S4nO [ S5eUR(a/UR(aU (a[RS5 Sn U (aGUcDUb.[[URS9[URS95O[URS9nU (a@[U[ 5(a+[RS5 [R""U5nUbUR%5OS nUcUR'U5nURR((aUR+UUS 9nOUR+UUS 9nUU-nUR,bUR-U5n[.R0R3UUR2URS 9nURn[5X/UU5nUbUb[7XHR8USS 9nU (aS OSnU (aS OSnU (aUbS OSn[;XV/SS/5HnunnUcM UR5S [=UR>5:wdM7[ SUS[=UR>5SUR5S S35e [AUR>5HunnU (aUU4- nUR(a([BRD"/5nUURF:aMLU"UUUUUbUUOSUbUUOSUU U U S9 nUS nU (dMyUUS4- nUcMUUS4- nM U (aUU4- nU (d[!SUUUUU455$[IUUUUUS9$)a Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer-zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)r(r)rr head_maskcross_attn_head_maskzThe `z` should be specified for z layers, but it is for .)rrrrrrrrrc30# UH nUcMUv M g7frIr).0vs r$ 'TrOCRDecoder.forward..srArs )last_hidden_staterr attentionscross_attentions)%rroutput_hidden_statesruse_return_dictrrer.rmrrlogger warning_oncer r rrfrom_legacy_cacheget_seq_lengthrrrrrrrrr r+ziplenr enumerater/randrr)r"r'rrrr r r inputs_embedsrrr return_dictrinput input_shaper( embed_posrall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_nameidx decoder_layerdropout_probability layer_outputss r$r5TrOCRDecoder.forwardsL^2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E!r5;;r?;I  &',,.s3K!!Q(+Ede e  & &4==##t" 0)4$L $DlZ^ZeZeFfg!5  OU;;   \  2CCOTOETE`!?!?!Afg   --i8M ;; 6 6,,UKa,bI,,YOe,fI% 1  # # / 44]CM --mt||VZVcVc-d kk : 8N  ! ,1G1S%?&(;(;[QS_& " #7BD0d&7>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03%r#,DKK"8 C#!m%55!}}&+jjn#&7)%'=3<3H3dI]Ii,@,Eos /"3#- M*!,M  =#3"55(4(]1-=,??(7#9< -!1 1 ':K^]qr  9+++%1   r&)rrrrrrrrF) NNNNNNNNNNNNN) r9r:r;r<r=rr!r5r@rArBs@r$rrsJ{B"#!!O O r&ra The TrOCR Model with a language modeling head. Can be used for summarization. This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. ) custom_introc.^\rSrSrU4SjrSrSrU=r$)TrOCRDecoderWrappericD>[TU]U5 [U5UlgrI)r r!rdecoderr"rr#s r$r!TrOCRDecoderWrapper.__init__s  #F+ r&c&UR"U0UD6$rIr4)r"argskwargss r$r5TrOCRDecoderWrapper.forwards||T,V,,r&r8)r9r:r;r<r!r5r@rArBs@r$r2r2s,--r&r2zy The TrOCR Decoder with a language modeling head. Can be used as the decoder part of [`EncoderDecoderModel`] and c"^\rSrSrS/rU4SjrSrSrSrSr Sr S r \ SS \ \RS \ \R S \ \R"S \ \RS\ \R S\ \R S\ \S\ \R"S\ \RS\ \S\ \S\ \S\ \S\ \R S\\\44Sjj5rSrU=r$)TrOCRForCausalLMizoutput_projection.weightc>SUlSUl[TU] U5 [ U5Ul[ R"URURSS9Ul UR5 g)NTFr) ris_encoder_decoderr r!r2rrrrroutput_projectionrr5s r$r!TrOCRForCausalLM.__init__sZ $)!  (0 !#6+=+=v?P?PW\!] r&cBURRR$rIrr4rr"s r$get_input_embeddings%TrOCRForCausalLM.get_input_embeddingsszz!!...r&c8XRRlgrIrC)r"values r$set_input_embeddings%TrOCRForCausalLM.set_input_embeddingss*/ 'r&cUR$rIr@rDs r$get_output_embeddings&TrOCRForCausalLM.get_output_embeddingss%%%r&cXlgrIrL)r"new_embeddingss r$set_output_embeddings&TrOCRForCausalLM.set_output_embeddingss!/r&c$XRlgrIrr4)r"r4s r$ set_decoderTrOCRForCausalLM.set_decoders $ r&c.URR$rIrTrDs r$ get_decoderTrOCRForCausalLM.get_decoderszz!!!r&r'rrrr r rr!labelsrrrr"rrcNU bU OURRn U bU OURRn U bU OURRn URR UUUUUUUUU U U U US9 nUR US5nSnU bF[5nU"URSURR5U RS55nU (dU4USS-nUbU4U-$U$[UUURURURURS9$)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import requests >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r'rrrr r rr!rrrr"rrr-r)losslogitsrrrr)rrrrrr4r@rrerrrrrr)r"r'rrrr r rr!rZrrrr"rrr]r\loss_fctoutputs r$r5TrOCRForCausalLM.forwardsKZ2C1N-TXT_T_TqTq$8$D $++JjJj &1%u $D>u 'tnu d^u !.u u77 8!u u r&r=)1r=r^typingrrr/rtorch.nnr activationsr cache_utilsr r r generationr modeling_attn_mask_utilsr rmodeling_layersrmodeling_outputsrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_trocrr get_loggerr9rrrrDModulerQr~rrrr2r=__all__rr&r$rrs.6 " %!CC):l-,0,   H %;bll;8 =r|| =;8;8|X2RYYX2vw2wt????$v 'v r-.-- V +_V  V r 5 6r&