cCiDSrSSKrSSKJr SSKJrJrJr SSKrSSKJ r SSK J r SSK J r Jr SS KJr SS KJr SS KJr SS KJrJrJrJr SS KJrJr SSKJrJrJ r SSK!J"r"J#r#J$r$J%r%J&r& SSK'J(r( SSK)J*r*J+r+ \%RX"\-5r.\\#"SS9"SS\"555r/"SS\ R`5r1"SS\ R`5r2"SS\ R`5r3S\20r4"SS\ R`5r5"S S!\ R`5r6"S"S#\ R`5r7"S$S%\5r8"S&S'\ R`5r9\#"S(S)\55r:"S*S+\ R`5r;"S,S-\ R`5r<SJS.\ R`S/\RzS0\RzS1\RzS2\\RzS3\>S4\>4S5jjr?"S6S7\ R`5r@"S8S9\5rA"S:S;\ R`5rB"S<S=\ R`5rC\#"S>S9"S?S@\:55rD"SASB\ R`5rE\#"SCS9"SDSE\:55rF\#"SFS9"SGSH\:\55rG/SIQrHg)KzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \RS4\ S'Sr \\ \RS4\ S'S rg) GitVisionModelOutput4z image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__r%rtorch FloatTensor__annotations__r&r'tupler(__static_attributes__r)^/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/git/modeling_git.pyr#r#4sr 15L(5,,-459x 1 129=AM8E%"3"3S"89:A:>Ju00#567>r4r#c ^\rSrSrSrU4SjrS S\\RS\\RS\\RS\ S\R4 S jjr S r U=r$) GitEmbeddingsGz;Construct the embeddings from word and position embeddings.c:>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5Ul[USS5UlUR#S[$R&"UR5R)S5SS9 g) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr=register_bufferr/arangeexpandselfconfig __class__s r5rEGitEmbeddings.__init__Js !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej  r4 input_idsr? inputs_embedspast_key_values_lengthreturncNUbUR5nOUR5SSnUSnUcURSS2XFU-24nUcURU5nOUnURS:XaUR U5nXx- nUR U5nUR U5nU$)NrArr>)sizer?rJr=rLrMrQ) rWr[r?r\r] input_shape seq_length embeddingsrLs r5forwardGitEmbeddings.forwardYs  #..*K',,.s3K ^  ,,Q0FVlIl0l-lmL  --i8J&J  ' ': 5"&":":<"H   -J^^J/ \\*- r4)rMrQr=rLrJ)NNNr)r*r+r,r-r.rErr/ LongTensorr0intTensorrdr3 __classcell__rYs@r5r7r7GsxE  "153759&' E,,-u//0  1 12  !$   r4r7c^\rSrSrSU4Sjjr\"SSSS9SS\RS\\RS \\RS\\ S \\ S \\ S \ \R4S jj5r SrU=r$)GitSelfAttentionwc,>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eX0lUc-[RSURRS35 URUl[URUR- 5Ul URUR-Ul [URRURR - S-S -5UlUR$bU=R"UR$-sl[&R("URUR5Ul[&R("URUR5Ul[&R("URUR5Ul[&R0"UR25UlU=(d [7US S 5UlUR8S :XdUR8S :XaHUR:Ul[&R<"SUR:-S - UR5Ulgg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr=r> relative_keyrelative_key_query) rDrErHnum_attention_headshasattr ValueError layer_idxlogger warning_oncerYr*rgattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerOattention_probs_dropout_probrQrRr=rKrFdistance_embeddingrWrXr=rwrYs r5rEGitSelfAttention.__init__xs)     : : :a ?PVXhHiHi#F$6$6#78 445Q8 #      !8!8 9:,,  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD #>rr4past_key_valuepast_key_values4.58new_nameversionr'attention_mask head_maskoutput_attentionspixel_values_presentr^c dURupxn URU5RUSURUR5R SS5n U(a UR OSn URU5RUSURUR5R SS5n URU5RUSURUR5R SS5n UbURU SS2SS2U S2SS24U SS2SS2U S2SS24UR5up[R"U SS2SS2SU 2SS24U/SS9n [R"U SS2SS2SU 2SS24U/SS9n [R"XR SS55nURS:XdURS:XGaU RSU RSnnUbB[R"US- [R UR"S 9RSS5nO>[R$"U[R UR"S 9RSS5n[R$"U[R UR"S 9RSS5nUU- nUR'UUR(-S- 5nUR+U R,S 9nURS:Xa[R."S U U5nUU-nOHURS:Xa8[R."S U U5n[R."S U U5nUU-U-nU[0R2"UR5- nUbUU-n[4R6R9USS9nUR;U5nUbUU-n[R"UU 5nUR=SSSS 5R?5nURA5SSURB4-nURU5nUU4$)NrArrqrdimrrrsdtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shaperviewrtrz transposerrrupdaterwr/catmatmulr=tensorlongrrTrrKtoreinsummathsqrtr functionalsoftmaxrQpermute contiguousr`r{)rWr'rrrrr batch_sizerb_ query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_pastattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r5rdGitSelfAttention.forwards%2$7$7!  JJ} % T*b$":":D/E/E!Q*+[Avw9I-JDNN0 ,N 9Q7F7A-=#>"OUVWI))[Aww1A%BDT$U[\]K!<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L*!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--r4) r{rzrrQrrrwrKrtr=rrNNNNNFF)r*r+r,r-rErr/rhrr0r boolr2rdr3rirjs@r5rlrlws uD%0A6R7;15+/,1/4R.||R.!!2!23R.E--. R. "% R. $D> R.'tnR. u|| R.SR.r4rlcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) GitSelfOutputc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gNr;) rDrErrrHdenserMrNrOrPrQrVs r5rEGitSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r4r' input_tensorr^cpURU5nURU5nURX-5nU$NrrQrMrWr'rs r5rdGitSelfOutput.forward5 =1  ]3 }'CD r4rMrrQ r*r+r,r-rEr/rhrdr3rirjs@r5rr6> U\\RWR^R^r4reagerc^\rSrSrSU4SjjrSr\"SSSS9SS\RS \ \RS \ \RS\ \ S \ \ S \ \ S \ \R4Sjj5rSrU=r$) GitAttentionic>[TU]5 [UR"XUS9Ul[ U5Ul[5Ulg)N)r=rw) rDrEGIT_SELF_ATTENTION_CLASSES_attn_implementationrWroutputset pruned_headsrs r5rEGitAttention.__init__sB .v/J/JK y  $F+ Er4c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)lenrrWrtrzrrrrrrrr{union)rWheadsindexs r5 prune_headsGitAttention.prune_headss  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r4rrrrr'rrrrr^cZURUUUUUU5upxURXq5n X4$r)rWr) rWr'rrrrr attn_outputself_attn_weightsattention_outputs r5rdGitAttention.forward sA*.      * &  ;;{B22r4)rrrWrr)r*r+r,r-rErrr/rhrr0r rr2rdr3rirjs@r5rrs";$%0A6R7;15+/,1/43||3!!2!233E--. 3 "% 3 $D> 3'tn3 u|| 3S3r4rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)GitIntermediatei7c>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rDrErrrHintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrVs r5rEGitIntermediate.__init__8s` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r4r'r^cJURU5nURU5nU$rrrrWr's r5rdGitIntermediate.forward@s& =1 00? r4rrrjs@r5rr7s(9U\\ellr4rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $) GitOutputiGc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rDrErrrrHrrMrNrOrPrQrVs r5rEGitOutput.__init__Hs` YYv779K9KL f&8&8f>S>STzz&"<"<= r4r'rr^cpURU5nURU5nURX-5nU$rrrs r5rdGitOutput.forwardNrr4rrrjs@r5rrGrr4rc^\rSrSrSU4Sjjr\"SSSS9SS\RS\\RS \\RS\\ S \\ S \\ S \ \R4S jj5r SrSrU=r$)GitLayeriUc>[TU]5 URUlSUl[ XS9Ul[ U5Ul[U5Ul g)Nr)rw) rDrEchunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rWrXrwrYs r5rEGitLayer.__init__VsI '-'E'E$%fB+F3' r4rrrrr'rrrrr^c URUUUUUUS9upx[URURURU5n X4$)N)rrr)rrfeed_forward_chunkrr) rWr'rrrrrrself_attention_weights layer_outputs r5rdGitLayer.forward^sa48>>   /+!5 4B4 01  # #T%A%A4CSCSUe 33r4cJURU5nURX!5nU$r)rr)rWrintermediate_outputr s r5rGitLayer.feed_forward_chunkws)"//0@A{{#6I r4)rrrrrrr)r*r+r,r-rErr/rhrr0r rr2rdrr3rirjs@r5rrUs(%0A6R7;15+/,1/44||4!!2!234E--. 4 "% 4 $D> 4'tn4 u|| 4S40r4rc(^\rSrSrU4SjrSS\R S\\RS\\RS\\ \ \ \ \R4S\\ S\\ S \\ S \\ S \\ S \ \ \R \ 44S jjrSrU=r$) GitEncoderi}c >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5UlSUl gs snfNF) rDrErXr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)rWrXirYs r5rEGitEncoder.__init__~sR  ]]vG_G_A`#aA`AHV$7A`#ab &+#$bA&r'rrr use_cacheroutput_hidden_statesr return_dictr^c UR(a/UR(aU(a[RS5 SnU(aUc[ UR S9nU(aSOSn U(aSOSn [ UR5H=upU(aX4-n UbX<OSnU "UUUUUU5nUSnU(dM5XS4-n M? U(aX4-n U (d[SUUU U 455$[UUU U S9$) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rXr)rrc30# UH nUcMUv M g7frr)).0vs r5 %GitEncoder.forward..s" A s r&rr'r() rtrainingrxryr rX enumeraterr2r)rWr'rrrrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r5rdGitEncoder.forwards"  & &4==##p" 0*$++>O"6BD$5b4(4OA#$58H$H!.7.CilO(!$ M*!,M  &91=M ||> !!2!23> E--. > "%uU5;L;L5M/N(N"OP > D> > $D>> 'tn> 'tn> d^>  uU\\"$;; <> > r4rc.\rSrSr%\\S'SrSrSrSr g)GitPreTrainedModelirXgitTc[U[5(a[RR UR SUR RS9 [RR URRUR RS9 [RR URRUR RS9 [U[R5(akURRR SUR RS9 URb%URRR5 gg[U[R5(axURRR SUR RS9 UR b2URRUR R5 gg[U[R"5(aJURRR5 URRR%S5 gg)zInitialize the weights)meanstd)r2Ng?)rGitVisionEmbeddingsrinitnormal_class_embeddingrXinitializer_rangepatch_embeddingweightposition_embeddingrdatabiaszero_rFr:rMfill_)rWmodules r5 _init_weights GitPreTrainedModel._init_weightss f1 2 2 GGOOF22$++B_B_O ` GGOOF2299t{{?\?\O ] GGOOF55<<$++B_B_O ` fbii ( ( MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S ).r4r)N) r*r+r,r-rr1base_model_prefixsupports_gradient_checkpointingr@r3r)r4r5r-r-s &*#*r4r-c^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S S \RS\R4S jjr S r U=r $) r3irXc>[TU]5 XlURUlUR UlUR Ul[R"[R"UR55Ul [R"URURUR UR SS9UlUR UR -S-UlURS-Ul[R""UR UR5UlUR'S[R("UR 5R+S5SS9 g)NF) in_channels out_channels kernel_sizestrider<rqrr?r@rB)rDrErXrH embed_dimr}r~r Parameterr/randnr6Conv2d num_channelsr8 num_patches num_positionsrFr:rSrTrUrVs r5rEGitVisionEmbeddings.__init__s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr4rcheightwidthr^cURSS- nURRRS5nURSS- n[R R 5(d%XF:Xa X#:XaURUR5$USS2SS24nUSS2SS24nURSn X R-n X0R-n [US-5n URSXU 5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU 5n[R "Xx4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrAg?rrqbicubicF)r`mode align_cornersr)rr:r9 unsqueezer/jit is_tracingr?r~rreshaperrr interpolaterr) rWrcrRrSrOr:rPclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r5interpolate_pos_encoding,GitVisionEmbeddings.interpolate_pos_encodingsi!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,QU3,QU3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr4 pixel_valuesc ^URup4pVU(dJXPR:wdX`R:wa,[SUSUSURSURS3 5eURRR nURUR US95nURS5RSS5nURRUSS5n [R"X/SS 9n U(aXRXU5-n U $XRUR5-n U $) NzInput image size (*z) doesn't match model ().rrqrrAr)rr}rvr8r9rrflattenrr6rUr/rrbr:r?) rWrdrbrrrRrS target_dtype patch_embeds class_embedsrcs r5rdGitVisionEmbeddings.forward!s$'3'9'9$ v'V-F%SbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C ##&C&CJX]&^^J$&=&=d>O>O&PPJr4) r6rXrJr}rOrPr8r~r:F)r*r+r,r-r rEr/rhrgrbr0rdr3rirjs@r5r3r3siqq,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr4r3cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $) GitVisionMLPi4c>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) rDrErXr r activation_fnrrrHrfc1fc2rVs r5rEGitVisionMLP.__init__5sb  #F$5$5699V//1I1IJ99V55v7I7IJr4r'r^clURU5nURU5nURU5nU$r)rrrqrsrs r5rdGitVisionMLP.forward<s4/ **=9 / r4)rqrXrrrsrrjs@r5roro4s)KU\\ellr4ror?rrrrscalingrQc [R"XRSS55U-nUbX-n[RR US[R S9RUR5n[RRXURS9n[R"X5n U RSS5R5n X4$)NrAr)rr)pr$rrq) r/rrrrrfloat32rrrQr$r) r?rrrrrwrQkwargs attn_weightsrs r5eager_attention_forwardr}Ds<<}}R'<=GL!#4 ==((2U]](SVVW\WbWbcL==((6??([L,,|3K''1-88:K  $$r4c^\rSrSrSrU4SjrS S\RS\\RS\\RS\\ S\ \R\\R44 S jjr S r U=r $) GitVisionAttentioni[z=Multi-headed attention from 'Attention Is All You Need' paperc >[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul URUl SUl [R"URUR5Ul[R"URUR5Ul[R"URUR5Ul[R"URUR5Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rggF)rDrErXrHrJrt num_headshead_dimrvscaleattention_dropoutrQ is_causalrrk_projv_projq_projout_projrVs r5rEGitVisionAttention.__init__^s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r4r'rcausal_attention_maskrr^c URupVnURU5nURU5n URU5n UR XVUR UR 5RSS5nU R XVUR UR 5RSS5n U R XVUR UR 5RSS5n URRS:waUbUbX#-nOUbUnO USLUl [n URRS:wa[URRn U "UUU U UURURUR(dSO URS9upU R!XVU5R#5n UR%U 5n U(dSn X4$)z#Input shape: Batch x Time x Channelrrqflash_attention_2Nrr0)rrwrQ)rrrrrrrrrXrrr}rrr$rQr[rr)rWr'rrrrrbrJquerieskeysvaluesattention_interfacerr|s r5rdGitVisionAttention.forwardrs-:,?,?) ++m,{{=)]+,,zt~~t}}U__`acdeyyOYYZ[]^_ZT^^T]]S]]^_abc ;; + +/B B).C.O!/!G&2!62$>DN(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! "))*)LWWY mmK0  L((r4) rXrQrJrrrrrrrr)NNF)r*r+r,r-r.rEr/rhrrr2rdr3rirjs@r5rr[sGB.268<,1 /)||/)!./) ( 5 /) $D> /) u||Xell33 4 /)/)r4rc ^\rSrSrS\4U4SjjrS S\RS\RS\RS\\ S\ \R4 S jjr S r U=r$) GitVisionEncoderLayerirXc<>[TU]5 URUl[ U5Ul[ R"URURS9Ul [U5Ul [ R"URURS9Ul gr) rDrErHrJr self_attnrrMrN layer_norm1romlp layer_norm2rVs r5rEGitVisionEncoderLayer.__init__sm +++F3<<F & u  ! &&r4rc^\rSrSrSrS\4U4Sjjr\S S\\ RS\\ RS\\ S\\ S \\ S \ \ \44 S jj5rS rU=r$)GitVisionEncoderiz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rXc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfr) rDrErXrrrrrlayersr)rWrXrrYs r5rEGitVisionEncoder.__init__sT  mmERXRjRjLk$lLkq%:6%BLk$lm &+#%mrrrrrrr^cUbUOURRnUbUOURRnUbUOURRnU(aSOSnU(aSOSnUn [ UR 5H0upU(aXy4-nU "U UUUS9n U Sn U(dM(XS4-nM2 U(aXy4-n[ XUS9$)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr))rrrr&r'r()rXrruse_return_dictr%rr) rWr\rrrrrencoder_statesall_attentionsr'idx encoder_layerr*s r5rdGitVisionEncoder.forwardsN2C1N-TXT_T_TqTq$8$D $++JjJj &1%>N+Vd  r4)rXrr)NNNNN)r*r+r,r-r.r rErrr/rhrrr2rrdr3rirjs@r5rrs,, 268<,0/3&*D !.D  ( 5 D $D> D 'tn D d^D  uo% &D D r4rc^\rSrSrS\4U4Sjjr\S S\\RS\\ S\\ S\\ S\\ S \ \ \ 44 S jj5rS rU=r$) GitVisionTransformeri/rXc>[TU]5 XlURn[ U5Ul[ R"X!RS9Ul [U5Ul [ R"X!RS9Ul gr) rDrErXrHr3rcrrMrN pre_layrnormrencoderpost_layernorm)rWrXrJrYs r5rEGitVisionTransformer.__init__1sd  && -f5LL8M8MN'/  ll9:O:OPr4rdrrrbrr^cUbUOURRnUbUOURRnUbUOURRnUc [ S5eUR XS9nUR U5nURUUUUS9nUSnURU5nU(d U4USS-$[UURURS9$)Nz You have to specify pixel_valuesrb)r\rrrrrr) rXrrrrvrcrrrrr'r() rWrdrrrbrr'encoder_outputsr&s r5rdGitVisionTransformer.forward;s2C1N-TXT_T_TqTq$8$D $++JjJj &1%& 'tn & #+4. & d^ &  uo% && & r4rzY The vision model from CLIP, used in GIT, without any head or projection on top. c^\rSrSr%\\S'SrS\4U4SjjrS\R4Sjr \ S S\ \ RS\ \S\ \S \S \ \S\\\44 S jj5rS rU=r$)GitVisionModelierXrdcd>[TU]U5 [U5UlUR 5 gr)rDrEr vision_model post_initrVs r5rEGitVisionModel.__init__os'  08 r4r^cBURRR$r)rrcr8rWs r5get_input_embeddings#GitVisionModel.get_input_embeddingsus  ++;;;r4rrrbrc^UbUOURRnURUUUUUS9$)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```)rdrrrbr)rXrr)rWrdrrrbrs r5rdGitVisionModel.forwardxsA8&1%# 'tn # #' # d^ #  uo% &# # r4rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) GitProjectionirXc.>[TU]5 Xl[R"[R "UR RUR5[R"URUR RS95Ul gr) rDrErXr Sequentialrr|rHrMrNvisual_projectionrVs r5rEGitProjection.__init__sd  !# IIf**668J8J K LL++1E1E1T1T U" r4rcr^c$URU5$r)r)rWrcs r5rdGitProjection.forwards%%j11r4)rXr) r*r+r,r-rrEr/rhrdr3rirjs@r5rrs/ y 2%,,25<<22r4rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states c^\rSrSrU4SjrSrSrSrS\S\ RS\ RS \ R4S jr SS jr\SS \\ RS \\ RS\\ RS\\ RS\\ RS\\ RS\\\\\ R(4S\\S\\S\\S\S\\S \\\ R\44Sjj5rSrU=r$)GitModelicr>^[TU]T5 TUl[T5Ul[ TR 5Ul[T5Ul [T5Ul TRb8[R"U4Sj[TR555UlUR#5 g)Nc3># UHEn[R"[R"SSTRR 55v MG g7f)rN)rrKr/zerosr|rH)rrrXs r5r!$GitModel.__init__..s=;?A U[[Av/C/C/O/OPQQ?sA A)rDrErXr7rcrr| image_encoderrrrrrr ParameterListrimg_temporal_embeddingrrVs `r5rEGitModel.__init__s   '/+F,@,@A!&) !.v!6  * * 6*,*:*:;v>>?;+D ' r4c.URR$rrcrJrs r5rGitModel.get_input_embeddingss...r4c$XRlgrr)rWrs r5set_input_embeddingsGitModel.set_input_embeddingss */'r4cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rWheads_to_prunerrs r5 _prune_headsGitModel._prune_headss< +002LE LL  u % / / ; ;E B3r4r`rrr^c [R"[R"XX2S9SS9nURUS:H[ S55nU$)Nrrr)diagonal-inf)r/triuones masked_fillfloat)rWr`rrmasks r5_generate_future_maskGitModel._generate_future_masks=zz%**TLWXY 5=9 r4cnURSnURSnURnURn [R"Xw4XS9n [R "XvU-4[ S5URU S9n [R"Xg4U URS9n US:a?[R"URSURSU-4U URS9n[R"X4SS9n [R"XRU 54SS9n[R"X4SS9SSS24nUc2[R "URSURS4SUS9nUR[R:wa [S 5e[R"XQRS 9n[ S5UU'URURSXv-Xt-U-45nUR5nUSS2SS2SU24nUSS2SSS24nUU-USS2SS2SU24'USS2SSS2SS24nU$) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr/rfullrrrrrv zeros_likerUclone)rWtgtmemorytgt_maskr]memory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r5create_attention_maskGitModel.create_attention_masks0))A,\\!_  ;; 7TJJ #99 : &M::  kk  !?? "A %{{"HNN1$58N$NOH yy(0a8 9kk%&89qA#ii 1=dAgF " *&+jj&,,q/6<>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerAz5You have to specify either input_ids or inputs_embedsrrrrz#pixel_values must be of rank 4 or 5)r[r?r\r]rqr)rrrr])tgt_len)rrrrrrrrr#)%rXrrrrrv%warn_if_padding_and_no_attention_maskr`rr get_seq_length get_head_maskrndimrr&rrrappendr/rrrcrrrrepeatrrr rrrrr'r()rWr[rr?rdrr\rrrrrbrrarbr]projected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr'rcombined_attention_maskexpanded_attn_maskrsequence_outputs r5rdGitModel.forward s,J2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y Q#..*K  &',,.s3KTU U ^ "#  &"/599 ..0$335 #&&y++2O2OP $(!  #  A%"&"4"4 #5### ""a'"$!&|'9'9!',>$Q 1a%78Sk-?-''**T-H-H-SS)#**+@A ">#())O"C!!FGG(,(>(>(O %??%'#9 +  % ,(- !''*A/?/E/Ea/HI&,,'..) %%>$D$D  ! !! $(A(F(Fq(I I1a% !  #<>N"OUVW --j:J:P:PRbRiRij#'"<"< ,#9 #=#   %"< 0 6 6 B"b!(() &)%71?U>U>VXY8Y%Z"'1{1~o.?+a.AR(RSWiiS,, 2+/!5#!-T!9'  *!,#%(;; ;&-+;;)77&11   r4)rXrcrrrrr) NNNNNNNNNNFN)r*r+r,r-rErrrrgr/rrrhrrrrrr listr0rr2rrdr3rirjs@r5rrs &/0C#ekk5<<\a\h\h 0#d-115/3/3,004KO$(,0/3).&*c ELL)c !.c u||, c u||, c ELL) c   -c "%tE4E4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\\"$>> ?c c r4rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c ^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ \ \\ R4S\ \S\ \S\ \S\S\ \S\ \\ R\44Sjj5rSSjrSrU=r$)GitForCausalLMiz output.weightc>[TU]U5 [U5Ul[R "UR UR5UlUR5 gr) rDrErr.rrrHrGrrrVs r5rEGitForCausalLM.__init__sF  F#ii 2 2F4E4EF  r4cUR$rrrs r5get_output_embeddings$GitForCausalLM.get_output_embeddingss {{r4cXlgrr)rWnew_embeddingss r5set_output_embeddings$GitForCausalLM.set_output_embeddingss$ r4r[rr?rdrr\labelsrrrrrbrr^c U bU OURRn UbSn URUUUUUUUU U U U U S9 nUSnURU5nSnUbURRR SR RRnUSS2US2SS24R5nUSS2SS24R5nUR"URSURR5URS54SURR0UD6nU (dU4USS-nUbU4U-$U$[UUURURUR S9$) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr?rdrr\rrrrrbrrrArrG)losslogitsrr'r()rXrr.rrrrrWrr loss_functionrrGrrr'r()rWr[rr?rdrr\r#rrrrrbrr{rrr&r%num_image_tokensshifted_logitsrs r5rdGitForCausalLM.forwardsj&1%IIKNAqrE]--/F%%##B (>(>? B ;;11 DY,F)-)9TGf$ Ev E%#33!//))   r4c HUbCUR5nURSU:aUnOURSS- nUSS2US24nURnUcURU5nUUURS5UUS.n UR 5HupX;dM XU 'M U $)Nrrd)r[rrdrr)rrnew_onesgetr) rWr[rrrr{ past_lengthremove_prefix_lengthra model_inputsrrs r5prepare_inputs_for_generation,GitForCausalLM.prepare_inputs_for_generations  &)88:Kq!K/'2$(1q'9A'=$!!%9%:":;I oo  !&// A $D>A 'tnA #'A d^A uU\\"$:: ;!A A HOS$$r4r)rrr-r)r0)Ir.r dataclassesrtypingrrrr/r activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrrutils.deprecationrconfiguration_gitrr get_loggerr*rxr#rr7rlrrrrrrrr-r3rorhrr}rrrrrrrr__all__r)r4r5rCs  !,, !.)B9 Gll19   H %  ?; ?   ?-BII-`v.ryyv.t BII   /3299/3f bii   %)%PE E P***6P"))Pf 299 .% II% <<% % << % U\\* %  %%.F)F)T/6/fT ryyT n3 2993 l 2 '2  2 j 2BII 2 ~ !~  ~ B z'z  zz Qr4