cCiFSrSSKrSSKJr SSKJrJrJrJr SSK r SSK J r SSK J r SSK Jr SS KJr SS KJr SS KJrJrJrJr SS KJrJr SS KJr SSKJrJrJ r SSK!J"r"J#r#J$r$J%r%J&r&J'r' SSK(J)r)J*r* SSK+J,r,J-r-J.r. SSK/J0r0J1r1J2r2 \&Rf"\45r5\\$"SS9"SS\"555r6"SS\ Rn5r8SHS\ RnS\ RrS\ RrS\ RrS\\ RrS \:S!\:4S"jjr;"S#S$\ Rn5r<"S%S&\ Rn5r="S'S(\5r>\$"S)S*\55r?"S+S,\ Rn5r@"S-S.\?5rA"S/S0\ Rn5rB"S1S2\ Rn5rC"S3S4\ Rn5rD"S5S6\ Rn5rE"S7S8\ Rn5rF"S9S:\5rG"S;S<\ Rn5rH"S=S>\ Rn5rI"S?S@\?5rJ\$"SAS9"SBSC\?55rK\$"SDS9"SESF\?\55rL/SGQrMg)IzPyTorch InstructBLIP model.N) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipConfigInstructBlipQFormerConfigInstructBlipVisionConfigzQ Class defining the outputs of [`InstructBlipForConditionalGeneration`]. ) custom_introc\rSrSr%SrSr\\\R\ S'Sr \\\R\ S'Sr \\R\ S'Sr \\\R\ S'Sr\\\R\ S'S \\4S jrS rg) /InstructBlipForConditionalGenerationModelOutput.a~ loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsreturncJ^[U4SjTR555$)Nc3n># UH*nUS;aTUO[TU5R5v M, g7f)r.r/r0N)getattrto_tuple).0kselfs p/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/instructblip/modeling_instructblip.py KInstructBlipForConditionalGenerationModelOutput.to_tuple..JsC !WW Gq!**, -!s25)tuplekeysr9s`r:r68InstructBlipForConditionalGenerationModelOutput.to_tupleIs% YY[    )__name__ __module__ __qualname____firstlineno____doc__r,rr=torch FloatTensor__annotations__r-r.r/r0rr6__static_attributes__rBrAr:r*r*.s 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* rAr*c^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S S \RS \ S\R4S jjr S r U=r$)InstructBlipVisionEmbeddingsSconfigcr>[TU]5 XlURUlUR UlUR Ul[R"[R"SSUR55Ul [R"SURUR UR S9Ul UR UR -S-UlURS-Ul[R"[R"SURUR55Ulg)Nr$r ) in_channels out_channels kernel_sizestrider )super__init__rO hidden_size embed_dim image_size patch_sizer ParameterrHrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingr9rO __class__s r:rV%InstructBlipVisionEmbeddings.__init__Ts  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"brA embeddingsheightwidthr1cURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rr r bicubicF)sizemode align_cornersdim)shaperbrHjit is_tracingrZrreshapepermuter functional interpolateviewcat) r9rfrgrhr`raclass_pos_embedpatch_pos_embedrp new_height new_widthsqrt_num_positionss r:interpolate_pos_encoding5InstructBlipVisionEmbeddings.interpolate_pos_encodingfsS!&&q)A- //55a81< yy##%%+*F6?** *11!RaR%811!QR%8r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCrA pixel_valuesrcURup4pVURRRnURUR US95nUR S5R SS5nURRUSS5R U5n [R"X/SS9n U(aURXU5n O URn XSS2SU RS52SS24R U5-n U $)Ndtyper r$rjro)rqr_weightrtoflatten transposer]expandrHryrrbrl) r9rr batch_size_rgrh target_dtype patch_embeds class_embedsrfrbs r:forward$InstructBlipVisionEmbeddings.forwards'3'9'9$ v++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY ;C #!%!>!>zSX!Y !%!8!8 Q8L*//!:L8La5O"P"S"ST`"aa rA) r]rOrXrYr`rar_rZrbF)rCrDrErFr'rVrHTensorintrrIboolrrK __classcell__rds@r:rMrMSsrc7c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn  rArMmodulequerykeyvalueattention_maskscalingdropoutc `[R"XRSS55U-nUbX-n[RR USS9n[RR XURS9n[R"X5n U RSS5R5n X4$)Nrjro)ptrainingr$r ) rHmatmulrrrvsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r:eager_attention_forwardrs<<}}R'<=GL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$rAc ^\rSrSrSrU4SjrS\RS\S\4Sjr S S\RS \ \RS \ \R\ \R\ \ \R44S jjr S r U=r$)InstructBlipAttentionz=Multi-headed attention from 'Attention Is All You Need' paperc>[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul SUl URUl [R"URSUR-SS9UlUR(ai[R "["R$"UR55n[R "["R$"UR55nOSnSnUbQ["R&"U["R("USS9U45n[R "U5URl[R"URUR5Ulg) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )bias) requires_grad)rUrVrOrWrXnum_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrLinearqkvqkv_biasr[rHzerosry zeros_liker projection)r9rOq_biasv_biasrrds r:rVInstructBlipAttention.__init__ss  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCrAtensorseq_lenbszcURX2URUR5RSS5R 5$)Nr$r )rxrrrr)r9rrrs r:_shapeInstructBlipAttention._shapes5{{3GQQRSUVWbbddrA hidden_states head_maskr1c 4UR5upEnURU5nURXESURX`R-5R SSSSS5nUSUSUSpn[ n UR RS:wa[UR Rn U "UUU U 4SUR(dSO URURS .UD6upU RXES 5R5n URU 5n X4$) z#Input shape: Batch x Time x Channelr r rr$eagerN)rrrrj)rlrrtrrurrO_attn_implementationrrrrrr)r9rrrrtgt_lenrX mixed_qkv query_states key_states value_statesattention_interfacerrs r:rInstructBlipAttention.forwards#0"4"4"6iHH]+ %%cAt~~yTbTbGbckk q!Q 2;1y|YWX\, (? ;; + +w 6"9$++:Z:Z"[ $7     %  #}}C$2H2HJJ %  % ! "))#;FFH ook2 ((rA) rrOrXrrrrrrN)rCrDrErFrGrVrHrrrrr=rrKrrs@r:rrsGD>eU\\eCece -1$)||$)ELL)$) u||Xell3XeELL>Q5RR S $)$)rArcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InstructBlipMLPic>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) rUrVrOr hidden_act activation_fnrrrWintermediate_sizefc1fc2rcs r:rVInstructBlipMLP.__init__sb  #F$5$5699V//1I1IJ99V55v7I7IJrArr1clURU5nURU5nURU5nU$r)rrrr9rs r:rInstructBlipMLP.forward s4/ **=9 / rA)rrOrr rCrDrErFrVrHrrrKrrs@r:rrs)KU\\ellrArc ^\rSrSrS\4U4Sjjr\S\RS\RS\ \ S\R4Sj5r S r U=r$) InstructBlipEncoderLayerirOc<>[TU]5 URUl[ U5Ul[ R"URURS9Ul [U5Ul [ R"URURS9Ul gNeps) rUrVrWrXr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rcs r:rV!InstructBlipEncoderLayer.__init__sm ++.v6<<F> '$   &0  ((7 / %0 rA)rXrrrr)rCrDrErFr%rVrrHrrrrIrrKrrs@r:rrs`S1S|| +,     rArcJ\rSrSr%\\S'SrSrSrSr Sr Sr Sr /SQr SrSrg) InstructBlipPreTrainedModeli5rOblipT)InstructBlipQFormerEmbeddingsr%InstructBlipQFormerMultiHeadAttentionInstructBlipQFormerSelfOutputcURRn[U[R[R 45(aWUR RRSUS9 URb%URRR5 gg[U[R5(a%UR RRSUS9 g[U[R5(aJURRR5 UR RRS5 g[U[5(aS[RR!UR"SUS9 [RR!UR$SUS9 g[U[&[(45(a%UR*RR5 gg)zInitialize the weightsr)meanstdN?)rOinitializer_range isinstancerrr^rdatanormal_rzero_ Embeddingrfill_rMinit trunc_normal_rbr]$InstructBlipForConditionalGenerationInstructBlipModel query_tokens)r9rfactors r: _init_weights)InstructBlipPreTrainedModel._init_weightsHsX.. fryy"))4 5 5 MM   & &CV & <{{&   &&('  - - MM   & &CV & <  - - KK   " " $ MM   $ $S )  < = = GG ! !&";";#6 ! R GG ! !&"8"8s ! O !EGX Y Z Z    $ $ * * ,[rArBN)rCrDrErFr%rJbase_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrrKrBrAr:rr5s> &*#"&N!-rArc ^\rSrSrSrS\4U4Sjjr\S S\\ RS\ \ S\ \\44Sjj5rS rU=r$) InstructBlipEncoderi]z Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipEncoderLayer`]. Args: config (`InstructBlipConfig`): The corresponding vision configuration for the `InstructBlipEncoder`. rOc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfNF) rUrVrOr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)r9rOrrds r:rVInstructBlipEncoder.__init__gsT  mmuU[UmUmOn$oOn!%=f%EOn$op &+#%pA&rrr1c TUnURHnU"U4SU0UD6nM [US9$)Nrlast_hidden_state)rr)r9 inputs_embedsrrr encoder_layers r:rInstructBlipEncoder.forwardmsC& ![[M)-M)??rA)rOrrr)rCrDrErFrGr%rVrrrHrrrrr=rrrKrrs@r:rr]sj,1, 26@!.@+, @ uo% & @@rArc ^\rSrSr%Sr\\S'\\S.r S\4U4Sjjr \ "SS9\ SS\ \RS\S \\S \\\44S jj55rS rS rU=r$)InstructBlipVisionModelirrO)r attentionsc>[TU]U5 XlURn[ U5Ul[ U5Ul[R"X!RS9Ul UR5 gr) rUrVrOrWrMrfrencoderrrrpost_layernorm post_init)r9rOrXrds r:rV InstructBlipVisionModel.__init__sY   && 6v>*62  ll9:O:OP rAF)tie_last_hidden_statesrrr1c Uc [S5eURXS9nUR"SSU0UD6nURnUR U5nUSS2SSS24nUR U5n[ UUS9$)Nz You have to specify pixel_values)rr rr pooler_outputrB)rrfr'rr(r)r9rrrrencoder_outputsr pooled_outputs r:rInstructBlipVisionModel.forwards  ?@ @ h +/<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  rAcUR$r)rfr?s r:get_input_embeddings,InstructBlipVisionModel.get_input_embeddingss rA)rOrfr'r(r)rCrDrErFmain_input_namer'rJrr_can_record_outputsrVrrrrHrIrrrrr=rrr3rKrrs@r:r$r$s$O $$1+ 7 u559). u001 #' +,  u00 1  6 6rAr$cj^\rSrSrS U4SjjrSrSrSrSrSr S S\ \ 4S jjr S r U=r$) ric>[TU]5 XlURUR-S:wa5[ US5(d$[ SURUR4-5eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul U(aa[R"URUR5Ul[R"URUR5UlO`[R"URUR5Ul[R"URUR5Ul[R "UR"5Ul['USS5UlUR(S:XdUR(S:XaGUR*Ul[R,"SUR*-S - UR5UlS Ulg) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryr r$F)rUrVrOrWrhasattrrrattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrr5r:max_position_embeddingsrdistance_embeddingsave_attentionr9rOis_cross_attentionrds r:rV.InstructBlipQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHiHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##rAcXlgrattn_gradients)r9rLs r:save_attn_gradients9InstructBlipQFormerMultiHeadAttention.save_attn_gradientss,rAcUR$rrKr?s r:get_attn_gradients8InstructBlipQFormerMultiHeadAttention.get_attn_gradientss"""rAcXlgr attention_map)r9rTs r:save_attention_map8InstructBlipQFormerMultiHeadAttention.save_attention_maps*rAcUR$rrSr?s r:get_attention_map7InstructBlipQFormerMultiHeadAttention.get_attention_maps!!!rAcUR5SSURUR4-nUR"U6nUR SSSS5$)Nrjrr r$r )rlrr?rxru)r9x new_x_shapes r:transpose_for_scores:InstructBlipQFormerMultiHeadAttention.transpose_for_scoressLffhsmt'?'?AYAY&ZZ FFK yyAq!$$rArc USLnU(aCURURU55nURURU55n UnO@URURU55nURURU55n URU5n URU 5n [R "XR SS55n URS:XdURS:XGaCUR5Sn [R"U [RURS9RSS5n[R"U [RURS9RSS5nX- nURUUR-S- 5nURU R S9nURS:Xa[R""SU U5nU U-n OHURS:Xa8[R""SU U5n[R""S UU5nU U-U-n U [$R&"UR(5- n U R nUbX-n [*R,"SS 9"U 5RU5nU(a=UR.(a,UR1U5 UR3UR45 UR7U5nUbUU-n[R "UU 5nUR9S S SS 5R;5nUR5SSUR<4-nUR"U6nUU4$)Nrjrr<r=r$rdevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrrorr r )r]rrrrHrrr:rlarangelongrarxrErDrreinsummathsqrtr?rSoftmaxrFrU register_hookrMrrurr@)r9rrrencoder_hidden_statesencoder_attention_maskrrH key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapes r:r-InstructBlipQFormerMultiHeadAttention.forwards3$> 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S%**,CD o--rA)r@r?rTrLrOrErrrDrr:rrFrrNNNN)rCrDrErFrVrMrPrUrXr]rrrrKrrs@r:rrsL$8-#+"%"# I.+,I.I.rArcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ri/c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gr) rUrVrrrWdenserrrBhidden_dropout_probrrcs r:rV&InstructBlipQFormerSelfOutput.__init__0s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rAr input_tensorr1cpURU5nURU5nURX-5nU$rrrrr9rrs r:r%InstructBlipQFormerSelfOutput.forward65 =1  ]3 }'CD rArrrrrs@r:rr/6> U\\RWR^R^rArc^\rSrSrS U4SjjrSrSS\RS\\RS\\RS\\RS\\RS \ \ S \R4S jjr S r U=r$)InstructBlipQFormerAttentioni>c>[TU]5 [X5Ul[ U5Ul[ 5Ulgr)rUrVr attentionroutputset pruned_headsrGs r:rV%InstructBlipQFormerAttention.__init__?s0 >vZ3F; ErAc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr$ro)lenrrrr?rrrrrrrr@union)r9headsindexs r: prune_heads(InstructBlipQFormerAttention.prune_headsEs u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:rArrrrirjrr1c \UR"SUUUUUS.UD6upxURXq5n U $)N)rrrrirjrB)rr) r9rrrrirjrrrattention_outputs r:r$InstructBlipQFormerAttention.forwardWsI ')"7#9      ;;{BrA)rrrrr~)rCrDrErFrVrrHrrrIrrrrKrrs@r:rr>s" ;*7;15=A>B  || !!2!23 E--.  ((9(9:  !)):): ;  +,    rArcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InstructBlipQFormerIntermediateimc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) rUrVrrrWrrrrstrr intermediate_act_fnrcs r:rV(InstructBlipQFormerIntermediate.__init__ns` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rArr1cJURU5nURU5nU$rrrrs r:r'InstructBlipQFormerIntermediate.forwardvs& =1 00? rArrrs@r:rrms(9U\\ellrArcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)InstructBlipQFormerOutputi}c(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rUrVrrrrWrrrrBrrrcs r:rV"InstructBlipQFormerOutput.__init__~s` YYv779K9KL f&8&8f>S>STzz&"<"<= rArrr1cpURU5nURU5nURX-5nU$rrrs r:r!InstructBlipQFormerOutput.forwardrrArrrs@r:rr}rrArcV^\rSrSrU4SjrSS\\4SjjrSrSr Sr U=r $) InstructBlipQFormerLayeric^>[TU]5 URUlSUl[ U5UlX lX!R-S:Xa[ USS9UlSUl OSUl [U5Ul [U5Ul [U5Ul[U5Ulg)Nr$rT)rHF)rUrVchunk_size_feed_forward seq_len_dimrr layer_idxcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterrintermediate_query output_queryr9rOrrds r:rV!InstructBlipQFormerLayer.__init__s '-'E'E$5f=" 77 71 <">vZ^"_D '+D $',D $;FC/7 "A&"I5f=rArc pUR"U4UUS.UD6nUS:aUSS2SU2SS24n UR(a&Uc [S5eUR"U 4UUUUS.UD6n [ UR UR URU 5n URSU:ag[ URUR URUSS2US2SS245RU R5n [R"X/SS9n U $[ URUR URU5n U $)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrrirjr$ro)rrrrrfeed_forward_chunk_queryrrrqfeed_forward_chunkrrarHry) r9rrrrirj query_lengthrrquery_attention_output layer_outputlayer_output_texts r:r InstructBlipQFormerLayer.forwardso >>  )    ! %5a,6I%J "''(0$%eff)-)<)<**#1'*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,)JPQR  5'',,   L rAcJURU5nURX!5nU$r)rrr9rintermediate_outputrs r:r+InstructBlipQFormerLayer.feed_forward_chunks)"//0@A{{#6I rAcJURU5nURX!5nU$r)rrrs r:r1InstructBlipQFormerLayer.feed_forward_chunk_querys+"556FG(()<O rA) rrrrrrrrrrNNNNr) rCrDrErFrVrrrrrrKrrs@r:rrs>>."#6+,6p rArcT^\rSrSrU4Sjr\SS\\4Sjj5rSr U=r $)InstructBlipQFormerEncoderic >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5UlSUl gs snfr) rUrVrOrrrrrlayerrrs r:rV#InstructBlipQFormerEncoder.__init__sY  ]]JOPVPhPhJi jJiY %f 8Ji j  ',# krrc [URR5H*nURUn UbX8OSn U "UUU U4UUS.UD6nM, [ US9$)N)rjrr)rrOrrr) r9rrrrirjrri layer_modulelayer_head_masks r:r"InstructBlipQFormerEncoder.forwardswt{{445A::a=L.7.CilO(%  (>) M 69+  rA)rOrrr) rCrDrErFrVrrrrrKrrs@r:rrs=,"# +,  rArc>^\rSrSrSrU4SjrSSjrSrU=r$)ri z;Construct the embeddings from word and position embeddings.cF>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR!S["R$"UR5R'S5SS9 [)USS5UlXlg) N) padding_idxr position_ids)r$rjF) persistentr:r;)rUrVrr vocab_sizerW pad_token_idword_embeddingsrDposition_embeddingsrr layernormrBrrregister_bufferrHrbrr5r:rOrcs r:rV&InstructBlipQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ rAcUbUR5SnOSnUc%URSS2XEU-24R5nUbiURU5nURS:Xa.UR UR UR55nXg-nUb[R"X64SS9nOUnUR URRR5nURU5nURU5nU$)Nr$rr;ro)rlrclonerr:rrrarHryrrrr)r9 input_idsr query_embedspast_key_values_lengthrprfrs r:r%InstructBlipQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'= '"YY 'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- rA)rOrrr:rr)NNNr) rCrDrErFrGrVrrKrrs@r:rr s#E$ rArc^\rSrSrSrSrSrSrSr\ \ "\ SSS9/\ "\ SSS9/S.r S \ 4U4S jjrS rS rS rSS\R&S\\S\R,S\S\R&4 Sjjr\"5\SS\R6S\\R:S\\R6S\\R&S\\R:S\\R:S\\R:S\\S\ \\R:\!44Sjj55r"Sr#U=r$$)InstructBlipQFormerModeli>z Querying Transformer (Q-Former), used in InstructBLIP. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)r layer_namez.crossattention)rr%cross_attentionsrOc>[TU]U5 Xl[U5Ul[ U5UlUR5 gr)rUrVrOrrfrr'r)rcs r:rV!InstructBlipQFormerModel.__init__Ss7   7?1&9  rAc.URR$rrfrr?s r:r3-InstructBlipQFormerModel.get_input_embeddings]s...rAc$XRlgrrr9rs r:set_input_embeddings-InstructBlipQFormerModel.set_input_embeddings`s */'rAcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr'rrr)r9heads_to_prunerrs r: _prune_heads%InstructBlipQFormerModel._prune_headscs< +002LE LL  u % / / ; ;E B3rAr input_shapera has_queryr1c UR5S:XaUSS2SSS2SS24nO>UR5S:XaUSS2SSSS24nO[SUSURS35eURURS9nSU- S -nU$) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nr z!Wrong shape for input_ids (shape z) or attention_mask (shape )rrg)rprrqrr)r9rrrarextended_attention_masks r:get_extended_attention_mask4InstructBlipQFormerModel.get_extended_attention_maskks.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&rArrrrrirjrc NUcUc [S5eUbURSOSn URUUUS9n U R5SSn U upU RnUc[ R "X4US9nURX+U5nUb[U[5(aUSR5unnnOUR5unnnUU4n[U[5(a"UVs/sHnURU5PM nnO>Uc'[ R "UUS9nURU5nOURU5nOSnURXPRR5nUR"U 4UUUUU S.UD6nURnUSS2SSS24n[!UUS 9$s snf) a query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rrrrj)ra)rrrirjrr-)rrqrfrlrarHonesrrlistinvert_attention_mask get_head_maskrOrr'rr)r9rrrrrrirjrrembedding_outputrrrprarencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapemaskencoder_extended_attention_maskr/sequence_outputr0s r:r InstructBlipQFormerModel.forwards$  !5VW W0<0H|))!,a ??%%+  '++-cr2 !, !((  !"ZZ*)A6RN#'"B"B>`f"g ! ,/66AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$77`v2w`vX\43M3Md3S`v/2w/'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y++2O2OP +/<< , 2"7#B% , , *;;'1a0 ;-'  93xs&F")rOrfr'r)NNNNNN)%rCrDrErFrGr r rrrrrr6r&rVr3rrrHrr=rrarrrr LongTensorrrIrrrrrrKrrs@r:rr>s #( N2 @Vb c  @Vg h 8/0C )' )'3Z)' )'  )'  )'V7;37/315=A>BO ##O !!2!23O u//0 O u||, O E--. O  ((9(9:O !)):): ;O +,O  uU&&')UU VO O rArz[ InstructBLIP base Model consisting of language model, qformer and vision encoder. c^\rSrSrSrS/rS\4U4SjjrSrSr Sr S r S \ RS \ R4S jr\\SS\ RS \ RS\\ RS \\ RS\\ RS\\ RS\\ RS \\ R&S\S\\S\\\44Sjj55rSrU=r$)rirrrOc4>[TU]U5 [UR5Ul[ R "[R"SURURR55Ul [UR5Ul[ R"URRUR R5Ul[$R&"UR 5UlUR(R*b/UR*R-UR(R*5 UR(R.b/UR.R-UR(R.5 UR15 gNr$)rUrVr$ vision_config vision_modelrr[rHrnum_query_tokensqformer_configrWrrqformerr text_configlanguage_projectionr! from_configlanguage_modelrextend_keep_in_fp32_modulesr)rcs r:rVInstructBlipModel.__init__s  3F4H4HILLQ8O8OQWQfQfQrQr)st/0E0EF #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG    0 0 <  " " ) )$*=*=*O*O P    4 4 @  & & - -d.A.A.W.W X rAc6URR5$rrr3r?s r:r3&InstructBlipModel.get_input_embeddings""7799rAc:URRU5 grrrrs r:r&InstructBlipModel.set_input_embeddings  007rAcURR(d_URRURRlURRURR lggrrOuse_decoder_only_language_modelrsharedr' embed_tokensdecoderr?s r: _tie_weightsInstructBlipModel._tie_weights T{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;rAc"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r$raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprrHcuda device_countloggerwarningr>rr/io_same_devicer9r1s r:_preprocess_accelerate(InstructBlipModel._preprocess_accelerate| ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3 3:>D   ( ( 7 4rArr c UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnURS5RU5RUR 5nU$zJ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. r`rj r3rHrrOimage_token_idrcraall unsqueeze expand_asrr9rr special_image_masks r:get_placeholder_mask&InstructBlipModel.get_placeholder_mask%  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H /99"=GG VYYZgZnZno!!rAqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskrrr1c "UR"S UU S.U D6n U Sn [R"U R5SS[RU R S9n UR RU RSSS5n[R"UR5SS[RU R S9nUc[R"U5n[R"X/SS9nUR"S UUUU U S.U D6nUSSS2SURS52SS24nUc9URR5"U5nUc[R"U5nURU5nURUR UR 5nUR#XHS 9nUR%UU5nUR&R((aUR"S UUS .U D6nOUR"S UUUUS .U D6n[+U UUS 9$)aK qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. )rrrNrjr`r$ro)rrrrirjr r r)r rrIrJr4rB)rrHrrlrcrarrrq ones_likeryrrr3rrrrDmasked_scatterrOr&r*)r9rrGrHrrrIrJr rrr. image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsrCoutputss r:rInstructBlipModel.forward4s@P** %%=   &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!&,@+Y_`!a  '1%".#7    %Q'+A\->->q-A+A1(DE   //DDFyQM%!&!;!% 8 8 F 5 8 89M9M}ObOb c!66y6^%445GI^_ ;; 6 6))+-G ))+-"3'=   G?))#*  rArrrrr)NNNNNNF)rCrDrErFr5rr%rVr3rr*r8rHr rIrDrrrrrrr rr=r*rrKrrs@r:rrs^ %O+,1&:8R ?( "e.>.> "uO`O` " >B15598<=A04).^ ''^ !,,^ !))9)9 : ^ E--. ^ !!1!12 ^ $E$4$45^ !))9)9 :^   -^ #'^ -.^  uEE F^ ^ rAra InstructBLIP Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c^\rSrSr%\\S'SrSrS/rS\4U4Sjjr Sr Sr S r S \ R4S jrS rS rSrSrS S\R*S\R,S\\R,S\\S\\4 SjjrS\R,S\R*4Sjr\\S!S\R*S\R*S\\R,S\\R*S\\R,S\\R,S\\R,S\\R*S\\R,S\S\\S \\ \!44Sjj55r"\RF"5S"S\R*S\\R,S\\R,S\\R,S\\R,S\\R*S\S \R,4Sjj5r$Sr%U=r&$)#rirOrTrcr>[TU]U5 [RUR5Ul[ R"[R"SURURR55Ul [RUR5Ul[ R "URRUR"R5UlUR&(a![(R*"UR"5nO [,R*"UR"5nUR.b%UR.R1UR.5 UR2b%UR2R1UR25 X lUR75 gr)rUrVr$ _from_configrrrr[rHrrrrWrrrrrrr&r"rr#rrrrr))r9rOrrds r:rV-InstructBlipForConditionalGeneration.__init__s1  3@@AUAUVLLQ8O8OQWQfQfQrQr)st/<P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, rAc6URR5$rrr?s r:r39InstructBlipForConditionalGeneration.get_input_embeddingsrrAc:URRU5 grr!rs r:r9InstructBlipForConditionalGeneration.set_input_embeddingsr#rAc:URRU5 gr)rset_output_embeddings)r9new_embeddingss r:rb:InstructBlipForConditionalGeneration.set_output_embeddingss 11.ArAr1c6URR5$r)rget_output_embeddingsr?s r:rf:InstructBlipForConditionalGeneration.get_output_embeddingss""88::rAc6URR5$r)r get_encoderr?s r:ri0InstructBlipForConditionalGeneration.get_encoder""..00rAc6URR5$r)r get_decoderr?s r:rm0InstructBlipForConditionalGeneration.get_decoderrkrAcURR(d_URRURRlURRURR lggrr%r?s r:r*1InstructBlipForConditionalGeneration._tie_weightsr,rAc"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggr.r0r7s r:r8;InstructBlipForConditionalGeneration._preprocess_accelerater:rArGrHr return_dictc URUUSS9nUSn[R"UR5SS[RUR S9nUR RURSSS5n [R"U R5SS[RUR S9n Uc[R"U5n[R"X/SS9nURUUU UUSS 9n U SSS2SU RS52SS24n URU 5n U(aXU 4$U $) z Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. T)rrrsrNrjr`r$ro)rrrrirjrs) rrHrrlrcrarrrqrNryrr)r9rrGrHrrsr.rPrQrrRrSrTrUs r:get_image_features7InstructBlipForConditionalGeneration.get_image_featuressT"**%%=+  &a(  %zz,*;*;*=cr*B%**]i]p]pq((// 0B0B10Er2N $zz,*;*;*=cr*B%**]i]p]pq ! )%*__5F%G "!&,@+Y_`!a '1%".#7 % %Q'+A\->->q-A+A1(DE !% 8 8 F (-G G$$rArr c UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnURS5RU5RUR 5nU$r<r=rBs r:rD9InstructBlipForConditionalGeneration.get_placeholder_maskrFrArrIrJlabelsrc ~URUUUU SS9upnUcUR5"U5nUc[R"U5nU R UR UR 5n URXHS9nURX5nURR(aSUR"S UUS.U D6nUSnSnU b3UR"S UXRRRS.U D6nO5SU S'UR"S UUUUU S .U D6nURnUR n[#UUU UUS 9$) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipProcessor`]. See [`InstructBlipProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration >>> import torch >>> from PIL import Image >>> import requests >>> model = InstructBlipForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> processor = InstructBlipProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> model.to(device) # doctest: +IGNORE_RESULT >>> url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> prompt = "What is unusual about this image?" >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... min_length=1, ... top_p=0.9, ... repetition_penalty=1.5, ... length_penalty=1.0, ... temperature=1, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) The unusual aspect of this image is that a man is ironing clothes on the back of a yellow SUV, which is parked in the middle of a busy city street. This is an unconventional approach to ironing clothes, as it requires the man to balance himself and his ironing equipment on top of the vehicle while navigating through traffic. Additionally, the presence of taxis and other vehicles in the scene further emphasizes the unusual nature of this situation. ```TrGrHrrsNrLrMr)r-ryrrs)r rrIrJry)r,r-r.r/r0rB)rur3rHrNrrarrDrOrOr&r loss_functionrrr,r-r*)r9rrGrHrrrIrJr ryrrrUr.rSrCrVr-r,s r:r,InstructBlipForConditionalGeneration.forward.sZ@D?V?V /#9%= @W@ <}   557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445G_ ;; 6 6))+-G QZFD!))!&[[=T=T=_=_ci %)F= !))+-"3'=   G<))#*   rAc D[US5(aUR5 URSn URUUUUSS9upn UcUcURR /URR -n XRRR/-n[R"U/[RURS9nURU S5nUR5"U5nUc[R"U5nU R!URUR"5n UR%XFS9nUR'X5nXeS.nUR(RR*(dUUS 'UR(R,"S 0UDUD6nU$) a Overrides `generate` function to be able to use the model as a conditional generator. Args: pixel_values (`torch.FloatTensor` of shape (batch_size, num_channels, height, width)): Input images to be processed. qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. input_ids (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): The sequence used as a prompt for the generation. attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Embedded representation of the inputs. Should be float, not int tokens. interpolate_pos_encoding (`bool`, *optional*, defaults to `False`): Whether to interpolate the positional encoding of the image embeddings. Returns: captions (list): A list of strings of length batch_size * num_captions. r1rTr{r`r$rLrMrrB)r>r8rqrurOimage_token_indexrr bos_token_idrHrrcrarepeatr3rNrrrDrOris_encoder_decodergenerate)r9rrGrHrrr rgenerate_kwargsrrUr.rS image_tokens start_tokensrCinputsrVs r:r-InstructBlipForConditionalGeneration.generatesD 4 ) )  ' ' )!''* ?C?V?V /#9%= @W@ <}   $ = =>A]A]] +{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z;  557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445G_ #0S""))<<"+F; %%..KK?KrArX)NFF)NNNNNNNF)NNNNNF)'rCrDrErFr%rJr5rrrVr3rrbrModulerfrirmr*r8rHrIr rrrurDrrrrrr=r*rno_gradrrKrrs@r:rrs $O!+,14:8B;ryy;11R ?0>B38&+ /%''/%!++/%!))9)9 : /% #+4. /% d^ /%b "e.>.> "uO`O` " >B15598<=A59-1).} ''} !,,} !))9)9 : } E--. } !!1!12 } $E$4$45} !))9)9 :}   1 12} ))*} #'} +,}  uEE F} } ~ ]]_9==A045959).C''C$E$4$45C!))9)9 : C E,,- C !!1!12 C  1 12C#'C   CCrAr)rrrrr$)r)NrGre dataclassesrtypingrrrrrHr activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor!r"r#configuration_instructblipr%r&r' get_loggerrCr4r*rrMrfloatrrrrrrr$rrrrrrrrrrr__all__rBrAr:rsn" !11 !)B9 G&lljj?IIoo   H %   k