cCiySSKrSSKJr SSKJrJrJrJr SSKrSSKJ r SSK J r SSK J r SSKJr SS KJr SS KJrJrJrJr SS KJrJr SS KJr SS KJrJrJr SSK J!r!J"r"J#r#J$r$J%r%J&r& SSK'J(r(J)r) SSK*J+r+J,r,J-r- SSK.J/r/J0r0J1r1 \%Rd"\35r4"SS\ Rj5r6\#"SS\55r7SGS\ RjS\RpS\RpS\RpS\\RpS\9S\94Sjjr:"S S!\ Rj5r;"S"S#\ Rj5r<"S$S%\5r="S&S'\ Rj5r>"S(S)\75r?"S*S+\ Rj5r@"S,S-\ Rj5rA"S.S/\ Rj5rB"S0S1\ Rj5rC"S2S3\ Rj5rD"S4S5\5rE"S6S7\ Rj5rF"S8S9\ Rj5rG"S:S;\75rH\\#"SS?\!555rI\#"S@S=9"SASB\755rJ\#"SCS=9"SDSE\7\ 55rK/SFQrLg)HN) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipVideoConfigInstructBlipVideoQFormerConfigInstructBlipVideoVisionConfigc^\rSrSrS\4U4SjjrS\RS\S\S\R4Sjr S S \RS \ S\R4S jjr S r U=r$)!InstructBlipVideoVisionEmbeddings7configcr>[TU]5 XlURUlUR UlUR Ul[R"[R"SSUR55Ul [R"SURUR UR S9Ul UR UR -S-UlURS-Ul[R"[R"SURUR55Ulg)Nr$r ) in_channels out_channels kernel_sizestrider )super__init__r+ hidden_size embed_dim image_size patch_sizer Parametertorchrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingselfr+ __class__s z/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/instructblipvideo/modeling_instructblipvideo.pyr2*InstructBlipVideoVisionEmbeddings.__init__8s  ++ ++ ++!||EKK1dnn,MN!yyDOO\`\k\k !OOt>1D!--1"$,,u{{1d>P>PRVR`R`/a"b embeddingsheightwidthreturncURSS- nURRSS- n[RR 5(dXE:XaX#:Xa UR$URSS2SS24nURSS2SS24nURSnX R -n X0R -n [ US-5n URSXU5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU5n[R"Xg4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rr r bicubicF)sizemode align_cornersdim)shaper?r8jit is_tracingr6rreshapepermuter functional interpolateviewcat) rArFrGrHr=r>class_pos_embedpatch_pos_embedrQ new_height new_widthsqrt_num_positionss rCinterpolate_pos_encoding:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingJsS!&&q)A- //55a81< yy##%%+*F6?** *11!RaR%811!QR%8r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCrE pixel_valuesr`cURup4pVURRRnURUR US95nUR S5R SS5nURRUSS5R U5n [R"X/SS9n U(aURXU5n O URn XSS2SU RS52SS24R U5-n U $)Ndtyper r$rKrP)rRr<weightretoflatten transposer:expandr8rZr`r?rM) rArbr` batch_size_rGrH target_dtype patch_embeds class_embedsrFr?s rCforward)InstructBlipVideoVisionEmbeddings.forwardrs'3'9'9$ v++2288 ++LOO,O,OP #++A.88A> ++22:q"EHHV YY ;C #!%!>!>zSX!Y !%!8!8 Q8L*//!:L8La5O"P"S"ST`"aa rE) r:r+r4r5r=r>r<r6r?F)__name__ __module__ __qualname____firstlineno__r'r2r8Tensorintr` FloatTensorboolrp__static_attributes__ __classcell__rBs@rCr)r)7src<c$&D5<<&D&DUX&D]b]i]i&DP E$5$5 QU bgbnbn  rEr)cJ\rSrSr%\\S'SrSrSrSr Sr Sr Sr /SQr SrSrg) InstructBlipVideoPreTrainedModelr+blipT)"InstructBlipVideoQFormerEmbeddingsInstructBlipVideoAttention*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputcURRn[U[R[R 45(aWUR RRSUS9 URb%URRR5 gg[U[R5(a%UR RRSUS9 g[U[R5(aJURRR5 UR RRS5 g[U[5(aS[RR!UR"SUS9 [RR!UR$SUS9 g[U[&[(45(a%UR*RR5 gg)zInitialize the weights)meanstdN?)r+initializer_range isinstancerLinearr;rfdatanormal_biaszero_ Embedding LayerNormfill_r)init trunc_normal_r?r:)InstructBlipVideoForConditionalGenerationInstructBlipVideoModel query_tokens)rAmodulefactors rC _init_weights.InstructBlipVideoPreTrainedModel._init_weightssX.. fryy"))4 5 5 MM   & &CV & <{{&   &&('  - - MM   & &CV & <  - - KK   " " $ MM   $ $S )  A B B GG ! !&";";#6 ! R GG ! !&"8"8s ! O !JLb c d d    $ $ * * ,erEN)rsrtrurvr%__annotations__base_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrr{rrErCrrs> ##&*#"&N!-rErrquerykeyvalueattention_maskscalingdropoutc `[R"XRSS55U-nUbX-n[RR USS9n[RR XURS9n[R"X5n U RSS5R5n X4$)NrKrP)ptrainingr$r ) r8matmulrirrWsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs rCeager_attention_forwardrs<<}}R'<=GL!#4 ==((2(>L==((6??([L,,|3K''1-88:K  $$rEc ^\rSrSrSrU4SjrS\RS\S\4Sjr S S\RS \ \RS \ \R\ \R\ \ \R44S jjr S r U=r$)rz=Multi-headed attention from 'Attention Is All You Need' paperc>[TU]5 XlURUlUR UlURUR -UlURUR -UR:wa&[SURSUR S35eURS-Ul SUl URUl [R"URSUR-SS9UlUR(ai[R "["R$"UR55n[R "["R$"UR55nOSnSnUbQ["R&"U["R("USS9U45n[R "U5URl[R"URUR5Ulg) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )r) requires_grad)r1r2r+r3r4num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrrqkvqkv_biasr7r8zerosrZ zeros_liker projection)rAr+q_biasv_biasrrBs rCr2#InstructBlipVideoAttention.__init__ss  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( !'!9!999T^^Q-?eL ??\\%++dnn"=>F\\%++dnn"=>FFF  yy&%*:*:6QV*WY_!`aHLL2DHHM))DNNDNNCrEtensorseq_lenbszcURX2URUR5RSS5R 5$)Nr$r )rYrrrir)rArrrs rC_shape!InstructBlipVideoAttention._shapes5{{3GQQRSUVWbbddrE hidden_states head_maskrIc 4UR5upEnURU5nURXESURX`R-5R SSSSS5nUSUSUSpn[ n UR RS:wa[UR Rn U "UUU U 4SUR(dSO URURS .UD6upU RXES 5R5n URU 5n X4$) z#Input shape: Batch x Time x Channelr r rr$eagerNr)rrrrK)rMrrUrrVrr+_attn_implementationrrrrrr)rArrrrtgt_lenr4 mixed_qkv query_states key_states value_statesattention_interfacerrs rCrp"InstructBlipVideoAttention.forwards#0"4"4"6iHH]+ %%cAt~~yTbTbGbckk q!Q 2;1y|YWX\, (? ;; + +w 6"9$++:Z:Z"[ $7     %  #}}C$2H2HJJ %  % ! "))#;FFH ook2 ((rE) rr+r4rrrrrrN)rsrtrurv__doc__r2r8rwrxrrtuplerpr{r|r}s@rCrrsGD>eU\\eCece -1$)||$)ELL)$) u||Xell3XeELL>Q5RR S $)$)rErcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)InstructBlipVideoMLPi c>[TU]5 Xl[URUl[ R"URUR5Ul [ R"URUR5Ul gr) r1r2r+r hidden_act activation_fnrrr3intermediate_sizefc1fc2r@s rCr2InstructBlipVideoMLP.__init__ sb  #F$5$5699V//1I1IJ99V55v7I7IJrErrIclURU5nURU5nURU5nU$r)rrrrArs rCrpInstructBlipVideoMLP.forwards4/ **=9 / rE)rr+rr rsrtrurvr2r8rwrpr{r|r}s@rCrr s)KU\\ellrErc ^\rSrSrS\4U4Sjjr\S\RS\RS\ \ S\R4Sj5r S r U=r$) InstructBlipVideoEncoderLayerir+c<>[TU]5 URUl[ U5Ul[ R"URURS9Ul [U5Ul [ R"URURS9Ul gNeps) r1r2r3r4r self_attnrrlayer_norm_eps layer_norm1rmlp layer_norm2r@s rCr2&InstructBlipVideoEncoderLayer.__init__sm ++3F;<<F> '$   &0  ((7 / %0 rE)r4rrrr)rsrtrurvr%r2rr8rwrrryrpr{r|r}s@rCrrs`S6S|| +,     rErc ^\rSrSrSrS\4U4Sjjr\S S\\ RS\ \ S\ \\44Sjj5rS rU=r$) InstructBlipVideoEncoderi=a  Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r+c>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snfNF) r1r2r+r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)rAr+rlrBs rCr2!InstructBlipVideoEncoder.__init__GsU  mmTYZ`ZrZrTs$tTsq%B6%JTs$tu &+#%uA&rrrIc TUnURHnU"U4SU0UD6nM [US9$)Nrlast_hidden_state)rr)rA inputs_embedsrrr encoder_layers rCrp InstructBlipVideoEncoder.forwardMsC& ![[M)-M)??rE)r+rrr)rsrtrurvrr%r2rrr8rwrrrrrrpr{r|r}s@rCrr=sj,6, 26@!.@+, @ uo% & @@rErc ^\rSrSr%Sr\\S'\\S.r S\4U4Sjjr \ "SS9\ SS\ \RS\S \\S \\\44S jj55rS rS rU=r$)InstructBlipVideoVisionModeli_rbr+)r attentionsc>[TU]U5 XlURn[ U5Ul[ U5Ul[R"X!RS9Ul UR5 gr) r1r2r+r3r)rFrencoderrrrpost_layernorm post_init)rAr+r4rBs rCr2%InstructBlipVideoVisionModel.__init__gsY   && ;FC/7  ll9:O:OP rEF)tie_last_hidden_statesr`rrIc Uc [S5eURXS9nUR"SSU0UD6nURnUR U5nUSS2SSS24nUR U5n[ UUS9$)Nz You have to specify pixel_values)r`r rr  pooler_outputr)rrFrr rr)rArbr`rrencoder_outputsr  pooled_outputs rCrp$InstructBlipVideoVisionModel.forwardrs  ?@ @ h +/<<, ', ,  ,== //0AB)!Q'2 ++M: )/'  rEcUR$r)rFrAs rCget_input_embeddings1InstructBlipVideoVisionModel.get_input_embeddingss rE)r+rFrrr)rsrtrurvmain_input_namer'rrr_can_record_outputsr2rrrr8ryrzrrrrrrpr r{r|r}s@rCrr_s$O ))60 < u559). u001 #' +,  u00 1  6 6rErcj^\rSrSrS U4SjjrSrSrSrSrSr S S\ \ 4S jjr S r U=r$) ric>[TU]5 XlURUR-S:wa5[ US5(d$[ SURUR4-5eURUl[URUR- 5UlURUR-Ul [R"URUR5Ul U(aa[R"URUR5Ul[R"URUR5UlO`[R"URUR5Ul[R"URUR5Ul[R "UR"5Ul['USS5UlUR(S:XdUR(S:XaGUR*Ul[R,"SUR*-S - UR5UlS Ulg) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryr r$F)r1r2r+r3rhasattrrrxattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrgetattrr'max_position_embeddingsrdistance_embeddingsave_attentionrAr+is_cross_attentionrBs rCr23InstructBlipVideoQFormerMultiHeadAttention.__init__s     : : :a ?PVXhHiHi^%%v'A'ABC  $*#=#= #&v'9'9F 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD ##rEcXlgrattn_gradients)rAr:s rCsave_attn_gradients>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientss,rEcUR$rr9rs rCget_attn_gradients=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientss"""rEcXlgr attention_map)rArBs rCsave_attention_map=InstructBlipVideoQFormerMultiHeadAttention.save_attention_maps*rEcUR$rrArs rCget_attention_mapbhlrzbhrd,lrd->bhlrrPrr r )rKrrrr8rrir'rMarangelongrOrYr3r2rgreeinsummathsqrtr,rSoftmaxr4rC register_hookr;rrVrr-)rArrrencoder_hidden_statesencoder_attention_maskrr6 key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapes rCrp2InstructBlipVideoQFormerMultiHeadAttention.forwards3$> 11$((;P2QRI33DJJ?T4UVK3N11$((=2IJI33DJJ}4MNK JJ}5//0AB !<< 5H5HR5PQ  ' '> 9T=Y=Y]q=q&++-a0J"\\*EJJ}OcOcdiijlnopN"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||AABXY $"5"5  # #O 4  ) )$*B*B C#',,"?  &= &I # %S"S%**,CD o--rE)r-r,rBr:r+r3rrr2rr'rr4rrrNNNN)rsrtrurvr2r;r>rCrFrKrrrpr{r|r}s@rCrrsL$8-#+"%"# I.+,I.I.rErcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)ri c(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gr) r1r2rrr3denserrr/hidden_dropout_probrr@s rCr2+InstructBlipVideoQFormerSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rEr input_tensorrIcpURU5nURU5nURX-5nU$rrorrrArrrs rCrp*InstructBlipVideoQFormerSelfOutput.forward5 =1  ]3 }'CD rErrorrr}s@rCrr 6> U\\RWR^R^rErc^\rSrSrS U4SjjrSrSS\RS\\RS\\RS\\RS\\RS \ \ S \R4S jjr S r U=r$)!InstructBlipVideoQFormerAttentionic>[TU]5 [X5Ul[ U5Ul[ 5Ulgr)r1r2r attentionroutputset pruned_headsr5s rCr2*InstructBlipVideoQFormerAttention.__init__s0 CF_8@ ErEc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrr$rP)lenrr}rr,rrrrrr~ror-union)rAheadsindexs rC prune_heads-InstructBlipVideoQFormerAttention.prune_heads"s u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:rErrrrWrXrrIc \UR"SUUUUUS.UD6upxURXq5n U $)N)rrrrWrXr)r}r~) rArrrrWrXrrrlattention_outputs rCrp)InstructBlipVideoQFormerAttention.forward4sI ')"7#9      ;;{BrE)r}r~rrrrl)rsrtrurvr2rr8rwrryrrrpr{r|r}s@rCr{r{s" ;*7;15=A>B  || !!2!23 E--.  ((9(9:  !)):): ;  +,    rEr{cb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)$InstructBlipVideoQFormerIntermediateiIc>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl gr) r1r2rrr3rrorrstrr intermediate_act_fnr@s rCr2-InstructBlipVideoQFormerIntermediate.__init__Js` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $rErrIcJURU5nURU5nU$rrorrs rCrp,InstructBlipVideoQFormerIntermediate.forwardRs& =1 00? rErrr}s@rCrrIs(9U\\ellrErcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)InstructBlipVideoQFormerOutputiXc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) r1r2rrrr3rorrr/rprr@s rCr2'InstructBlipVideoQFormerOutput.__init__Ys` YYv779K9KL f&8&8f>S>STzz&"<"<= rErrrrIcpURU5nURU5nURX-5nU$rrtrus rCrp&InstructBlipVideoQFormerOutput.forward_rwrErxrr}s@rCrrXryrErcV^\rSrSrU4SjrSS\\4SjjrSrSr Sr U=r $) InstructBlipVideoQFormerLayerifc^>[TU]5 URUlSUl[ U5UlX lX!R-S:Xa[ USS9UlSUl OSUl [U5Ul [U5Ul [U5Ul[U5Ulg)Nr$rT)r6F)r1r2chunk_size_feed_forward seq_len_dimr{r} layer_idxcross_attention_frequencycrossattentionhas_cross_attentionr intermediaterr~intermediate_query output_queryrAr+rrBs rCr2&InstructBlipVideoQFormerLayer.__init__gs '-'E'E$:6B" 77 71 <"CF_c"dD '+D $',D $@H4V< "Fv"N:6BrErc pUR"U4UUS.UD6nUS:aUSS2SU2SS24n UR(a&Uc [S5eUR"U 4UUUUS.UD6n [ UR UR URU 5n URSU:ag[ URUR URUSS2US2SS245RU R5n [R"X/SS9n U $[ URUR URU5n U $)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrrWrXr$rP)r}rrrrfeed_forward_chunk_queryrrrRfeed_forward_chunkrgrOr8rZ) rArrrrWrX query_lengthrrquery_attention_output layer_outputlayer_output_texts rCrp%InstructBlipVideoQFormerLayer.forward{so >>  )    ! %5a,6I%J "''(0$%eff)-)<)<**#1'*?+A *  *&5--,,  & L %%a(<7$=++00$$$Q q%89 % "\(() " %yy,)JPQR  5'',,   L rEcJURU5nURX!5nU$r)rr~rArintermediate_outputrs rCr0InstructBlipVideoQFormerLayer.feed_forward_chunks)"//0@A{{#6I rEcJURU5nURX!5nU$r)rrrs rCr6InstructBlipVideoQFormerLayer.feed_forward_chunk_querys+"556FG(()<O rE) r}rrrrrrr~rrNNNNr) rsrtrurvr2rrrprrr{r|r}s@rCrrfs?C."#6+,6p rErcT^\rSrSrU4Sjr\SS\\4Sjj5rSr U=r $)InstructBlipVideoQFormerEncoderic >[TU]5 Xl[R"[ UR 5Vs/sHn[X5PM sn5UlSUl gs snfr) r1r2r+rrrrrlayerrrs rCr2(InstructBlipVideoQFormerEncoder.__init__sY  ]]OTU[UmUmOn oOn) *6 =On o  ',# prrc [URR5H*nURUn UbX8OSn U "UUU U4UUS.UD6nM, [ US9$)N)rXrr )rr+rrr) rArrrrWrXrri layer_modulelayer_head_masks rCrp'InstructBlipVideoQFormerEncoder.forwardswt{{445A::a=L.7.CilO(%  (>) M 69+  rE)r+rrr) rsrtrurvr2rrrrpr{r|r}s@rCrrs=,"# +,  rErc>^\rSrSrSrU4SjrSSjrSrU=r$)riz;Construct the embeddings from word and position embeddings.cF>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"UR URS9Ul [R"UR5UlUR!S["R$"UR5R'S5SS9 [)USS5UlXlg) N) padding_idxr position_ids)r$rKF) persistentr'r()r1r2rr vocab_sizer3 pad_token_idword_embeddingsr2position_embeddingsrr layernormr/rprregister_bufferr8rPrjr1r'r+r@s rCr2+InstructBlipVideoQFormerEmbeddings.__init__s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  (/v7PR\']$ rEcUbUR5SnOSnUc%URSS2XEU-24R5nUbiURU5nURS:Xa.UR UR UR55nXg-nUb[R"X64SS9nOUnUR URRR5nURU5nURU5nU$)Nr$rr(rP)rMrclonerr'rrgrOr8rZrrfrer)rA input_idsr query_embedspast_key_values_lengthr^rFrs rCrp*InstructBlipVideoQFormerEmbeddings.forwards  ")!,JJ  ,,Q0FVlIl0l-lmssuL  --i8J++z9&*&>&>|zO`O`?a&b#'= '"YY 'AqI %J]]4>>#8#8#>#>? ^^J/ \\*- rE)r+rrr'rr)NNNr) rsrtrurvrr2rpr{r|r}s@rCrrs#E$ rErc^\rSrSrSrSrSrSrSr\ \ "\ SSS9/\ "\ SSS9/S.r S \ 4U4S jjrS rS rS rSS\R&S\\S\R,S\S\R&4 Sjjr\"5\SS\R6S\\R:S\\R6S\\R&S\\R:S\\R:S\\R:S\\S\ \\R:\!44Sjj55r"Sr#U=r$$)InstructBlipVideoQFormerModeliz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)r layer_namez.crossattention)rrcross_attentionsr+c>[TU]U5 Xl[U5Ul[ U5UlUR5 gr)r1r2r+rrFrrrr@s rCr2&InstructBlipVideoQFormerModel.__init__-s7     rEc.URR$rrFrrs rCr 2InstructBlipVideoQFormerModel.get_input_embeddings7s...rEc$XRlgrrrArs rCset_input_embeddings2InstructBlipVideoQFormerModel.set_input_embeddings:s */'rEcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr}r)rAheads_to_prunerrs rC _prune_heads*InstructBlipVideoQFormerModel._prune_heads=s< +002LE LL  u % / / ; ;E B3rEr input_shaperO has_queryrIc UR5S:XaUSS2SSS2SS24nO>UR5S:XaUSS2SSSS24nO[SUSURS35eURURS9nSU- S -nU$) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nr z!Wrong shape for input_ids (shape z) or attention_mask (shape )rdrg)rQrrRrgre)rArrrOrextended_attention_masks rCget_extended_attention_mask9InstructBlipVideoQFormerModel.get_extended_attention_maskEs.    1 $&4Qa]&C #    !Q &'5QdA5E&F #3K=@[\j\p\p[qqrs #:"<"<4::"<"N#&)@#@H"L&&rErrrrrWrXrc NUcUc [S5eUbURSOSn URUUUS9n U R5SSn U upU RnUc[ R "X4US9nURX+U5nUb[U[5(aUSR5unnnOUR5unnnUU4n[U[5(a"UVs/sHnURU5PM nnO>Uc'[ R "UUS9nURU5nOURU5nOSnURXPRR5nUR"U 4UUUUU S.UD6nURnUSS2SSS24n[!UUS 9$s snf) a query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rrrrK)rO)rrrWrXrr)rrRrFrMrOr8onesrrlistinvert_attention_mask get_head_maskr+rrr r)rArrrrrrWrXrrembedding_outputrrkr^rOrencoder_batch_sizeencoder_sequence_lengthrlencoder_hidden_shapemaskencoder_extended_attention_maskrsequence_outputrs rCrp%InstructBlipVideoQFormerModel.forwardps$  !5VW W0<0H|))!,a ??%%+  '++-cr2 !, !((  !"ZZ*)A6RN#'"B"B>`f"g ! ,/66AVWXAYA^A^A`>"$;QAVA[A[A]>"$;Q$68O#P 0$77`v2w`vX\43M3Md3S`v/2w/'/).4HQW)X&262L2LMc2d/262L2LMc2d/.2 +&&y++2O2OP +/<< , 2"7#B% , , *;;'1a0 ;-'  93xs&F")r+rFrrr)NNNNNN)%rsrtrurvrrrrrrrrr#r&r2r rrr8rwrrxrOrzrrr LongTensorrryrrrrrpr{r|r}s@rCrrs #( N7 EQ[g h  EQ[l m =/0C )' )'3Z)' )'  )'  )'V7;37/315=A>BO ##O !!2!23O u//0 O u||, O E--. O  ((9(9:O !)):): ;O +,O  uU&&')UU VO O rErzV Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. ) custom_introc\rSrSr%SrSr\\\R\ S'Sr \\\R\ S'Sr \\R\ S'Sr \\\R\ S'Sr\\\R\ S'S \\4S jrS rg) 4InstructBlipVideoForConditionalGenerationModelOutputia~ loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsrIcJ^[U4SjTR555$)Nc3n># UH*nUS;aTUO[TU5R5v M, g7f)rrrN)r1to_tuple).0krAs rC PInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..sC !WW Gq!**, -!s25)rkeysrs`rCr=InstructBlipVideoForConditionalGenerationModelOutput.to_tuples% YY[    rEr)rsrtrurvrrrrr8ryrrrrrrrr{rrErCrrs 04D(5**+ ,315FHU5,,- .526NHU../6:>OXeE$5$567>AEHU5+<+<%=>E %* rErz` InstructBlipVideo base Model consisting of language model, qformer and vision encoder. c#^\rSrSrSrS/rS\4U4SjjrSrSr Sr S r S \ RS \ R4S jr\\SS\ RS \ RS\\ RS \\ RS\\ RS\\ RS\\ RS \\ R&S\\S\\S\\S\S\\S\\S\\\44Sjj55rSrU=r$)rirbrr+c4>[TU]U5 [UR5Ul[ R "[R"SURURR55Ul [UR5Ul[ R"URRUR R5Ul[$R&"UR 5UlUR(R*b/UR*R-UR(R*5 UR(R.b/UR.R-UR(R.5 UR15 gNr$)r1r2r vision_config vision_modelrr7r8rnum_query_tokensqformer_configr3rrqformerr text_configlanguage_projectionr! from_configlanguage_modelrextend_keep_in_fp32_modulesrr@s rCr2InstructBlipVideoModel.__init__s  89M9MNLLQ8O8OQWQfQfQrQr)st4V5J5JK #%99V-B-B-N-NPVPbPbPnPn#o '33F4F4FG    0 0 <  " " ) )$*=*=*O*O P    4 4 @  & & - -d.A.A.W.W X rEc6URR5$rrr rs rCr +InstructBlipVideoModel.get_input_embeddings""7799rEc:URRU5 grrrrs rCr+InstructBlipVideoModel.set_input_embeddings 007rEcURR(d_URRURRlURRURR lggrr+use_decoder_only_language_modelrsharedr embed_tokensdecoderrs rC _tie_weights#InstructBlipVideoModel._tie_weights T{{::7;7J7J7Q7QD   ' ' 47;7J7J7Q7QD   ' ' 4;rEc"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggz Some pre-processing hacks to make the model `accelerate` compatible. Check https://github.com/huggingface/transformers/pull/21707 for more details. r$raThe `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning. Please refer to https://github.com/huggingface/blog/blob/main/accelerate-large-models.md for more details on creating a `device_map` for large models._hf_hookTN hf_device_maprr8cuda device_countloggerwarningr+rr/io_same_devicerAr1s rC_preprocess_accelerate-InstructBlipVideoModel._preprocess_accelerate| ** }  !&6m&KPUPZPZPgPgPilmPm NNM  4&& 3 3:>D   ( ( 7 4rErr c UcjX R5"[R"URR[R UR S95:HnURS5nOXRR:HnURS5RU5RUR 5nU$zJ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. rNrK) r r8rr+image_token_idrQrOall unsqueeze expand_asrgrArr special_image_masks rCget_placeholder_mask+InstructBlipVideoModel.get_placeholder_mask"  !.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*kk.H.H!H /99"=GG VYYZgZnZno!!rEqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr` use_cacherrIc U bU OURRn URunnnnnURUU-UUU5nUR UU U U U S9nUSn[ R "UR5SS[ RURS9nURRURSSS5n[ R "UR5SS[ RURS9nUc[ R"U5nURUSS9nURUSS9n[ R"UU/SS9nURUUUUUU U U S9nUSSS2SURS52SS24nUR!U5nURXRR"U-S5nUcRUR$R'5"U5nX@RR(:HnUc[ R"U5nOiXR'5"[ R*"URR([ RURS95:HnUR-S5nUR/S5R1U5R3UR5nUR3URUR45nUR7UU5nURR8(aUR$"S UUU U U U S .UD6nOUR$"S UUUUU U U U S .UD6n[;UUUS 9$) aU qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)rbrJrKrLr`rrKrNrPr$)rrrrWrXrJrKrLr rrJrKrLrM)r rrHrIrJrKrLrMrr)r+use_return_dictrRrUrr8rrMrQrOrrj ones_likerepeat_interleaverZrrrrr video_token_idrr>r?r@rgremasked_scatterr&r)rArbrFrGrrrHrIr rJrKrLr`rMrrkframeschannelrGrHr image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsrBoutputss rCrpInstructBlipVideoModel.forward1swR&1%->q-A+A1(DE !% 8 8 F!6 = =j++JfJfioJoqs t   //DDFyQM!*kk.H.H!H %!&!;!.2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; /99"=GG VYYZgZnZno 5 8 89M9M}ObOb c%445GI^_ ;; 6 6))+-"3%9'# G)) +-"3'="3%9'#  GD))#*  rErrrrr) NNNNNNNNNFN)rsrtrurvr"rr%r2r rr*r8r8rryrCrrrrwrzrr rrrrpr{r|r}s@rCrrs %O+,6&:8R ?( "e.>.> "uO`O` " >B15598<=A04,0/3&*).$( '' !,, !))9)9 :  E--.  !!1!12  $E$4$45 !))9)9 :   - $D> 'tn d^ #' D> -. uJJ K!  rEra InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c%H^\rSrSr%\\S'SrSrS/rS\4U4Sjjr Sr Sr S r S \ R4S jrS rS rSrSrS$S\R*S\R,S\\R,S\\S\\4 SjjrS\R,S\R*4Sjr\\S%S\R*S\R*S\\R,S\\R*S\\R,S\\R,S\\R,S\\R*S\\S\\S\\R,S\\S\S\\S\\S \\ \!44 S jj55r"\RF"5S&S\R*S\\R,S\\R,S\\R,S\\R,S\\R*S\S \R,4S!jj5r$S$S\R*S\R,S\\R,S\\S\\4 S"jjr%S#r&U=r'$)'rir+rbTrcr>[TU]U5 [RUR5Ul[ R"[R"SURURR55Ul [RUR5Ul[ R "URRUR"R5UlUR&(a![(R*"UR"5nO [,R*"UR"5nUR.b%UR.R1UR.5 UR2b%UR2R1UR25 X lUR75 gr)r1r2r _from_configrrrr7r8rrrr3rrrrrrr&r"rr#rrrrr)rAr+rrBs rCr22InstructBlipVideoForConditionalGeneration.__init__s1  8EEfFZFZ[LLQ8O8OQWQfQfQrQr)st4AA&BWBWX #%99V-B-B-N-NPVPbPbPnPn#o  1 11==f>P>PQN2>>v?Q?QRN  + + 7  " " ) ).*J*J K  / / ;  & & - -n.R.R S, rEc6URR5$rrrs rCr >InstructBlipVideoForConditionalGeneration.get_input_embeddingsrrEc:URRU5 grr!rs rCr>InstructBlipVideoForConditionalGeneration.set_input_embeddingsr#rEc:URRU5 gr)rset_output_embeddings)rAnew_embeddingss rCri?InstructBlipVideoForConditionalGeneration.set_output_embeddingss 11.ArErIc6URR5$r)rget_output_embeddingsrs rCrm?InstructBlipVideoForConditionalGeneration.get_output_embeddingss""88::rEc6URR5$r)r get_encoderrs rCrp5InstructBlipVideoForConditionalGeneration.get_encoder""..00rEc6URR5$r)r get_decoderrs rCrt5InstructBlipVideoForConditionalGeneration.get_decoderrrrEcURR(d_URRURRlURRURR lggrr%rs rCr*6InstructBlipVideoForConditionalGeneration._tie_weightsr,rEc"URn[U5S:a=SU;a7[RR 5S:a[ R S5 [URS5(aSURRl ggr.r0r7s rCr8@InstructBlipVideoForConditionalGeneration._preprocess_accelerater:rErFrGr`rLcg) Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. Nr)rArbrFrGr`rLs rCget_image_featuresr?r@rgrAs rCrC>InstructBlipVideoForConditionalGeneration.get_placeholder_maskrErErrHrIrJrKlabelsrMrc xU bU OURRn URUUUU SS9unnnU (dUR5OUnU (dUR5OUnUcUR 5"U5nUc[ R "U5nURURUR5nURXHS9nURUU5nURR(ajUR"S UUU U U US.UD6nU (a UROUSnSnU b3UR"S UXRR R"S.UD6nOLUR"S UUUUU U U U US. UD6nU (a UR$OUSnU (a UROUS n['UUUUUS 9$) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTrFrGr`rLr rOr)rrr) r rrHrIrJrKrLrrMr$)rrrrrr)r+rPget_video_featuresrr r8rQrgrOrerCrTr&rr loss_functionrrrr)rArbrFrGrrrHrIr rJrKrrLr`rMrr\rrZrBr]rrs rCrp1InstructBlipVideoForConditionalGeneration.forward*sf&1%8C ..0   557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445GI^_ ;; 6 6))+-"3%9'# G(3W^^ FD!))!&[[=T=T=_=_ci )) +-"3'="3%9'#  G$/7<A]A]]`aa +{{/F/F/S/S.TT !LL,uzzR^ReRef %,,Z;  557 BM  !"__Y7N 5 8 89M9M}ObOb c!66y6^%445G_ #0S""))<<"+F; %%..KK?KrEc ^URupgpn URXg-XU 5nURUUSS9n U Sn [R"U R 5SS[R U RS9n URRU RSSS5n[R"UR 5SS[R U RS9nUc[R"U5nURUSS9nURUSS9n[R"X/SS9nURUUUU U SS 9nUSSS2SUR S52SS24nURU5nURX`RR U-S5nU(aUU U4$U$) r{T)rbr`rLrNrKrNrPr$)rrrrWrXrL)rRrUrr8rrMrQrOrrjrQrRrZrrr+r)rArbrFrGr`rLrkrUrVrGrHrrWrXrrYrZr[r\s rCr->q-A+A1(DE !% 8 8 F!6 = =j++JfJfioJoqs t (.-G G$$rEr_)NFF) NNNNNNNNNNFN)NNNNNF)(rsrtrurvr%rr"rrr2r rrirModulermrprtr*r8r8ryrrrzr|rCrrrrrrrrpno_gradrrr{r|r}s@rCrrs} $#$O!+,64:8B;ryy;11R ?0>B38&+  '' !++ !))9)9 :  #+4.  d^  " "e.>.> "uO`O` " >B15598<=A59,0/3-1&*).$(N ''N !,,N !))9)9 : N E--. N !!1!12 N $E$4$45N !))9)9 :N   1 12N $D>N 'tnN ))*N d^N #'N D>N +,!N " uJJ K#N N ` ]]_9==A045959).C''C$E$4$45C!))9)9 : C E,,- C !!1!12 C  1 12C#'C   CCR>B38&+ 9%''9%!++9%!))9)9 : 9% #+4. 9% d^ 9%9%rEr)rrrrr)r)MrS dataclassesrtypingrrrrr8r activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor!r"r#configuration_instructblipvideor%r&r' get_loggerrsr4rr)rrwfloatrrrrrrrrr{rrrrrrrrr__all__rrErCrsp, !11 !)B9 G&lljj?II   H %G GT#-#-#-\% II% <<% % << % U\\* %  %%.I)I)X 299 $>D@ryy@D1#C1hw.w.t  +  + \ 299  RYY U$>Up$ bii$ N00fi $Di X   ;   : F =F  F R}%0PRa}%}%@  rE