bCiLSrSSKrSSKJr SSKJr SSKJrJr SSK r SSK J r SSK J r SS K JrJr SS KJrJrJr SS KJr SS KJrJrJrJrJr SS KJr SSKJrJ r J!r! SSK"J#r#J$r$J%r% SSK&J'r' SSK(J)r)J*r*J+r+ \$RX"\-5r.Sr/\\#"SS9"SS\555r0\\#"SS9"SS\555r1"SS\ Rd5r3"SS\ Rd5r4"SS \ Rd5r5"S!S"\ Rd5r6"S#S$\ Rd5r7"S%S&\ Rd5r8"S'S(\ Rd5r9"S)S*\ Rd5r:"S+S,\ Rd5r;"S-S.\ Rd5r"S2S3\ Rd5r?"S4S5\5r@"S6S7\ Rd5rA"S8S9\ Rd5rBSWS:jrC\#"S;S<\55rD"S=S>\D5rE\#"S?S9"S@SA\D55rF\#"SBS9"SCSD\D55rG"SESF\ Rd5rH"SGSH\ Rd5rI"SISJ\ Rd5rJ\#"SKS9"SLSM\D55rK\#"SNS9"SOSP\D55rL"SQSR\ Rd5rM\#"SSS9"STSU\D55rN/SVQrOg)XzPyTorch BridgeTower ModelN) OrderedDict) dataclass)OptionalUnion)nn)CrossEntropyLoss)ACT2FNQuickGELUActivation)Cache DynamicCacheEncoderDecoderCache)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int)deprecate_kwarg)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigRobertaTokenizerz. Output type of [`BridgeTowerModel`]. ) custom_introc\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr\\ \R\ S'S rg) BridgeTowerModelOutput0a text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. N text_featuresimage_features pooler_output hidden_states attentions)__name__ __module__ __qualname____firstlineno____doc__r&rtorch FloatTensor__annotations__r'r(r)tupler*__static_attributes__r+n/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/bridgetower/modeling_bridgetower.pyr$r$0s|26M8E--.526NHU../615M8E--.58 Output type of ['BridgeTowerForContrastiveLearning'] cP\rSrSr%SrSr\\R\ S'Sr \\R\ S'Sr \\ \R\ S'Sr \\ \R\ S'Sr\\ \R\ S'Sr\\ \R\ S 'Sr\\ \R\ S 'S rg) BridgeTowerContrastiveOutputHa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nlosslogits text_embeds image_embeds cross_embedsr)r*r+)r,r-r.r/r0r;rr1r2r3r<r=r4r>r?r)r*r5r+r6r7r9r9Hs )-D(5$$ %,*.FHU&& '.6:K% 1 123:7;L(5!2!234;7;L(5!2!234;8[TU]5 [R"URURS-5Ul[R "URURS9Ul[R"[S[R"URURS-54S[54S[R"URS-UR54/55Ul [R "URURS9UlSUlg)N@epsc_fcgeluc_proj)super__init__rMultiheadAttention hidden_sizeattn LayerNormlayer_norm_epsln_1 ModuleDictrLinearr mlpln_2 attn_maskselfconfig __class__s r7rL%BridgeTowerResidualAttention.__init__is ))&*<*P>PTV>VW LL!3!39N9NO == RYYv'9'96;M;MPQ;QRS023ryy););a)?ASASTU  LL!3!39N9NO r6 hidden_stateattention_maskc Ub(UR[RURS9nURb.URRUR URS9OSUlUR UUUSURUS9S$)NdtypedeviceF) need_weightsrWkey_padding_maskr)tor1boolrbrWrarO)rYr]r^s r7 attention&BridgeTowerResidualAttention.attentionzs  %+..UZZ H[H[.\N~~) NN  L$6$6|?R?R  S  yy   nn+     r6cXRURU5U5-nURU5nURR 5H nU"U5nM X1-nU$N)rgrRrVrUvalues)rYr]r^residual_statelayers r7forward$BridgeTowerResidualAttention.forwardsZ%tyy7NP^(__yy0 XX__&E .L'%4 r6)rOrWrRrVrUrj) r,r-r.r/rLr1Tensorrgrrnr5 __classcell__r[s@r7rArAhsH" ell ELL "ELL(5<[TU]5 URUlURUlUR(aL[ R "[URS- 5Vs/sHn[U5PM sn5Ul OH[ R "[UR5Vs/sHn[U5PM sn5Ul URUl gs snfs snf)Nr) rKrLrNnum_hidden_layersremove_last_layerr ModuleListrangerA resblocks stop_gradientrYrZ_r[s r7rLBridgeTowerTransformer.__init__s !--!'!9!9  # #]]?DTE[E[^_E_?`a?`!-f5?`aDN ]]?DTE[E[?\]?\!-f5?\]DN$11 b^s -C)6C.r]r^c/nURHNnU"X5nUR(a!URUR55 M=URU5 MP U$rj)r{r|appenddetach)rYr]r^r)blocks r7rnBridgeTowerTransformer.forwardsU ^^E >L!!$$\%8%8%:;$$\2 $ r6)rNrwr{r|rj) r,r-r.r/rLr1rprrnr5rqrrs@r7rtrts- 2ELL(5<[TU]5 XlURUlUR UlUR Ul[R"[R"UR55Ul [R"URURUR UR SS9UlUR UR -S-UlURS-Ul[R""UR UR5UlUR'S[R("UR 5R+S5SS9 g)NF) in_channels out_channels kernel_sizestridebiasr position_idsr persistent)rKrLrZrN embed_dim image_size patch_sizer Parameterr1randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferarangeexpandrXs r7rL$BridgeTowerVisionEmbeddings.__init__s  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr6 embeddingsheightwidthreturncURSS- nURRRS5nURSS- n[R R 5(d%XF:Xa X#:XaURUR5$USS2SS24nUSS2SS24nURSn X R-n X0R-n [US-5n URSXU 5nURSSSS5n[RRUX4SS S 9nURSSSS5RSSU 5n[R "Xx4SS 9$) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr?r rbicubicF)sizemode align_cornersdim)shaperweight unsqueezer1jit is_tracingrrrreshapepermuter functional interpolateviewcat) rYrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r7interpolate_pos_encoding4BridgeTowerVisionEmbeddings.interpolate_pos_encodingsi!&&q)A- !44;;EEaH*003a7 yy##%%+*F6?**4+<+<= =,QU3,QU3r". __, &}c'9:)11!5G]`a)11!Q1=--33 ( 4 *11!Q1=BB1b#Nyy/;CCr6 pixel_valuesc ^URup4pVU(dJXPR:wdX`R:wa,[SUSUSURSURS3 5eURRR nURUR US95nURS5RSS5nURRUSS5n [R"X/SS 9n U(aXRXU5-n U $XRUR5-n U $) NzInput image size (*z) doesn't match model (z).rarrrr)rr ValueErrorrrrareflatten transposerrr1rrrr) rYrr batch_sizer~rr target_dtype patch_embeds class_embedsrs r7rn#BridgeTowerVisionEmbeddings.forwards$'3'9'9$ v'V-F%SbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY ;C ##&C&CJX]&^^J$&=&=d>O>O&PPJr6) rrZrrrrrrrF)r,r-r.r/r rLr1rpintrr2rnr5rqrrs@r7rrsjq6q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr6rc^\rSrSrU4SjrS S\R S\4SjjrS S\R S\4Sjjr S\R 4Sjr S r U=r $) BridgeTowerVisionTransformeric 6>[TU]5 [U5Ul[R "UR URS9Ul[U5Ul [R "UR URS9Ul URUl UR(dg[R"[UR5Vs/sH,n[R "UR URS9PM. sn5Ulggs snfNrE)rKrLrrrrPrNrQln_prert transformerln_postshare_layernormryrzrw ln_separater}s r7rL%BridgeTowerVisionTransformer.__init__s 5f=ll6#5#56;P;PQ 1&9||F$6$6F[TU]5 URUlURUlURS;aURS:Xa0[R "[ R"S55UlO?URS:Xa/[R "[ R"S55Ul [R"URURS9Ul g[SURS35e) N)add scaled_addrr?rrrElink_tower_type  is not implemented) rKrLlink_tower_typerNrrr1tensor scaled_factorbetarPrQNotImplementedErrorrXs r7rLBridgeTowerLinkTower.__init__=s %55!--  ! !%I I%%5%'\\%,,s2C%D"''=8LLc):; \\$*:*:@U@UVDN%(89O9O8PPc&de er6cZURS:XaURX-5$URS:Xa URXR-U-5$URS:Xa0URUSUR- -X R--5$[ SURS35e)Nrrrrrr)rrPrrr)rYr)cross_modal_hidden_statesr^s r7rnBridgeTowerLinkTower.forwardJs   5 (>>-"KL L  ! !\ 1>>-2D2D"DG`"`a a  ! !] 2>>-1tyy="AD]`i`iDi"ij j%(89M9M8NNa&bc cr6)rPrrNrrr,r-r.r/rLrnr5rqrrs@r7rr<s fddr6rcz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BridgeTowerSelfOutputiVc(>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"UR5Ul gr) rKrLrrTrNdenserPrQDropouthidden_dropout_probdropoutrXs r7rLBridgeTowerSelfOutput.__init__Ws` YYv1163E3EF f&8&8f>S>STzz&"<"<= r6r) input_tensorrcpURU5nURU5nURX-5nU$rjrrrPrYr)rs r7rnBridgeTowerSelfOutput.forward]5 =1  ]3 }'CD r6rPrr r,r-r.r/rLr1rprnr5rqrrs@r7rrV6> U\\RWR^R^r6rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BridgeTowerIntermediateiec>[TU]5 [R"URUR 5Ul[UR[5(a[URUl gURUl grj) rKrLrrTrNintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrXs r7rL BridgeTowerIntermediate.__init__fs` YYv1163K3KL f'' - -'-f.?.?'@D $'-'8'8D $r6r)rcJURU5nURU5nU$rjrrrYr)s r7rnBridgeTowerIntermediate.forwardns& =1 00? r6rr rrs@r7r r es(9U\\ellr6r cz^\rSrSrU4SjrS\R S\R S\R 4SjrSrU=r $)BridgeTowerOutputiuc(>[TU]5 [R"URUR 5Ul[R"UR URS9Ul[R"UR5Ul gr) rKrLrrTrrNrrPrQrrrrXs r7rLBridgeTowerOutput.__init__vs` YYv779K9KL f&8&8f>S>STzz&"<"<= r6r)rrcpURU5nURU5nURX-5nU$rjrrs r7rnBridgeTowerOutput.forward|rr6r r rrs@r7rrur r6rcb^\rSrSrU4SjrS\R S\R 4SjrSrU=r $)BridgeTowerPooleric>[TU]5 [R"URUR5Ul[R "5Ulgrj)rKrLrrTrNrTanh activationrXs r7rLBridgeTowerPooler.__init__s9 YYv1163E3EF '')r6r)rc\USS2S4nURU5nURU5nU$)Nr)rr#)rYr)first_token_tensor pooled_outputs r7rnBridgeTowerPooler.forwards6+1a40 #56  6 r6)r#rr rrs@r7r r s($ U\\ellr6r c^\rSrSrSU4Sjjr\"SSSS9SS\RS\\RS \\RS \\RS\\ S \\ S \\RS \ \R4Sjj5r SrU=r$)BridgeTowerSelfAttentionic>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eURUl[ URUR- 5UlURUR-Ul[R"URUR5Ul [R"URUR5Ul [R"URUR5Ul [R"UR5UlU=(d [#USS5UlUR$S:XdUR$S :XaGUR&Ul[R("S UR&-S - UR5UlUR,UlX0lg) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryrr)rKrLrNnum_attention_headshasattrrrattention_head_size all_head_sizerrTquerykeyvaluerattention_probs_dropout_probrgetattrr.max_position_embeddingsrdistance_embedding is_decoder layer_idxrYrZr.r>r[s r7rL!BridgeTowerSelfAttention.__init__s     : : :a ?PVXhHiHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F( ' -zC $  ' '> 9T=Y=Y]q=q+1+I+ID (&(ll1v7U7U3UXY3Y[_[s[s&tD # ++"r6past_key_valuepast_key_values4.58new_nameversionr)r^ head_maskencoder_hidden_statesoutput_attentionscache_positionrc URupn URU5n U RUSURUR5R SS5n Sn USLn Ub][ U[5(aFURRUR5n U (a URnOURnOUnU (aUOUnU (aQUbNU (aGWRURRnURURRnOUR!U5nURUSURUR5R SS5nUR#U5nURUSURUR5R SS5nUbcU (dUOSnWR%UUURSU05unnU (a.[ U[5(aSURUR'[&R("U UR SS55nUR*S:XdUR*S :XGaU RSURSnnUbB[&R,"US- [&R.UR0S 9RSS5nO>[&R2"U[&R.UR0S 9RSS5n[&R2"U[&R.UR0S 9RSS5nUU- nUR5UUR6-S- 5nUR9U R:S 9nUR*S:Xa[&R<"S U U5nUU-nOHUR*S :Xa8[&R<"S U U5n[&R<"S UU5nUU-U-nU[>R@"UR5- nUbUU-n[BRDRGUSS9nURIU5nUbUU-n[&R("UU5nURKSSSS5RM5nURO5SSURP4-nURU5nUU4$)NrrrFrJTr0r1r`rzbhld,lrd->bhlrzbhrd,lrd->bhlrrrr ))rr6rr2r4rrr is_updatedgetr>cross_attention_cacheself_attention_cachelayerskeysrkr7r8updater1matmulr.rlongrbrr<r;reraeinsummathsqrtrrsoftmaxrr contiguousrr5)rYr)r^rGrHrBrIrJr seq_lengthr~ query_layerrMis_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r7rn BridgeTowerSelfAttention.forwardsa%2$7$7! jj/ !&&z2t7O7OQUQiQijtt q  2$>  &/+>??,77;;DNNK %*9*O*O'*9*N*N'&5#2D.- /"=*+224>>BGGI-44T^^DKKK0I!z2t7O7OQUQiQijtt1I**^4K%**B 8 8$:R:Ri1o *7It)<)C)C{DNN=M~<^*& ;&*_FY*Z*ZAEO..t~~>!<< Y5H5HR5PQ  ' '> 9T=Y=Y]q=q'2'8'8';Y__Q=O*L*!&j1nEJJWdWkWk!l!q!q""'l%**UbUiUi!j!o!oprtu!v"\\*EJJ}OcOcdiijkmopN%6H#'#:#:8dFbFb;bef;f#g #7#:#:ARAR#:#S ++~=+0<<8H+Wk+l(#36N#N --1EE16>NP[]q1r./4||S"S%**+BC o--r6) r5r4r<rr=r7r>r;r2r.r6r8NNNNNNFN)r,r-r.r/rLrr1rprr2r rfr4rnr5rqrrs@r7r*r*s#6%0A6R7;15=A+/,115e.||e.!!2!23e.E--. e. ((9(9: e. "% e.$D>e.!.e. u|| e.Se.r6r*eagerc$^\rSrSrSU4SjjrSr\"SSSS9SS\RS \ \RS \ \RS \ \RS\ \ S \ \ S \ \RS\ \R4Sjj5rSrU=r$)BridgeTowerAttentionic>[TU]5 [UR"UUUS9Ul[ U5Ul[5Ulg)Nr.r>) rKrL#BRIDGE_TOWER_SELF_ATTENTION_CLASSES_attn_implementationrYroutputset pruned_headsr?s r7rLBridgeTowerAttention.__init__ sF 78S8ST $;  ,F3 Er6c6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrr)lenrrYr2r4r{rr6r7r8ryrr5union)rYheadsindexs r7 prune_heads BridgeTowerAttention.prune_heads*s  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:r6rArBrCrDr)r^rGrHrIrJrc lURUUUUUUUS9nURUSU5n U 4USS-n U $)Nr^rGrHrBrIrJrr)rYry) rYr)r^rGrHrBrIrJ self_outputsattention_outputoutputss r7rnBridgeTowerAttention.forward<s\yy )"7+/)!  ;;|A F#% QR(88r6)ryr{rYrprq)r,r-r.r/rLrrr1rprr2r rfr4rnr5rqrrs@r7rtrts";$%0A6R7;15=A+/,115||!!2!23E--.  ((9(9:  "% $D>!. u|| Sr6rtc^^\rSrSrS U4Sjjr\"SSSS9S Sj5rSrS rU=r $) BridgeTowerBertCrossLayeriUc>[TU]5 URUlSUl[ XS9UlUR UlURUl[ XS9Ul[U5Ul [U5Ul g)Nrr>) rKrLchunk_size_feed_forward seq_len_dimrtrgr=add_cross_attentioncrossattentionr  intermediaterryrYrZr>r[s r7rL"BridgeTowerBertCrossLayer.__init__Vsq '-'E'E$-fJ ++#)#=#= 26O3F;'/ r6rArBrCrDc URUUSUSS9n U Sn U SSn URU UUUUUUS9n U Sn XSS-n [URURUR U 5n U 4U -n U $)N)r^rGrIrBrrr)rgrrfeed_forward_chunkrr)rYr)rHr^rGencoder_attention_maskrBrIrJself_attention_outputsrrcross_attention_outputs layer_outputs r7rn!BridgeTowerBertCrossLayer.forwardas"& )/ "0" 2!4),"&"5"5 1"7+/)#6# 315AB770  # #T%A%A4CSCSUe  /G+r6cJURU5nURX!5nU$rjrryrYrintermediate_outputrs r7r,BridgeTowerBertCrossLayer.feed_forward_chunk)"//0@A{{#6I r6rrgrrrr=ryrrjrq) r,r-r.r/rLrrnrr5rqrrs@r7rrUsG 0%0A6R #+S+Zr6rcD^\rSrSrSU4Sjjr\"SSSS9SS\RS\\RS \\RS \\RS \\RS\\ S \\ S \\RS\ \R4Sjj5r SrSrU=r$)BridgeTowerTextLayericr>[TU]5 URUlSUl[ XS9UlUR UlURUlUR(a/UR (d[US35e[ USUS9Ul [U5Ul [U5Ul g)Nrrz> should be used as a decoder model if cross attention is addedr/rv)rKrLrrrtrgr=rrrr rrryrs r7rLBridgeTowerTextLayer.__init__s '-'E'E$-fJ ++#)#=#=  # #?? D6)g!hii"6vWamv"wD 3F;'/ r6rArBrCrDr)r^rGrHrrIrJrc zURUUUUUUS9n U Sn UR(aU SSn OU SSn UR(aDUbA[US5(d[SUS35eUR U UUUUUUS9n U Sn XSS-n [ UR URURU 5n U 4U -$) N)r^rGrIrBrJrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rgr=r3rrrrrr)rYr)r^rGrHrrBrIrJrrrrrs r7rnBridgeTowerTextLayer.forwards"& )/+) "0" 2!4 ??,Qr2G,QR0G ??4@4!122 =dVDDD '+&9&9 5#&; /"3-':' # 7q9 " ==G0  # #T%A%A4CSCSUe ((r6cJURU5nURX!5nU$rjrrs r7r'BridgeTowerTextLayer.feed_forward_chunkrr6rrj)NNNNNFN)r,r-r.r/rLrr1rprr2r rfr4rnrr5rqrrs@r7rrs 0%0A6R7;15=A>B+/,1152)||2)!!2!232)E--. 2) ((9(9: 2) !)):): ; 2)"%2)$D>2)!.2) u|| 2)S2)hr6rcV^\rSrSrSU4SjjrSS\R S\\RS\\RS\\RS\\RS\\ S \\ S \\ S \\ S \\ S \\R S\ \ \R \ 44SjjrSrU=r$)BridgeTowerTextEncoderic >[TU]5 Xl[R"[ UR 5Vs/sH n[XS9PM sn5UlSUl gs snf)NrF) rKrLrZrryrzrwrrmgradient_checkpointing)rYrZr>ir[s r7rLBridgeTowerTextEncoder.__init__sX  ]]@EfF^F^@_ `@_1 !& 6@_ `  ',# asA$r)r^rGrHrrB use_cacherIoutput_hidden_states return_dictrJrc U (aSOSn U(aSOSn U(aURR(aSOSnUR(a/UR(aU(a[R S5 SnU(aLURR (a1Uc.[[URS9[URS95nU(a[URR (a@[U[5(a+[R S5 [R"U5n[UR5HeunnU (aX4-n UbX?OSnU"UUUUUUUU S9nUSnU(dM6U US4-n URR(dM\UUS 4-nMg U (aX4-n U (d[S UUU U U455$[UUU U US 9$) Nr+zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rZzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.)rrBrIrJrrrc30# UH nUcMUv M g7frjr+.0vs r7 1BridgeTowerTextEncoder.forward..(s" A )last_hidden_staterBr)r*cross_attentions)rZrrtraininglogger warning_oncer=rr rr4from_legacy_cache enumeratermr)rYr)r^rGrHrrBrrIrrrJall_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputss r7rnBridgeTowerTextEncoder.forwards#7BD$5b4%64;;;Z;Zr`d  & &4==##p" //O4K1,dkk2RT`hlhshsTtuO //JPU4V4V   \  2CCOTO(4OA|#$58H$H!.7.CilO(%'= /"3- M*!,M  &9]1=MB+/$(,1/4&*15P ||P !!2!23P E--. P ((9(9: P !)):): ; P "%P D>P $D>P 'tnP d^P !.P  uU\\"$MM NP P r6rc>^\rSrSrSrU4SjrSSjrSrSrU=r $)BridgeTowerTextEmbeddingsi=zN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. c>[TU]5 [R"URUR UR S9Ul[R"URUR 5Ul [R"URUR 5Ul [R"UR URS9Ul [R"UR5Ul[#USS5UlUR'S[(R*"UR5R-S5SS9 UR'S [(R."UR0R35[(R4S 9SS9 UR Ul[R"URUR UR6S9Ul g) N) padding_idxrEr.r/rrFrtoken_type_idsr)rKrLrr vocab_sizerN pad_token_idword_embeddingsr;position_embeddingstype_vocab_sizetoken_type_embeddingsrPrQrrrr:r.rr1rrzerosrrrUrrXs r7rL"BridgeTowerTextEmbeddings.__init__Csi !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  r6cUc+Ub[XRU5nOURU5nUbUR5nOUR5SSnUSnUcv[ US5(a-UR SS2SU24nUR USU5n U nO8[R"U[RURRS9nUcURU5nURU5n XJ-n URS:XaURU5n X- n UR!U 5n UR#U 5n U $)Nrrrrr`r/)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrr3rrr1rrUrrbrrr.rrPr) rY input_idsrr inputs_embedspast_key_values_length input_shaper[buffered_token_type_ids buffered_token_type_ids_expandedrrrs r7rn!BridgeTowerTextEmbeddings.forward\sM  $A)M]M]_uv #JJ=Y  #..*K',,.s3K ^  !t-..*.*=*=a*n*M'3J3Q3QR]^_R`bl3m0!A!&[ SWSdSdSkSk!l   00;M $ : :> J":  ' ': 5"&":":<"H   -J^^J/ \\*- r6cUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nrrr`r)rr1rrrUrbrr)rYrrsequence_lengthrs r7r@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <RR;URR@5 [U[R0[B45(a3UR8b%UR8RR5 ggg)Ngr)stdgg?)meanrr)"rZinitializer_factorrrrNrwrr{rinitnormal_rOin_proj_weight in_proj_biasdatazero_out_projrrUrGrJrrrrTrrrPrfill_!BridgeTowerForContrastiveLearning logit_scalelogit_scale_init_valueBridgeTowerMLMHead)rYrrproj_stdattn_stdfc_stdrs r7 _init_weights(BridgeTowerPreTrainedModel._init_weightsszkk,, f: ; ; //51t{{?\?\;\ae:efH{{..4H$++111d:F++55 9 9x~N '',,224 3 3 : :O  5 56<H 0 0 7 7X^L 6 GGOOF--==8>O R GGOOF--@@GGX^O \ BIIr|| D E E MM   & &CTCZ & @  - - KK   " " $ MM   $ $S )  A B B    # # ) )$++*L*L M fryy*<= > >6;;CZ KK   " " $D[ >r6r+N)r,r-r.r/rr3base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementrModulerr5r+r6r7rrs6 %&+#35ST"3%BII%r6rcN^\rSrSr%\\S'U4Sjr\S5rSSjr Sr U=r $)BridgeTowerVisionModelirZcD>[TU]U5 [U5Ulgrj)rKrLrvisualrXs r7rLBridgeTowerVisionModel.__init__s  26: r6cjURRRRR$rj)r rrrrarYs r7raBridgeTowerVisionModel.dtypes${{%%55<<BBBr6cXURURUR5X#5$rj)r typera)rYimage image_maskrs r7rnBridgeTowerVisionModel.forwards {{5::djj1:XXr6)r )NF) r,r-r.r/r r3rLpropertyrarnr5rqrrs@r7r r s0 ##;CCYYr6r a0 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 c"^\rSrSr%\\S'SU4SjjrSrSrSr \ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ RS\ \S\ \S\ \S\ \S\ \S\ \ RS\\\ R\44Sjj5rSrU=r$)BridgeTowerTextModelirZc>[TU]U5 Xl[U5Ul[ U5UlU(a [U5OSUlUR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rKrLrZrrrencoderr pooler post_init)rYrZadd_pooling_layerr[s r7rLBridgeTowerTextModel.__init__sL   3F;-f5 3D'/$  r6c.URR$rjrrrs r7get_input_embeddings)BridgeTowerTextModel.get_input_embeddingss...r6c$XRlgrjrrYr8s r7set_input_embeddings)BridgeTowerTextModel.set_input_embeddingss */'r6cUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrmrgr)rYheads_to_prunermrs r7 _prune_heads!BridgeTowerTextModel._prune_headss< +002LE LL  u % / / ; ;E B3r6rr^rrrGrrHrrBrrIrrrJrc U bU OURRn U bU OURRn U bU OURRn URR(aU bU OURR n OSn UbUb [ S5eUb"URX5 UR5nO"UbUR5SSnO [ S5eUunnUb URO URnSnU b:[U [5(dU SSRSOU R5nUc[R"UUU-4US9nUcs[!UR"S5(a4UR"R$SS2SU24nUR'UU5nUnO$[R("U[R*US 9nUR-X/5nURR(aEUbBUR5unnnUU4nUc[R"UUS9nUR/U5nOSnUR1XPRR25nUR#UUUUUS 9nUR5UUUUUU U U U U US 9 nUSnUR6bUR7U5OSnU (d UU4US S-$[9UUUR:UR<UR>UR@S 9$)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrrLrbrr`)rrrrr) r^rGrHrrBrrIrrrJr)rr(rBr)r*r)!rZrIruse_return_dictr=rr%warn_if_padding_and_no_attention_maskrrbrr rget_seq_lengthr1onesr3rrrrrUget_extended_attention_maskinvert_attention_mask get_head_maskrwrrrrBr)r*r) rYrr^rrrGrrHrrBrrIrrrJrrr[rbrrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthr~encoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputr's r7rnBridgeTowerTextModel.forwards3$2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T!"  &"/599 "1%++B/$335 #  !"ZZ*jCY6Y)ZdjkN  !t(899*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z150P0PQ_0m ;; ! !&;&G=R=W=W=Y :  7$68O#P %-).4HQW)X&.2.H.HI_.` +.2 +&&y++2O2OP ??%)'#9 + ,, 2"7#B+/!5#)'  *!,8< 8O O4UY #]3oab6II I;-'+;;)77&11,==   r6)rZrrr)T)NNNNNNNNNNNNNN)r,r-r.r/rr3rLr r$r)rrr1rpr rfrr4rrnr5rqrrs@r7rrs~ "! /0C-11515/3,0048<9=+/$(,0/3&*15s ELL)s !.s !. s u||, s ELL) s   -s  ( 5s !) 6s "%s D>s $D>s 'tns d^s !.s uU\\"$PP Q!s s r6rzv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on c"^\rSrSrU4SjrSrSr\SS\\ RS\\ RS\\ RS\\ RS \\ RS \\ RS \\ RS \\ RS \\ S\\ S\\ S\\ S\\ RS\ S\\\ R \44Sjj5rSrSrU=r$)BridgeTowerModelic  >[TU]U5 XlURnURnUR (aa[ R"URUR5Ul [ R"URUR5Ul O[ R"[UR5Vs/sH.n[ R"URUR5PM0 sn5Ul [ R"[UR5Vs/sH.n[ R"URUR5PM0 sn5Ul [ R"SUR5Ul[!U5Ul[%U5UlUR((dUR*(aUR"R,R.HnUR"R,R0R2R4UR2lUR"R,R0R6R4UR6lM [ R"[UR5Vs/sH n[9X6S9PM sn5Ul[ R"[UR5Vs/sH n[9X6S9PM sn5Ul[?U5Ul [?U5Ul![ RD"URURFS9Ul$[ RD"URURFS9Ul%URL(a![OU5Ul([OU5Ul)O[ R"[URS- 5Vs/sHn[OU5PM sn5Ul([ R"[URS- 5Vs/sHn[OU5PM sn5Ul)URU5 gs snfs snfs snfs snfs snfs snf)NrrrEr)+rKrLrZ vision_config text_config$share_cross_modal_transformer_layersrrTrNcross_modal_text_transformcross_modal_image_transformryrzrwrrr  vision_modelr text_modelr"init_layernorm_from_vision_encoderr cross_modal_ln_separaterrrrrcross_modal_image_layerscross_modal_text_layersr cross_modal_image_poolercross_modal_text_poolerrPrQcross_modal_text_layernormcross_modal_image_layernormshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerr)rYrZr@rAr~rrr[s r7rLBridgeTowerModel.__init__sW   ,, ((  6 6.0ii 8O8OQWQcQc.dD +/1yy9R9RTZTfTf/gD ,.0mmQVW]WoWoQpqQpA;22F4F4FGQpq/D +02}}SXY_YqYqSrsSra=44f6H6HISrs0D ,&(\\!V5G5G%H"2=A.{;,,1Z1Z''..FF!%!2!2!9!9!A!A!H!H!M!M #0077??DDII G)+ JOPVPhPhJi jJiQ &{ @Ji j) %(*}}JOPVPhPhJi jJiQ &{ @Ji j( $ ):&(A%'8'@$+-,,v7I7IvOdOd*e'+-<<8J8JPVPePe+f(  ) )/CF/KD ,0DV0LD -/1}}7image_token_type_idxrIrrlabelsrrc . U bU OURRn U bU OURRn U (aSOSnU (aSOSnU (aSOSnU (aSOSnU (aSOSnUbUc [S5eU bU OURRn U (aU OSn UR 5nUR RUS9nU (aUU4- nUc.[R"U[RURS9nUR RUU5RUR5n[UR RR 5URR"- S-nUR RR SUHnU"UU5SnU (dMUU4- nM UcHUR$R&R)UR+UR$R,5US9nOUR/SSS 5nU (aUU4- nUR$R&R0R2SUHnU"U5nU (dMUU4- nM UR$R&R5UR+UR$R,55nUR7U5nUR9[R:"S[RURS95R=U5nUR?UU-5nURAU5nUR9[RB"S U [RURS95R=U5nUU-nUREU5n[R"UR S5UR S54[RURS9nUR RXUR 55RUR5nURFS"UUUUU S 9n U Sn!URHS"UUUUU S 9n"U"Sn#U (aUU!U#44- nU (aUU SU"S44- nSn$[KU[UR RR 55GHxn%UR RR U%"UU5SnUR$R&R0R2U%"U5R+UR$R,5nURAUR$R&R5U55U-nURLU$n&URNU$n'U&"UR7U5U-U!U5n(U'"UU#U5n)URFU$S-"U(U)UUU S 9n U Sn!URHU$S-"U)U(UUU S 9n"U"Sn#U$S- n$U (aUU4- nUU4- nUU!U#44- nU (dGMjUU SU"S44- nGM{ U!U#n+n*URQU*U+5n,U (aUUU4nU (d[SS U*U+U,UU455$[UU*U+U,UUS 9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. output_hidden_states (`bool`, *optional*): If set to `True`, hidden states are returned as a list containing the hidden states of text, image, and cross-modal components respectively. i.e. `(hidden_states_text, hidden_states_image, hidden_states_cross_modal)` where each element is a list of the hidden states of the corresponding modality. `hidden_states_txt/img` are a list of tensors corresponding to unimodal hidden states and `hidden_states_cross_modal` is a list of tuples containing `cross_modal_text_hidden_states` and `cross_modal_image_hidden_states` of each brdige layer. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```Nr+zYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.r)rr`rrrr)r^rrIc30# UH nUcMUv M g7frjr+rs r7r+BridgeTowerModel.forward..snAnr)r&r'r(r)r*)+rZrIrrr-rrFrr1r0rUrbr1rer~rrmrwrEr rrrarrr{rrCrr expand_asrMrDfullrNrJrIrzrPrQget_cls_featuresr4r$)-rYrr^rrrWrGrr>rXrIrrrYrall_hidden_states_textall_hidden_states_imageall_hidden_states_crossrrrr=extend_text_masks split_indexrmrimage_embeds_with_lncross_modal_texttext_token_type_embeddingsimage_token_type_embeddingscross_modal_imageextend_image_maskslayer_outputs_textcross_text_featureslayer_outputs_imagecross_image_featureslink_layer_indexrtext_link_towerimage_link_towercross_text_features_cross_image_features_r&r' cls_featuress- r7rnBridgeTowerModel.forwardsj2C1N-TXT_T_TqTq$8$D $++JjJj (<(<"$(<"$"6BD$5b4  $):%k &1%?BK##&;.8& A  ,,33??!!$"3"3"9"9:Um@L (//1a8L  #  6 #&&--99CCL[QE .L##'L?:'R $0077DD\EVEVW[WhWhWnWnEop ::;G%)%?%? KKI4D4D E& )$ % # ::;KNh;hi#??@TU&*&@&@ JJt1IL\L\ ]' )( ) $ 46QQ <<=QRZZ  # #A &(9(>(>q(A B**## "__HHUdUdUfgjj    "99!<  ,#5/  13";;A>  -#4/   315  #)<>R(S'U U #  %7%:>?OPO#@@AQR $3// NQR>R!S$%0'9"3 " #5Q"7 "&"?"?@PST@T"U%$1'8"3 # $7q#9  ! #&;.8&'L?:''-@BV,W+YY'  #);A)>@STU@V(W'YY#aHf)<=Q~ ,,]NK !79PRi j 'GXZmn  &')&+*   r6crURU5nURU5n[R"X4/SS9$)Nrr)rLrKr1r)rYr&r'cls_features_textcls_features_images r7r`!BridgeTowerModel.get_cls_featuress9 88G!::>Jyy+@bIIr6)rZrNrIrQrKrDrMrJrPrLrCrFrrE)NNNNNNNNNNNNNF)r,r-r.r/rLr r$rrr1 LongTensorr2rrfrr4rpr$rnr`r5rqrrs@r7r>r>s 6p64156:594815155948.2,0/3&*-1).j E,,-j !!2!23j !!1!12 j u001 j U--. j E--.j   1 12j u001j 'smj $D>j 'tnj d^j ))*j #'j uU\\"$:: ;!j j XJJr6r>c.^\rSrSrU4SjrSrSrU=r$)"BridgeTowerPredictionHeadTransformicp>[TU]5 [R"URUR5Ul[ UR[5(a[URUl OURUl [R"URURS9Ul gr) rKrLrrTrNrrrrr transform_act_fnrPrQrXs r7rL+BridgeTowerPredictionHeadTransform.__init__s~ YYv1163E3EF f'' - -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STr6clURU5nURU5nURU5nU$rj)rrrPrs r7rn*BridgeTowerPredictionHeadTransform.forwards4 =1 --m< }5 r6)rPrrrrrs@r7r}r}sUr6r}c2^\rSrSrSU4SjjrSrSrU=r$)ricn>[TU]5 Xl[U5Ul[ R "URURRSS9Ul [ R"[R"URR55UlUbX Rlgg)NF)r)rKrLrZr} transformrrTrNrArdecoderrr1rrr)rYrZrr[s r7rLBridgeTowerMLMHead.__init__s  ;FCyy!3!3V5G5G5R5RY^_ LLV-?-?-J-J!KL  "(LL  r6cdURU5nURU5UR-nU$rj)rrr)rYx mlm_scores r7rnBridgeTowerMLMHead.forwards-NN1% LL+dii7 r6)rrZrrrjrrrs@r7rrs)r6rc.^\rSrSrU4SjrSrSrU=r$)BridgeTowerITMHeadicZ>[TU]5 [R"US5UlgNrrKrLrrTfc)rYrNr[s r7rLBridgeTowerITMHead.__init__s  ))K+r6c(URU5nU$rjr)rYr itm_scores r7rnBridgeTowerITMHead.forwardsGGAJ r6rrrrs@r7rrs,r6rz\ BridgeTower Model with a language modeling head on top as done during pretraining. c^\rSrSrS/rU4SjrSrSr\SS\ \ RS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ S\ \ S\ \ S\ \ RS\\\\ R44Sjj5rSrU=r$)BridgeTowerForMaskedLMizmlm_score.decoder.weightc>[TU]U5 [U5Ul[ U5UlUR 5 grj)rKrLr>rrrrrXs r7rLBridgeTowerForMaskedLM.__init__s5  +F3+F3 r6c.URR$rjrrrs r7get_output_embeddings,BridgeTowerForMaskedLM.get_output_embeddingss~~%%%r6c$XRlgrjr)rYnew_embeddingss r7set_output_embeddings,BridgeTowerForMaskedLM.set_output_embeddingss !/r6rr^rrrWrGrr>rIrrrYrc U bU OURRn URUUUUUUUUU U U S9 n URU (a U ROU S5nSnU bk[ 5nU R UR5n U"URSURRR5U RS55nU (d[U5nUbU4U-$U$[UUU RU RS9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```N r^rrrWrGrr>rIrrrrr;r<r)r*)rZr-rrr&rrerbrrArr4rr)r*)rYrr^rrrWrGrr>rIrrrYr mlm_logitsmasked_lm_lossloss_fctrys r7rnBridgeTowerForMaskedLM.forwardsd&1%Q 'tnQ d^Q ))*Q  ~uU%6%677 8Q Q r6rz BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. c^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\RS\ \ \ \R44Sjj5rSrU=r$)#BridgeTowerForImageAndTextRetrievaliNc>[TU]U5 [U5Ul[ UR S-5UlUR5 gr)rKrLr>rrrNrrrXs r7rL,BridgeTowerForImageAndTextRetrieval.__init__Us@  +F3+F,>,>,BC r6rr^rrrWrGrr>rIrrrYrc U bU OURRn URUUUUUUUUU U U S9 n U (a U ROU SnUR U5nSnU b-[ 5nU R UR5n U"X5nU (d[U5nUbU4U-$U$[UUU RU RS9$)aM image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```Nrrr) rZr-rr(rrrerbr4rr)r*)rYrr^rrrWrGrr>rIrrrYrr(r<itm_lossrrys r7rn+BridgeTowerForImageAndTextRetrieval.forward_s\&1%Q 'tnQ d^Q ))*Q  'u/@/@)AA BQ Q r6rc.^\rSrSrU4SjrSrSrU=r$)BridgeTowerContrastiveHeadicX>[TU]5 [R"X5Ulgrjr)rYrN embed_sizer[s r7rL#BridgeTowerContrastiveHead.__init__s ))K4r6c(URU5nU$rjr)rYrs r7rn"BridgeTowerContrastiveHead.forwards GGAJr6rrrrs@r7rrs5r6rzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. c^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\ \ \ \R44Sjj5rSrU=r$)ric>[TU]U5 [U5Ul[ UR UR 5Ul[ UR UR 5Ul[ UR S-UR 5Ul [R"[R"URR55UlUR#5 gr)rKrLr>rrrNcontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrr1rrZrrrrXs r7rL*BridgeTowerForContrastiveLearning.__init__s  +F378J8JFLjLjk89K9KVMkMkl$>v?Q?QTU?UW]WuWu$v!<< T[[5W5W(XY r6rr^rrrWrGrr>rIrr return_lossrc >U bU OURRn URUUUUUUUUU SU S9 n U (a U ROU SnU (a U ROU SunnnUSnUSnURR R RU5nURR[R"SS[RURRRRS 95RU5nURRU5U-n[ R"R%UR'USS2S SS245SSS 9n[ R"R%UR)USS2S SS245SSS 9R+URS 9n[ R"R%UR-U5SSS 9R+URS 9n[R."UUU/S S9nUR0R35R+URS 9n[R4"UUR755U-n[R4"UUR755U-n[R4"UUR755U-nSnU (a[R8"[;U5URS 9n[ R"R=UU5n[ R"R=UU5n[ R"R=UU5nUU-U-S- nU (dUUUU4U SS-n UbU4U -$U $[?UUUUUU RU R@S9$)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import requests >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```NTrrr rr[rr`r)rpr,rLrg@)r;r<r=r>r?r)r*)!rZr-rr(r)rEr rrr1r_rUrrbr^rDrr normalizerrrerrrexprTtrr~ cross_entropyr9r*)!rYrr^rrrWrGrr>rIrrrrr(hidden_states_txthidden_states_imghidden_states_cross_modalr=rfrir?r<rlogits_text_to_imagelogits_text_to_crosslogits_image_to_crossitc_lossrYtext_to_image_losstext_to_cross_lossimage_to_cross_lossrys! r7rn)BridgeTowerForContrastiveLearning.forwardszj&1%>;K L{ Z \\#f+fmmDF!#!SU["\ *-??BUUY\\Hk<FQRQSTF-5-AXK&( Mv M+#%%!//))  r6)rrrrr) NNNNNNNNNTNN)r,r-r.r/rLrrr1r{r2rfrr9r4rnr5rqrrs@r7rrsO 156:594815155948,0/3&*&*x E,,-x !!2!23x !!1!12 x u001 x U--. x E--.x   1 12x u001x $D>x 'tnx d^x d^x  +U53D3D-EE Fx x r6r)rrrr>r)r)Pr0rW collectionsr dataclassesrtypingrrr1rtorch.nnr activationsr r cache_utilsr r rmodeling_layersrmodeling_outputsrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrutils.deprecationrconfiguration_bridgetowerrrr get_loggerr,r_TOKENIZER_FOR_DOCr$r9rrArtrrrrr rr r*rwrtrrrrrrr rr>r}rrrrrr__all__r+r6r7rs #!" %6CC9.ll770hh   H %'  :[:  :$  :;:  :4)299)XRYY6P"))Pf7"2997"td299d4 BII  bii      B.ryyB.L %'# 32993l= =@G5GVY RYYY zV= V=t 4 %%%D Y7 Y  U 5U  U p oJ1oJ  oJf "    d 7d  d N ] *D]  ] @ G (BG  G T r6