cCi S r SSKrSSKJr SSKJrJrJr SSK r
SSK r SSK J r SSK
Jr SSKJrJr SS KJr SS
KJrJr SS KJr SS KJrJrJ r J r J r S
SK J r J!r!J"r" \ RF " \$5 r%S\ RL S\ RL 4S jr'S\ RL S\ RL 4S jr(S\ RL S\)4S jr*SDS\ RL S\+S\,S\)S\ RL 4
S jjr-SES jr.S r/ " S S \ R` 5 r1 " S S \ R` 5 r2 " S S \ R` 5 r3\\ " S! S"\5 5 5 r4 " S# S$\ R` 5 r5 " S% S&\ R` 5 r6 " S' S(\ R` 5 r7 " S) S*\ R` 5 r8 " S+ S,\ R` 5 r9 " S- S.\95 r: " S/ S0\ R` 5 r; " S1 S2\5 r<\ " S3 S4\5 5 r= " S5 S6\ R` 5 r> " S7 S8\ R` 5 r? " S9 S:\ R` 5 r@ " S; S<\=5 rA " S= S>\ R` 5 rB " S? S@\=5 rC\ " SA SB\=5 5 rD/ SCQrEg)FzPyTorch GroupViT model. N) dataclass)AnyOptionalUnion)nn )ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstring filter_out_non_signature_kwargslogging torch_int )GroupViTConfigGroupViTTextConfigGroupViTVisionConfiglogitsreturnc [ R R U [ R " [ U 5 U R S95 $ )Ndevice)r
functional
cross_entropytorcharangelenr )r s h/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/groupvit/modeling_groupvit.pycontrastive_lossr$ ' s/
== & &vu||CKPVP]P]/^ __
similarityc X [ U 5 n[ U R 5 5 nX- S- $ )Ng @)r$ t)r& caption_loss
image_losss r#
groupvit_lossr+ , s* #J/L!*,,.1J % ,,r% dimc U R U5 nUR USS9S n[ R " U [ R S9R XS5 nXBR
5 -
U- nU$ )NTkeepdimr
memory_format ?)softmaxmaxr
zeros_likelegacy_contiguous_formatscatter_detach)r r, y_softindexy_hardrets r# hard_softmaxr= 2 sf
^^C
F JJsDJ )! ,E
fE4R4R
S
\
\]`il
mF
==?
"V
+C Jr% tauhardc , [ R R R [ R " SU R
U R S9[ R " SU R
U R S95 nUR U R 5 nX- U- nUR U5 nU( a] UR USS9S n[ R " U [ R S9R X7S5 nXR 5 -
U- n U $ Un U $ )N )r dtyper2 Tr. r r0 )r
distributionsgumbelGumbeltensorr rB sampleshaper3 r4 r5 r6 r7 r8 )
r r> r? r, gumbel_distgumbelsr9 r: r; r< s
r# gumbel_softmaxrK < s %%,,33
S fllC
S fllCK .G 3&G
__S
!F
3
-a0!!&8V8VW``admpq}}&/ J Jr% c X- U R S - S- nX: a4 [ [ R " X$- 5 5 nU R S U- nO3[ [ R " X- 5 5 nU R S U- nU R S nU R S nU R XxXe5 n [
R R XU4SUS9n U $ )a
Args:
attentions (`torch.Tensor`): attention map of shape [batch_size, groups, feat_height*feat_width]
height (`int`): height of the output attention map
width (`int`): width of the output attention map
align_corners (`bool`, *optional*): the `align_corner` argument for `nn.functional.interpolate`.
Returns:
`torch.Tensor`: resized attention map of shape [batch_size, groups, height, width]
?r r bilinearsizemode
align_corners)rH intnproundreshaper r interpolate)
attentionsheightwidthrS scale
feat_width feat_height
batch_sizegroupss r# resize_attention_mapra R s ^z / / 2
2s :E
~%-01
&&q)Z7 "((6>23 %%a(K7
!!!$J
a
F##J PJ **% z + J r% c V / n[ R " 5 SnU Hj nUR SSS5 R 5 nUc UnOX4- n[ UR SSS5 R 5 /UQ76 nUR U5 Ml SSS5 US nU$ ! , ( d f N= f)a
Args:
attentions (`tuple(torch.FloatTensor)`: tuple of attention maps returned by `GroupViTVisionTransformer`
hw_shape (`tuple(int)`): height and width of the output attention map
Returns:
`torch.Tensor`: the attention map of shape [batch_size, groups, height, width]
Nr rM r )r no_gradpermute
contiguousra append)rY hw_shape attn_mapsprev_attn_masks
attn_masks cur_attn_mapfinal_groupings r# get_grouping_from_attentionsrn p s I $J#++Aq!4??AJ &","1">/0G0G1a0P0[0[0]i`hiL \ * %
r]N !
s A3B
B(c 6 ^ \ rS rSrS\4U 4S jjrS rSrU =r$ )GroupViTCrossAttentionLayer configc > [ TU ] 5 [ U5 U l [ R
" UR UR S9U l [ U5 U l
[ R
" UR UR S9U l g Neps) super__init__GroupViTAttentionattnr LayerNorm hidden_sizelayer_norm_epsnorm2 GroupViTMLPmlp norm_postselfrr __class__s r# rx $GroupViTCrossAttentionLayer.__init__ sb
%f- \\&"4"4&:O:OP
v& f&8&8f>S>STr% c UnX0R XS9S - nX0R U R U5 5 - nU R U5 nU$ )N)encoder_hidden_statesr rz r r~ r )r querykeyxs r# forward#GroupViTCrossAttentionLayer.forward sK
% ;A> >
A ' ' NN1 r% r ) __name__
__module__ __qualname____firstlineno__r rx r __static_attributes__
__classcell__r s @r# rp rp s U 3 U r% rp c @ ^ \ rS rSrS\4U 4S jjrSS jrS rSrU =r $ )GroupViTAssignAttention rr c > [ TU ] 5 UR S- U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l [ R
" UR UR 5 U l UR U l
g )N ) rw rx r| r\ r Linearq_projk_projv_projproj
assign_epsr s r# rx GroupViTAssignAttention.__init__ s
''-
ii 2 2F4F4FG ii 2 2F4F4FG ii 2 2F4F4FG IIf 0 0&2D2DE ++r% c U( a U R ( a
[ USUS9nU$ U( a [ USS9nU$ [ R R USS9nU$ )N)r, r? r, )trainingrK r= r r r3 )r rz rD r? s r# get_attn GroupViTAssignAttention.get_attn sX dmm!$BT:D #Db1 }},,Tr,: r% c ` UnU R U5 nU R U5 nU R U5 nXR SS5 - U R - nU R U5 nU R USSS9nXUR
SSS9U R - - nXS- nU R U5 nXv4$ )Nr rc F)rD r? Tr, r/ ) r r r transposer\ r sumr r )r r r valueraw_attnrz soft_attnouts r# r GroupViTAssignAttention.forward s E" kk# E" MM"b 11TZZ?}}X&MM(5uME xxBx5GHliin~ r% )r r r r r\ r )TT)
r r r r r rx r r r r r s @r# r r s , 3 , r% r c < ^ \ rS rSrS\4U 4S jjrS rS rSrU =r $ )GroupViTTokenAssign rr c V > [ TU ] 5 X0l [ R " UR
UR S9U l [ UR [ R R 5 ( a UR OUR UR 4nU Vs/ s H n[ XQR
- 5 PM snu pg[ XXc5 U l [ R " UR
UR S9U l [ R " UR
UR S9U l [% U5 U l [) U5 U l [ R " UR
UR S9U l [/ XR
XqR
5 U l g s snf rt )rw rx num_output_groupr r{ r| r} norm_tokens
isinstanceassign_mlp_ratio collectionsabcIterablerT GroupViTMixerMLP mlp_internorm_post_tokensnorm_xrp pre_assign_attnr assign
norm_new_xr mlp_channels) r rr num_group_tokenr r r
tokens_dim channels_dimr s r# rx GroupViTTokenAssign.__init__ sD
0 <<(:(:@U@UV &11;??3K3KLL
# #))6+B+BC
JZ#ZIYAC,>,>(>$?IY#Z
)&:` " V-?-?VEZEZ [ ll6#5#56;P;PQ :6 B -f5 ,,v'9'9v?T?TU '0B0BLRdRd e $[s !F&c J U R U5 nU R U5 nU$ )z
Args:
group_tokens (torch.Tensor): group tokens, [batch_size, num_group_tokens, channels]
Returns:
projected_group_tokens (torch.Tensor): [batch_size, num_output_groups, channels]
)r r )r group_tokensprojected_group_tokenss r# project_group_token'GroupViTTokenAssign.project_group_token s+ "& != !%!6!67M!N %%r% c U R U5 nU R U5 nU R U5 nU R X15 nU R X15 u pEXC-
nX@R U R
U5 5 - nXE4$ )z
Args:
image_tokens (`torch.Tensor`): image tokens, of shape [batch_size, input_length, channels]
group_tokens (`torch.Tensor`): group tokens, [batch_size, num_group_tokens, channels]
)r r r r r r r )r image_tokensr r new_image_tokens attentions r# r GroupViTTokenAssign.forward s '' 5 {{<0 !%!9!9,!G !%!5!56L![ &*kk2H&W#2+.?.?P`@a.bb **r% ) r r r r r r r r r )
r r r r r rx r r r r r s @r# r r s! f 3 f* &+ +r% r c : \ rS rSr% SrSr\\R \ S' Sr
\\R \ S' Sr \\R \ S' Sr \\R \ S' Sr
\\R \ S' Sr\\R \ S ' Sr\\ S
' Sr\\ S ' S \\ 4S
jrSrg)GroupViTModelOutputi a
loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`):
Contrastive loss for image-text similarity.
logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`):
The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text
similarity scores.
logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`):
The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image
similarity scores.
segmentation_logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels, logits_height, logits_width)`):
Classification scores for each pixel.
The logits returned do not necessarily have the same size as the `pixel_values` passed as inputs. This is
to avoid doing two interpolations and lose some quality when a user needs to resize the logits to the
original image size as post-processing. You should always check your logits shape and resize as needed.
text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The text embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTTextModel`].
image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`):
The image embeddings obtained by applying the projection layer to the pooled output of
[`GroupViTVisionModel`].
text_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTTextModel`].
vision_model_output (`BaseModelOutputWithPooling`):
The output of the [`GroupViTVisionModel`].
Nlosslogits_per_imagelogits_per_textsegmentation_logits text_embeds image_embedstext_model_outputvision_model_outputr c J ^ [ U 4S jT R 5 5 5 $ )Nc 3 n ># U H* nUS ; a TU O[ TU5 R 5 v M, g7f))r r N)getattrto_tuple).0kr s r# /GroupViTModelOutput.to_tuple..0 s<
LLDGRYZ^`aRbRkRkRm m s 25)tuplekeysr s `r# r GroupViTModelOutput.to_tuple/ s#
YY[
r% )r r r r __doc__r r r FloatTensor__annotations__r r r r r r r r r r r r r r% r# r r s > )-D(5$$
%,48hu 0 01837OXe / /077;%"3"34;/3K%++,304L(5,,-448186:3:
%*
r% r c ^ \ rS rSrSr S
S\S\\\\\4 4 S\S\4U 4S jjjrSS\ R S \ S
\ R 4S jjr S r
U =r$ )GroupViTPatchEmbeddingsi6 z
Image to Patch Embedding.
image_size
patch_size num_channels embed_dimc ` > [ TU ] 5 [ U[ R R
5 ( a UOX4n[ U[ R R
5 ( a UOX"4nUS US - US US - - nXl X l XPl [ R " X4X"S9U l g )Nr r ) kernel_sizestride) rw rx r r r r r r num_patchesr Conv2d
projection)r r r r r r r s r# rx GroupViTPatchEmbeddings.__init__; s #-j+//:R:R#S#SZZdYq
#-j+//:R:R#S#SZZdYq
!!}
1
5*Q-:VW=:XY $$& ))Lgr% pixel_valuesinterpolate_pos_encodingr c > UR u p4pVU( dV XPR S :w d X`R S :w a2 [ SU SU SU R S SU R S S3 5 eU R U5 R S5 R SS5 nU$ )Nr r zInput image size (*z) doesn't match model ().rM )rH r
ValueErrorr flattenr )r r r r_ r rZ r[ r s r# r GroupViTPatchEmbeddings.forwardL s 2>2D2D/
&' ++u8J/J (% 9 +,Adooa.@-AE
OOL ) 1 1! 4 > >q! Dr% )r r r r ) r i F)r r r r r rT r r rx r Tensorboolr r r r s @r# r r 6 s 24 hh #uS#X./h h
h h" ELL D ]b]i]i r% r c ^ \ rS rSrS\4U 4S jjrS\R S\S\S\R 4S jr S
S \R S
\
S\R 4S jjr S r U =r
$ )GroupViTVisionEmbeddingsiX rr c > [ TU ] 5 [ UR UR UR
UR S9U l U R R n[ R " [ R " SX!R 5 5 U l
[ R " UR 5 U l [ R " UR UR" S9U l UR U l Xl g )N)r r r r r ru )rw rx r r r r r| patch_embeddingsr r Parameterr zerosposition_embeddingsDropoutdropoutr{ r} layernormrr )r rr r r s r# rx !GroupViTVisionEmbeddings.__init__Y s
7 (( (( ,, (( !
++77 #%<< A{L^L^0_#` zz&..1 f&8&8f>S>ST ++ r%
embeddingsrZ r[ r c , UR S nU R R S n[ R R 5 ( d XE:X a X#:X a U R $ U R nUR S nX R
- nX0R
- n [
US- 5 n
UR SXU5 nUR SSSS5 n[ R R UX4SSS 9nUR SSSS5 R SSU5 nU$ )
a
This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution
images. This method is also adapted to support torch.jit tracing and no class embeddings.
Adapted from:
- https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and
- https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211
r rc rN r r rM bicubicFrP )
rH r r jit
is_tracingr r rW re r r rX view) r r rZ r[ r
num_positionspatch_pos_embedr,
new_height new_widthsqrt_num_positionss r# r 1GroupViTVisionEmbeddings.interpolate_pos_encodingi s !&&q) 0066q9
yy##%%+*F6?++ + 22 r".
__, &}c'9 :)11!5G]`a)11!Q1= --33 ( 4
*11!Q1=BB1b#N r% r r c UR u p4pVU R XS9nU R U5 nUR 5 u p8n U( a XpR XuU5 - nOXpR
- nU R
U5 nU$ )N)r )rH r r rQ r r r
)
r r r r_ r rZ r[ r seq_len_s
r# r GroupViTVisionEmbeddings.forward s 2>2D2D/
&**<*k
^^J/
!+ !2
Q $#&C&CJX]&^^J#&>&>>J\\*-
r% )rr r
r r r r r )r r r r r rx r r rT r r r r r r s @r# r r X sh 3 $ 5<< $ $ UX $ ]b]i]i $ LELL D ]b]i]i r% r c ^ \ rS rSrS\4U 4S jjr S
S\\R S\\R S\\R S\R 4S jjr S r U =r
$ ) GroupViTTextEmbeddingsi rr c N > [ TU ] 5 UR n[ R " UR
U5 U l [ R " UR U5 U l U R S[ R " UR 5 R S5 SS9 g )N position_ids)r rc F)
persistent)
rw rx r| r Embedding
vocab_sizetoken_embeddingmax_position_embeddingsposition_embeddingregister_bufferr r! expandr rr r r s r# rx GroupViTTextEmbeddings.__init__ s
&& !||F,=,=y I "$,,v/M/My"Y
ELL)G)G H O OPW Xej
r% input_idsr#
inputs_embedsr c < Ub UR S OUR S nU R R R S nXE: a [ SU SU 35 eUc U R S S 2S U24 nUc U R U5 nU R U5 nX6- nU$ )Nrc r r zRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rH r) weightr r# r' )r r. r# r/
seq_lengthmax_position_embeddingr r s r# r GroupViTTextEmbeddings.forward s -6,AY__R(}GZGZ[]G^
!%!8!8!?!?!E!Ea!H . d , <=S [ TU ] 5 X l X@l US: a; [ R
" [ R " SXAR 5 5 U l OS U l [ R " [ U5 Vs/ s H n[ U5 PM sn5 U l
US: a [ UUUS9U l OS U l US: ab US: a\ [ R " [ R" " UR UR$ S9[' XUR S- U5 5 U l g S U l g s snf )Nr r )rr r r ru rM )rw rx r9 r r r r r
r| group_token
ModuleListrangeGroupViTEncoderLayerlayersr
downsample
Sequentialr{ r} r group_projector)r rr r9 r: r r r r s r# rx GroupViTStage.__init__ s
. Q !||EKK?L^L^,_ `D #D mm5QV<$X '
u ! ' 'r% r8 c
^ \ rS rSr S S\S\\ S\\ S\\ 4U 4S jjjrS\R S\R 4S jr
S
r U =r $ ) r i) rr r| intermediate_size output_sizec > [ TU ] 5 Xl [ UR U l Ub UO UR nUb UO UR nUb UOUn[ R " X#5 U l
[ R " X45 U l g rF ) rw rx rr r
hidden_act
activation_fnr| r^ r r fc1fc2)r rr r| r^ r_ r s r# rx GroupViTMLP.__init__* s| #F$5$5 6%0% [ TU ] UR SS5 5 nUR SS5 $ Nr rM )rw r r )r r r s r# r GroupViTMixerMLP.forwardB s- GOAKK1 - .{{1a r% r )r r r r r r r r s @r# r r A s
! !r% r c F ^ \ rS rSrSrU 4S jrS\R S\S\4S jr SS\R S \
\R S
\
\R S \
\R S \
\ S
\
\R \
\R \
\
\R 4 4 S jjrSrU =r$ )ry iG z=Multi-headed attention from 'Attention Is All You Need' paperc > [ TU ] 5 Xl UR U l UR
U l U R U R - U l U R U R - U R :w a& [ SU R SU R S35 eU R S- U l UR U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l [ R " U R U R 5 U l g )Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: r r )rw rx rr r| r num_attention_heads num_headshead_dimr r\ attention_dropoutr
r r r r r out_projr s r# rx GroupViTAttention.__init__J s
++ 33 $..8
==4>> )T^^ ; MdnnM] ^NN#2'
]]D(
// ii? ii? ii? $..$..A
r% rF r bszc UR X2U R U R 5 R SS5 R 5 $ rj )r ro rp r rf )r rF r rt s r# _shapeGroupViTAttention._shape] s5 {{3 GQQRSUVWbbddr% rP rT rU r rR r c UR 5 u pgnUSLn U R U5 U R - n
U ( aE U R U R U5 SU5 n U R U R U5 SU5 n ODU R U R U5 SU5 n U R U R U5 SU5 n X`R - SU R 4n
U R XU5 R " U
6 n
U R " U
6 n U R " U
6 n U R S5 n[ R " XR SS5 5 nUR 5 X`R - X~4:w a- [ SX`R - X~4 SUR 5 35 eUbv UR 5 USX~4:w a" [ SUSX~4 SUR 5 35 eUR X`R X~5 U- nUR X`R - X~5 nUbv UR 5 USX~4:w a" [ SUSX~4 SUR 5 35 eUR X`R X~5 U- nUR X`R - X~5 n[ R R USS9nU( a; UR X`R X~5 nUR X`R - X~5 nOSn[ R R! XR U R" S 9n[ R " UU 5 nUR 5 X`R - XpR 4:w a5 [ S
X`R XpR 4 SUR 5 35 eUR X`R XpR 5 nUR SS5 nUR% XgU5 nU R' U5 nUU4$ ) z#Input shape: Batch x Time x ChannelNrc r rM z$Attention weights should be of size z , but is z!Attention mask should be of size r )pr z `attn_output` should be of size )rQ r r\ rv r r ro rp r r bmmr r r r r3 r
r rW rr )r rP rT rU r rR rt tgt_lenr is_cross_attention query_states
key_states value_states
proj_shapesrc_len attn_weightsattn_weights_reshaped
attn_probs attn_outputs r# r GroupViTAttention.forward` s #0"4"4"6 i 2$ > {{=1DJJ> T[[1F%GSQJ ;;t{{3H'I2sSL T[[ %?SIJ ;;t{{='A2sKLNN*B
>
{{<#>CCZP __j1
#((*5 //!$ yy /C/CAq/IJ 3 #7"J J 6nn8Lg7_6` a %%'(*
! ,$))+Q/II 7a8R7S T-22457 (,,S..'SVkkL',,S>>-A7TL % ""$a(BB 7a8R7SS\]k]p]p]r\st (,,S..'SVddL',,S>>-A7TL}},,\r,B
%1$5$5c>>7$\ !055cNN6JG]L$( !]]**<<
R2
u||Xell 3XeELL>Q5RR SR2 R2r% ry c ^ \ rS rSrS\4U 4S jjr S S\R S\R S\R S\\ S\
\R 4
S jjr S
r
U =r$ ) r? i rr c < > [ TU ] 5 UR U l [ U5 U l [ R " U R UR S9U l [ U5 U l [ R " U R UR S9U l g rt )
rw rx r| r ry self_attnr r{ r} layer_norm1r r layer_norm2r s r# rx GroupViTEncoderLayer.__init__ sm
++*62 <<F &
u !
& &r% r? c . \ rS rSr% \\S' SrSrS rSr g)GroupViTPreTrainedModeli rr groupvitTc b U R R n[ U[ R [ R
45 ( aV UR R R SUS9 UR b$ UR R R 5 Oh[ U[ R 5 ( aI UR R R 5 UR R R S5 U R R n[ U[ 5 ( ac UR R R R SUS- S9 UR R R R SUS- S9 g[ U[" 5 ( Ga U R R nUR$ S- SUR R& - S- - U- nUR$ S- U- n[ R( R UR* R US9 [ R( R UR, R US9 [ R( R UR. R US9 [ R( R UR0 R US9 g[ U[2 5 ( a U R R nUR R4 S- SUR R& - S- - U- nSUR R4 - S- U- n[ R( R UR6 R US9 [ R( R UR8 R US9 gg) zInitialize the weightsrA )meanstdNr2 g{Gz?r rM )r ) rr initializer_ranger r r r r1 datanormal_biaszero_r{ fill_initializer_factorr! r' r) ry r num_hidden_layersinitr r r rr r r| rc rd )r module
init_rangefactor in_proj_std out_proj_stdfc_stds r#
_init_weights%GroupViTPreTrainedModel._init_weights s [[22
fryy")) 4 5 5
MM & &CZ & @{{& &&(
-
- KK " " $ MM $ $S )// f 4 5 5 " " ) ) . . 6 6CVd] 6 S % % , , 1 1 9 9sQU
9 V
1
2
2[[33F!++T1q6==;Z;Z7Z_c6cdgmmK " , ,d 2f [ TU ] 5 Xl [ R " [ [
UR 5 5 Vs/ s HO n[ UUR U UR U UR U US: a UR US-
OSS9PMQ sn5 U l SU l g s snf )Nr r )rr r9 r r r: F)
rw rx rr r r= r> r" depthsr8 num_group_tokensnum_output_groupsstagesgradient_checkpointing)r rr ir s r# rx GroupViTVisionEncoder.__init__ s
mm s6== 12
3A ! --*$*$;$;A$>%+%=%=a%@LMPQE)A)A!a%)HWX 3
',#
s AB0rP output_hidden_statesrR return_dictc Ub UOU R R nUb UOU R R nUb UOU R R nU( a SOS nU( a SOS nS n[ U R
5 H= u pU( a XQ4- nU " XU5 n
U
S nU
S nU( d M- U
S c M5 XjS 4- nM? U( a XQ4- nU( d [
S XU4 5 5 $ [ XUS9$ )Nr r r rM c 3 . # U H oc M Uv M
g 7frF r r vs r# r 0GroupViTVisionEncoder.forward..D s g$Uq$U last_hidden_staterP rY )rr rR r use_return_dict enumerater r r
) r rP r rR r all_hidden_states
all_groupingsr r stage
layer_outputss r# r GroupViTVisionEncoder.forward" s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% %
d^ %
uo% &
%
%
r% r c ^ \ rS rSrSrS\4U 4S jjr S
S\\R S\\R S\\
S\\
S \\
S
\ \ \
4 4 S jjrS rU =r$ )GroupViTTextEncoderiJ z
Transformer encoder consisting of `config.num_hidden_layers` self-attention layers. Each layer is a
[`GroupViTEncoderLayer`].
Args:
config: GroupViTTextConfig
rr c > [ TU ] 5 Xl [ R " [ UR 5 Vs/ s H n[ U5 PM sn5 U l SU l g s snf r[ )
rw rx rr r r= r> r r? r@ r )r rr r r s r# rx GroupViTTextEncoder.__init__S sT
mm5QWQiQiKj$kKja%9&%AKj$kl &+# %ls A&rT rU rR r r r c Ub UOU R R nUb UOU R R nUb UOU R R nU( a SOSnU( a SOSnUn [ U R
5 H0 u pU( a Xy4- nU " U UUUS9n U S n U( d M( XS 4- nM2 U( a Xy4- nU( d [
S XU4 5 5 $ [ XUS9$ )a
Args:
inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`):
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.
This is useful if you want more control over how to convert `input_ids` indices into associated vectors
than the model's internal embedding lookup matrix.
attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
Causal mask for the text model. Mask values selected in `[0, 1]`:
- 1 for tokens that are **not masked**,
- 0 for tokens that are **masked**.
[What are attention masks?](../glossary#attention-mask)
output_attentions (`bool`, *optional*):
Whether or not to return the attentions tensors of all attention layers. See `attentions` under
returned tensors for more detail.
output_hidden_states (`bool`, *optional*):
Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors
for more detail.
return_dict (`bool`, *optional*):
Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
Nr )rR r r c 3 . # U H oc M Uv M
g 7frF r r s r# r .GroupViTTextEncoder.forward.. s e$Sq$Sr r )rr rR r r r r@ r r
)
r r/ rT rU rR r r encoder_statesall_attentionsrP idx
encoder_layerr s
r# r GroupViTTextEncoder.forwardY s L 2C1N -TXT_T_TqTq$8$D $++JjJj &1% >Ne]N$See e +Vd
r% )rr r r@ )NNNNN)r r r r r r rx r r r r r r r
r r r r s @r# r r J s , 1 , 268<,0/3&*F
! .F
( 5 F
$D> F
'tn
F
d^F
uo% &F
F
r% r c ^ \ rS rSrS\4U 4S jjr\ S
S\\R S\\R S\\R S\\
S\\
S \\
S
\ \ \
4 4S jj5 rS rU =r$ )GroupViTTextTransformeri rr c > [ TU ] 5 Xl UR n[ U5 U l [
U5 U l [ R " X!R S9U l UR U l g rt )
rw rx rr r| r! r r encoderr r{ r} final_layer_norm eos_token_idr, s r# rx GroupViTTextTransformer.__init__ s]
&& 08*62 " Y L
'tn
L
d^L
u00 1L
L
r% r c ^ \ rS rSr% \\S' S\4U 4S jjrS\R 4S jr S r
\ SS\ \
R S\ \
R S \ \
R S
\ \ S \ \ S \ \ S\\\4 4S
jj5 rSrU =r$ )GroupViTTextModeli rr c d > [ TU ] U5 [ U5 U l U R 5 g rF )rw rx r
text_model post_initr s r# rx GroupViTTextModel.__init__ s&
1&9 r% r c B U R R R $ rF r r r' r s r# get_input_embeddings&GroupViTTextModel.get_input_embeddings s ))999r% c 8 XR R l g rF r )r r s r# set_input_embeddings&GroupViTTextModel.set_input_embeddings
s 5:""2r% r. rT r# rR r r c * U R UUUUUUS9$ )a
Examples:
```python
>>> from transformers import CLIPTokenizer, GroupViTTextModel
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTTextModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled (EOS token) states
```r. rT r# rR r r r )r r. rT r# rR r r s r# r GroupViTTextModel.forward
s, 2 )% /!5#
r% r r )r r r r r r rx r Moduler r r r r r r r r r r r r r s @r# r r s 1 :bii :; -115/3,0/3&*
ELL)
! .
u||,
$D>
'tn
d^
u00 1
r% r c ^ \ rS rSrS\4U 4S jjr\ S S\\R S\\
S\\
S\\
S\ \ \
4 4
S jj5 rS
rU =r$ ) GroupViTVisionTransformeri0 rr c > [ TU ] 5 Xl UR n[ U5 U l [
U5 U l [ R " X!R S9U l g rt ) rw rx rr r| r r r r r r{ r} r r, s r# rx "GroupViTVisionTransformer.__init__1 sL
&& 26:,V4 i5J5JKr% r r rR r r c Ub UOU R R nUb UOU R R nUb UOU R R nUc [ S5 eU R U5 nU R
UUUUS9nUS nU R U5 nUR SS9nU( d Xx4USS - $ [ UUUR UR S9$ )Nz You have to specify pixel_values)rP r rR r r r r r ) rr rR r r r r r r r r rP rY ) r r r rR r rP r r r s r# r !GroupViTVisionTransformer.forward: s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% '
d^ '
u00 1
'
'
r% r c ^ \ rS rSr% \\S' SrS\4U 4S jjrS\4S jr \
S S\ \ R S\ \ S\ \ S \ \ S\\\4 4
S
jj5 rS rU =r$ )
GroupViTVisionModelie rr r c d > [ TU ] U5 [ U5 U l U R 5 g rF )rw rx r vision_modelr r s r# rx GroupViTVisionModel.__init__i s'
5f = r% r c B U R R R $ rF )r r r r s r# r (GroupViTVisionModel.get_input_embeddingso s ++<<>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTVisionModel
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> model = GroupViTVisionModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(images=image, return_tensors="pt")
>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output # pooled CLS states
```r rR r r r )r r rR r r s r# r GroupViTVisionModel.forwardr s( 8 % /!5# !
r% r r )r r r r r r main_input_namerx r r r r r r r r r r r r r r s @r# r r e s $O 3 =&= = 59,0/3&*
u 0 01
$D>
'tn
d^
u00 1
r% r c ^ \ rS rSr% \\S' S\4U 4S jjr\" 5 \ SS\ R S\ \ R S\ \ R S\ R 4S jj5 5 r
\" 5 \S \ R S\ R 4S
j5 5 r\ SS\ \ R S \ \ R S\ \ R S\ \ R S \ \ S \ \ S
\ \ S\ \ S\ \ S\\\4 4S jj5 rSrU =r$ )
GroupViTModeli rr c > > [ TU ] U5 [ UR [ 5 ( d" [ S[
UR 5 S35 e[ UR [ 5 ( d" [ S[
UR 5 S35 eUR nUR nUR U l UR U l
UR U l UR U l
[ U5 U l [! U5 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R& " [$ R( " U R U R SS9[$ R* " U R 5 [$ R, " SS9[$ R( " U R U R SS95 U l [$ R2 " [4 R6 " U R8 R: 5 5 U l U R? 5 g )NzOconfig.text_config is expected to be of type GroupViTTextConfig but is of type .zSconfig.vision_config is expected to be of type GroupViTVisionConfig but is of type T)r )inplace) rw rx r text_configr TypeErrortype
vision_configr projection_dimprojection_intermediate_dimr| text_embed_dimvision_embed_dimr r r r r rB r BatchNorm1dReLUvisual_projectiontext_projectionr r rF rr logit_scale_init_value logit_scaler )r rr r r r s r# rx GroupViTModel.__init__ s
&,,.@AA++,-Q0
&..0DEE--./q2
(( ,,
$ 3 3+1+M+M( ) 5 5 - 9 9 1+> 5m D!# IId++T-M-MTX Y NN4;; < GGD ! IId668K8KRV W "
"}} IId))4+K+KRV W NN4;; < GGD ! IId668K8KRV W
<< T[[5W5W(XY
r% r. rT r# r c ^ U R UUUS9nU R UR 5 nU$ )a|
Returns:
text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by
applying the projection layer to the pooled output of [`GroupViTTextModel`].
Examples:
```python
>>> import torch
>>> from transformers import CLIPTokenizer, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> tokenizer = CLIPTokenizer.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
>>> with torch.inference_mode():
... text_features = model.get_text_features(**inputs)
```)r. rT r# )r r r )r r. rT r# text_outputs
text_featuress r# get_text_features GroupViTModel.get_text_features s? 4 48?? )% 4C 4
,,\-G-GH
r% r c ^ U R U5 nU R UR 5 nU$ )a
Returns:
image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by
applying the projection layer to the pooled output of [`GroupViTVisionModel`].
Examples:
```python
>>> import torch
>>> from transformers import AutoProcessor, GroupViTModel
>>> from transformers.image_utils import load_image
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = load_image(url)
>>> inputs = processor(images=image, return_tensors="pt")
>>> with torch.inference_mode():
... image_features = model.get_image_features(**inputs)
```)r r r )r r vision_outputsimage_featuress r# get_image_features GroupViTModel.get_image_features s0 4 6:5F5F|5T //0L0LM r% return_lossrR r output_segmentationr c
Ub UOU R R nUb UOU R R nU( a SnUb UOU R R nU b U OU R R n U R UUUU S9n
U R
UUUUUU S9n U
S n U R U 5 n U S n
U R U
5 n
XR SSS9- n XR SSS9- n
U R R 5 n[ R " XR 5 5 U- nUR 5 nSnU( Gaf U
S nU R UR SUR S 5 5 nU( a U
S nOU
S
n[# UUR S
S 5 nUUR SSS9- n[ R " UU
R 5 5 U- nUR U R S SU
R S 5 R% SS
S5 nUR UR S UR S S5 n[ R " UU5 U- nUR UR S UR S UR S
UR S 5 nSnU( a [' U5 nU ( d Ub
UUUU
U U U
4nOUXXU
4nUb U4U- $ U$ [) UUUUU
U U U
S 9$ ) a
return_loss (`bool`, *optional*):
Whether or not to return the contrastive loss.
output_segmentation (`bool`, *optional*):
Whether or not to return the segmentation logits.
Examples:
```python
>>> from PIL import Image
>>> import requests
>>> from transformers import AutoProcessor, GroupViTModel
>>> model = GroupViTModel.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> processor = AutoProcessor.from_pretrained("nvidia/groupvit-gcc-yfcc")
>>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
>>> image = Image.open(requests.get(url, stream=True).raw)
>>> inputs = processor(
... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
... )
>>> outputs = model(**inputs)
>>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score
>>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities
```NTr r r rc r r r rM )r r r r r r r r )rr rR r% r r r r r r normr expr matmulr( rW rH rn re r+ r )r r. r rT r# r$ rR r r% r r r r r r r r
seg_logitsimage_group_embedsrY groupinglogits_per_image_groupflatten_groupingr outputs r# r GroupViTModel.forward sc R 2C1N -TXT_T_TqTq#6#B HgHg $ $8$D $++JjJj &1%>RSCTVX Y &<>NOR]]J#++ #Z%5%5a%8(..:KX^^\]M^J 1D%$# " +O,ftu)-)9TGf$ Ev E" - + *#% * .
r% ) r r r r r r r r r )NN) NNNNNNNNN)r r r r r r rx r r r r r r r r" r6 r r r r r r r r s @r# r r s )~ )V %& 26/3 << ! . u||,
' @ %& u|| @Q@Q ' 8 15481537&*,0/3.2&*N
E , ,-N
u 0 01N
! . N
u / /0 N
d^
N
$D>N
'tnN
&d^N
d^N
u)) *N
N
r% r )r r r r )r Frc r )Fr collections.abcr dataclassesr typingr r r numpyrU r r activationsr modeling_attn_mask_utilsr
r modeling_layersr modeling_outputsr
r modeling_utilsr utilsr r r r r configuration_groupvitr r r
get_loggerr loggerr r$ r+ rT r= floatr rK ra rn r rp r r r r r r! r8 r r ry r? r r r r r r r r __all__r r% r# r@ s` ! ' ' ! d 9 K - e e \ \
H %
`U\\ `ell `
-ell -u|| - C 5<< e t RU _d_k_k ,<:
"))
- bii - `4+")) 4+n -
+ -
-
` bii DGryy GV%RYY %P[BII [| ")) 0!{ ! k2 k2^/5 /d $@o $@ $@N7
BII 7
tU
")) U
pY
bii Y
x/
/ /
d2
2
j.
1 .
b }
+ }
}
@ cr%