cCiorSrSSKJrJr SSKrSSKJr SSKJr SSKJ r SSK J r J r J r SS KJrJr SS KJr SS KJrJr SS KJrJrJrJr SS KJrJr SSKJr \R@"\!5r""SS\RF5r$"SS\RF5r%S5S\RFS\RLS\RLS\RLS\\RLS\'S\'4Sjjr("SS\RF5r)"SS\RF5r*"S S!\RF5r+"S"S#\RF5r,"S$S%\RF5r-"S&S'\ 5r."S(S)\RF5r/"S*S+\RF5r0\"S,S-\55r1\"S.S/\155r2\"S0S19"S2S3\155r3/S4Qr4g)6zPyTorch ViViT model.)CallableOptionalN)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) find_pruneable_heads_and_indicesprune_linear_layer)TransformersKwargsauto_docstringlogging torch_int)can_return_tuplecheck_model_inputs) VivitConfigcv^\rSrSrSrS\4U4SjjrS S\RS\ S\R4Sjjr S r U=r $) VivitTubeletEmbeddings$az Construct Vivit Tubelet embeddings. This module turns a batch of videos of shape (batch_size, num_frames, num_channels, height, width) into a tensor of shape (batch_size, seq_len, hidden_size) to be consumed by a Transformer encoder. The seq_len (the number of patches) equals (number of frames // tubelet_size[0]) * (height // tubelet_size[1]) * (width // tubelet_size[2]). configc>[TU]5 URUlURUlURUlURUR S-URUR S--URUR S--UlURUl[R"URURURURS9Ul g)Nrr) kernel_sizestride) super__init__ num_frames image_size tubelet_size patch_size num_patches hidden_size embed_dimrConv3d num_channels projectionselfr __class__s b/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/vivit/modeling_vivit.pyr"VivitTubeletEmbeddings.__init__/s  ++ ++ -- __ 2 2$//!"44 6$//!"44 6   ++))   !3!3ATAT]c]p]p  pixel_valuesinterpolate_pos_encodingreturnc bURup4pVnU(dPX`R:wdXpR:wa2[SUSUSURSSURSS3 5eURSSSSS 5nUR U5nUR S5R SS5nU$) NzImage image size (*z) doesn't match model (rrz).rr)shaper$ ValueErrorpermuter,flatten transpose) r.r3r4 batch_sizer#r+heightwidthxs r0forwardVivitTubeletEmbeddings.forward?s>J>P>P; e'V-F%SbSbJb$VHAeW4KDOO\]L^K__`aeapapqras`ttvw  $++Aq!Q: OOL ) IIaL " "1a (r2)r)r$r#r'r&r,F) __name__ __module__ __qualname____firstlineno____doc__rr"torchTensorboolrB__static_attributes__ __classcell__r/s@r0rr$s> { ELLD]b]i]ir2rc^\rSrSrSrS\4U4SjjrS\RS\ S\ S\R4S jr SS \RS \ S\R4S jjr S r U=r$)VivitEmbeddingsPz| Vivit Embeddings. Creates embeddings from a video using VivitTubeletEmbeddings, adds CLS token and positional embeddings. rc>[TU]5 [R"[R "SSUR 55Ul[U5Ul [R"[R "SURRS-UR 55Ul [R"UR5UlURSSUlXlg)Nr)r!r"r ParameterrJzerosr( cls_tokenrpatch_embeddingsr'position_embeddingsDropouthidden_dropout_probdropoutr%r&rr-s r0r"VivitEmbeddings.__init__Ws ekk!Q8J8J&KL 6v >#%<< KK400<r#r+r?r@r] cls_tokenss r0rBVivitEmbeddings.forwards>J>P>P; e**<*k ^^((*a);< YY 7Q?  $#&C&CJX]&^^J#&>&>>J\\*- r2)rVrr[rWr&rXrD)rErFrGrHrIrr"rJrKintr4rLrBrMrNrOs@r0rQrQPsq { &D5<<&D&DUX&D]b]i]i&DPELLD]b]i]ir2rQmodulequerykeyvalueattention_maskscalingr[c [R"XRSS55U-n[RR US[R S9RUR5n[RRXURS9nUbX-n[R"X5n U RSS5R5n X4$)Nr_)redtype)ptrainingrr) rJmatmulr=rrisoftmaxfloat32torr[r contiguous) rzr{r|r}r~rr[kwargs attn_weights attn_outputs r0eager_attention_forwardrs<<}}R'<=GL==((2U]](SVVW\WbWbcL==((6??([L!#4 ,,|3K''1-88:K  $$r2c ^\rSrSrS\4U4SjjrS S\RS\\RS\ \R\R44Sjjr Sr U=r $) VivitSelfAttentionrc0>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eXlURUl[URUR- 5UlURUR-Ul URUl URS-Ul SUl [R"URURUR S9Ul[R"URURUR S9Ul[R"URURUR S9Ulg) Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .gF)bias)r!r"r(num_attention_headshasattrr:rryattention_head_size all_head_sizeattention_probs_dropout_prob dropout_probr is_causalrLinearqkv_biasr{r|r}r-s r0r"VivitSelfAttention.__init__sG     : : :a ?PVXhHiHi"6#5#5"67334A7   #)#=#= #&v'9'9F new_shape key_layer value_layer query_layerattention_interface context_layerattention_probsnew_context_layer_shapes r0rBVivitSelfAttention.forwardsH#((+ D$<$V>VV HH]+00)<FFq!L jj/44i@JJ1aP jj/44i@JJ1aP (? ;; + +w 6"9$++:Z:Z"[ )<     nnLL#}}C$2C2C * & #0"4"4"6s";t?Q?Q>S"S%--.EF --r2) rrrrrr|rr{rr}N) rErFrGrHrr"rJrKrtuplerBrMrNrOs@r0rrsY]{]*PT."\\.6>u||6L. u||U\\) *..r2rc^\rSrSrSrS\4U4SjjrS\RS\RS\R4Sjr S r U=r $) VivitSelfOutputz The residual connection is defined in VivitLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rc>[TU]5 [R"URUR5Ul[R "UR5Ulgr) r!r"rrr(denserYrZr[r-s r0r"VivitSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r2r input_tensorr5cJURU5nURU5nU$rrr[r.rrs r0rBVivitSelfOutput.forwards$ =1  ]3 r2r) rErFrGrHrIrr"rJrKrBrMrNrOs@r0rrsB >{> U\\RWR^R^r2rc^\rSrSrS\4U4SjjrS\\4SjrS S\ RS\ \ RS\ R4S jjr S r U=r$) VivitAttentionirc>[TU]5 [U5Ul[ U5Ul[ 5Ulgr)r!r"r attentionroutputset pruned_headsr-s r0r"VivitAttention.__init__s0 +F3%f- Er2headsc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrrd)lenrrrrrrr{r|r}rrrunion)r.rindexs r0 prune_headsVivitAttention.prune_heads s u:? 7 >>55t~~7Y7Y[_[l[l  2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r2rrr5cNURX5up4URX15nU$r)rr)r.rrself_attn_output_rs r0rBVivitAttention.forwards("nn]F-= r2)rrrr)rErFrGrHrr"rryrrJrKrrBrMrNrOs@r0rrsR"{" ;S;$U\\hu||>T`e`l`lr2rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $)VivitIntermediatei%rc^>[TU]5 [R"URUR 5Ul[R"UR5Ul [UR[5(a[URUlgURUlgr)r!r"rrr(intermediate_sizerrYrZr[ isinstance hidden_actstrrintermediate_act_fnr-s r0r"VivitIntermediate.__init__&sv YYv1163K3KL zz&"<"<= f'' - -'-f.?.?'@D $'-'8'8D $r2rr5clURU5nURU5nURU5nU$r)rrr[)r.rs r0rBVivitIntermediate.forward/s4 =1 00?  ]3 r2)rr[r rErFrGrHrr"rJrKrBrMrNrOs@r0rr%s/9{9U\\ellr2rc^\rSrSrS\4U4SjjrS\RS\RS\R4SjrSr U=r $) VivitOutputi7rc>[TU]5 [R"URUR 5Ul[R"UR5Ul gr) r!r"rrrr(rrYrZr[r-s r0r"VivitOutput.__init__8sB YYv779K9KL zz&"<"<= r2rrr5cRURU5nURU5nX-nU$rrrs r0rBVivitOutput.forward=s, =1  ]3 %4 r2rrrOs@r0rr7s=>{> U\\RWR^R^r2rc^\rSrSrSrS\4U4SjjrS S\RS\ \RS\R4Sjjr S r U=r $) VivitLayeriDzNThis corresponds to the EncoderBlock class in the scenic/vivit implementation.rcj>[TU]5 URUlSUl[ U5Ul[ U5Ul[U5Ul [R"URURS9Ul[R"URURS9Ulg)Nreps)r!r"chunk_size_feed_forward seq_len_dimrrr intermediaterrr LayerNormr(layer_norm_epslayernorm_beforelayernorm_afterr-s r0r"VivitLayer.__init__Gs '-'E'E$'/-f5!&) " V-?-?VEZEZ [!||F,>,>FDYDYZr2rrr5cURU5nURX25nXA-nURU5nURU5nUR XQ5nU$r)rrrrr)r.rrhidden_states_normattention_output layer_outputs r0rBVivitLayer.forwardQse!22=A>>*<H)8 ++M: ((6 {{<? r2)rrrrrrrr) rErFrGrHrIrr"rJrKrrBrMrNrOs@r0rrDsGX[{[U\\hu||>T`e`l`lr2rcx^\rSrSrS\4U4SjjrS S\RS\\RS\ 4Sjjr Sr U=r $) VivitEncoderibrc>[TU]5 Xl[R"[ UR 5Vs/sHn[U5PM sn5UlSUl gs snf)NF) r!r"rr ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r.rrr/s r0r"VivitEncoder.__init__csR  ]]fF^F^@_#`@_1Jv$6@_#`a &+#$asA&rrr5cr[UR5Hup4UbX#OSnU"X5nM [US9$)N)last_hidden_state) enumeraterr )r.rri layer_modulelayer_head_masks r0rBVivitEncoder.forwardis<(4OA.7.CilO(HM 5??r2)rrrr) rErFrGrHrr"rJrKrr rBrMrNrOs@r0rrbsA,{, @U\\@hu||>T@`o@@r2rcj^\rSrSrS\4U4SjjrS\RS\R4SjrSr U=r $) VivitPooleriqrc>[TU]5 [R"URUR5Ul[R "5Ulgr)r!r"rrr(rTanh activationr-s r0r"VivitPooler.__init__rs9 YYv1163E3EF '')r2rr5c\USS2S4nURU5nURU5nU$)Nr)rr)r.rfirst_token_tensor pooled_outputs r0rBVivitPooler.forwardws6+1a40 #56  6 r2)rrrrOs@r0rrqs/${$ U\\ellr2rcP\rSrSr%\\S'SrSrSr/r Sr Sr Sr Sr \\S.rSrSrg ) VivitPreTrainedModelirvivitr3T)r attentionsc[U[R[R45(akURR R SURRS9 URb%URR R5 gg[U[R5(axURR R SURRS9 URb2URR URR5 gg[U[R5(aJURR R5 URR RS5 g[U[5(aIUR R R5 UR"R R5 gg)zInitialize the weightsr)meanstdNg?)rrrr*weightdatanormal_rinitializer_rangerzero_ Embedding padding_idxrfill_rQrVrX)r.rzs r0 _init_weights"VivitPreTrainedModel._init_weightssU fryy"))4 5 5 MM   & &CT[[5R5R & S{{&   &&('  - - MM   & &CT[[5R5R & S!!- ""6#5#56<<>.  - - KK   " " $ MM   $ $S )  0 0    ! ! ' ' )  & & + + 1 1 31r2N)rErFrGrHr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsrrMrr2r0r r sI $O&*#N"&#( 4r2r c ^\rSrSrSS\S\4U4SjjjrSrSr\ "SS9\ SS \ \ RS \ \ RS \S \\S \4 Sjj55rSrU=r$) VivitModeliradd_pooling_layerc>[TU]U5 Xl[U5Ul[ U5Ul[R"URURS9Ul U(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r!r"rrQr]rencoderrrr(r layernormrpooler post_init)r.rr*r/s r0r"VivitModel.__init__si   )&1#F+ f&8&8f>S>ST->k&)D  r2c.URR$r)r]rW)r.s r0get_input_embeddingsVivitModel.get_input_embeddingss///r2cUR5H7up#URRURR U5 M9 g)zy Prunes heads of the model. Args: heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsr,rrr)r.heads_to_prunerrs r0 _prune_headsVivitModel._prune_headss<+002LE LL  u % / / ; ;E B3r2F)tie_last_hidden_statesr3rr4rr5c 0Uc [S5eURX RR5nUR XS9nUR XRS9nUR nURU5nURbURU5OSn[XxS9$)a( Examples: ```python >>> import av >>> import numpy as np >>> from transformers import VivitImageProcessor, VivitModel >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=1, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitModel.from_pretrained("google/vivit-b-16x2-kinetics400") >>> # prepare video for the model >>> inputs = image_processor(list(video), return_tensors="pt") >>> # forward pass >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 3137, 768] ```Nz You have to specify pixel_valuesru)r)r pooler_output) r: get_head_maskrrr]r,rr-r.r ) r.r3rr4rembedding_outputencoder_outputssequence_outputr s r0rBVivitModel.forwardsh  ?@ @&&y++2O2OP ?? Note that it's possible to fine-tune ViT on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution. ) custom_introc^\rSrSrS\4U4Sjjr\\S S\\ RS\\ RS\\ RS\ S\ \S \4 S jj55rS rU=r$) VivitForVideoClassificationi'rc.>[TU]U5 URUl[USS9UlURS:a+[ R "URUR5O[ R"5Ul UR5 g)NF)r*r) r!r" num_labelsr)rrrr(Identity classifierr/r-s r0r"$VivitForVideoClassification.__init__6ss   ++%@ OUN_N_bcNc"))F$6$68I8IJikititiv r2r3rlabelsr4rr5c UR"U4X$S.UD6nURnURUSS2SSS245nSn UbUR"X8UR40UD6n [ U UUR URS9$)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> import av >>> import numpy as np >>> import torch >>> from transformers import VivitImageProcessor, VivitForVideoClassification >>> from huggingface_hub import hf_hub_download >>> np.random.seed(0) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # video clip consists of 300 frames (10 seconds at 30 FPS) >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample 32 frames >>> indices = sample_frame_indices(clip_len=32, frame_sample_rate=4, seg_len=container.streams.video[0].frames) >>> video = read_video_pyav(container=container, indices=indices) >>> image_processor = VivitImageProcessor.from_pretrained("google/vivit-b-16x2-kinetics400") >>> model = VivitForVideoClassification.from_pretrained("google/vivit-b-16x2-kinetics400") >>> inputs = image_processor(list(video), return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) ... logits = outputs.logits >>> # model predicts one of the 400 Kinetics-400 classes >>> predicted_label = logits.argmax(-1).item() >>> print(model.config.id2label[predicted_label]) LABEL_116 ```)rr4Nr)losslogitsrr)rrrH loss_functionrr rr) r.r3rrJr4routputsr?rMrLs r0rB#VivitForVideoClassification.forwardBsz$(:: $ $-$ dj$ "33Aq!9:  %%fdkkLVLD$!//))   r2)rHrFr)NNNF)rErFrGrHrr"rrrrJrA LongTensorrLrrr rBrMrNrOs@r0rDrD's { 5915-1). j u001j E--.j ))* j #' j +, j  j j r2rD)r)r rD)r)5rItypingrrrJr activationsrmodeling_layersrmodeling_outputsr r r modeling_utilsr r processing_utilsr pytorch_utilsrrutilsrrrr utils.genericrrconfiguration_vivitr get_loggerrEloggerModulerrQrKfloatrrrrrrrrrr r)rD__all__rr2r0ras% !9bbF&QKKA,   H %)RYY)XLbiiLn% II% <<% % << % U\\* %  %%>1.1.jbii$RYY> $ ")) +< @299 @ "))  4? 4 4Fj%jjD  y "6y  y x Pr2