cCi AJSrSSKrSSKJrJr SSKrSSKrSSKJr SSK J r J r J r SSK Jr SSKJrJrJr SS KJr SS KJr SS KJr SS KJrJr SS KJr SSKJrJ r J!r!J"r"J#r# SSK$J%r%J&r& SSK'J(r(J)r) SSK*J+r+ SSK,J-r-J.r. \)R^"\05r1Sr2S\RfS\4S\44Sjr5S~S\RfS\4S\\Rf4Sjjr6SS\7\4\44S\8S\4S\\RrS\4S \Rt4 S!jjr;"S"S#\5r<"S$S%\5r="S&S'\5r>"S(S)\R~5r@"S*S+\R~5rA"S,S-\R~5rB"S.S/\RR~5rC"S0S1\R~5rD"S2S3\R~5rE"S4S5\R~5rF"S6S7\R~5rG"S8S9\R~5rH"S:S;\R~5rI"S<S=\R~5rJ"S>S?\R~\%5rK"S@SA\R~\%5rL"SBSC\R~\%5rM"SDSE\R~5rN"SFSG\R~5rO"SHSI\5rP"SJSK\5rQ\("SLSM\&55rR"SNSO\R5rS"SPSQ\R5rT"SRSS\R5rU"STSU\R5rV"SVSW\R5rW"SXSY\R5rX"SZS[\R5rY"S\S]\R5rZ"S^S_\R~5r["S`Sa\R~5r\\("SbSc9"SdSe\R55r]\("SfSc9"SgSh\R\55r^SSi\RS\RSj\\RS\\RrSk\8Sl\8Sm\8Sn\\R~So\`Sp\`S \\R\7\R\R444Sqjjra\("SrSc9"SsSt\R55rb\("SuSc9"SvSw\R55rc"SxSy\R~5rd\("SzSc9"S{S|\&55re/S}Qrfg)zPyTorch SpeechT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossL1Loss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSeq2SeqSpectrogramOutput)EmbeddingAccessMixinPreTrainedModel)auto_docstringlogging)deprecate_kwarg)SpeechT5ConfigSpeechT5HifiGanConfig input_ids pad_token_iddecoder_start_token_idcURUR5nUSS2SS24R5USS2SS24'X#SS2S4'Uc [S5eUR US:HU5 U$)z) Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)r!r"r#shifted_input_idss h/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/speecht5/modeling_speecht5.pyshift_tokens_rightr-6sz"++IOO<(CRC0668ae4adLMM""#4#AB%%&:f&DcJ  //r.r' mask_prob mask_length min_masksreturnc ^^^^^UunmTS:a [S5eTT:a[STSTS35e[RRS5R 5mUUUUU4SjnUb-UR 5R S5R5O[U5Vs/sHnTPM snn[R"UT4[S 9n /n U"T5n U S :XaU $UHn U"U 5n [RR[R"U TS- - 5U S S 9n[U5S :XaTS- nOUS n[R"U[R"X- [R S 9U-/5nU R#U5 M [R$"U 5n [R&"U SS2SS2S4X[T45n U R)X[T-5n [R"T5SSSS24n[R&"UX[T45R)X[T-5nU U-n U R+5TS- :a TS- XTS- :'[R,"XSS5 U $s snf) a2 Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `c>[TU-T- T-5n[UT5nUT-T:aTT-nUTS- - U:a[UTS- - S5nU$)z;Given input length, compute how many spans should be maskedrr)intmax) input_lengthnum_masked_spanepsilonr8r7r9sequence_lengths r,compute_num_masked_span6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :-[TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl g)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r,rq%SpeechT5NoLayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [ R"URSS9Ul [URUlg)NrrrlT)elementwise_affine)rprqrrrsrtrrurvrwrxry LayerNorm layer_normr rzr{r|s r,rq#SpeechT5LayerNormConvLayer.__init__s [TU]5 US:aURUS- OSUlURUUl[ R "URURURUURUURS9Ul [URUl [ R"URURSS9Ulg)NrrrlT) num_groups num_channelsaffine)rprqrrrsrtrrurvrwrxryr rzr{ GroupNormrr|s r,rq#SpeechT5GroupNormConvLayer.__init__s [TU]5 SUlX lX0lUR XR-X#5 gN)rprqoffsetrr make_weights)r}rrrrs r,rq.SpeechT5SinusoidalPositionalEmbedding.__init__"s8  *& -++5}Rr.num_embeddingscURXU5n[US5(a8URURRURR S9nUR SUSS9 g)NweightsrGdeviceF persistent) get_embeddinghasattrtorrGrregister_buffer)r}rrr emb_weightss r,r2SpeechT5SinusoidalPositionalEmbedding.make_weights)s\(( T 4 # #%..t||/A/A$,,J]J].^K Y Fr.cUS-n[R"S5US- - n[R"[R"U[R S9R 5U*-5n[R"U[R S9R 5RS5URS5-n[R"[R"U5[R"U5/SS9RUS5nUS-S:Xa,[R"U[R"US5/SS9nUbSXBSS24'UR[R"55$) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'rrFrdimr%N)mathlogtorchexprTint64float unsqueezecatsincosviewrQrget_default_dtype)rrrhalf_dimembs r,r3SpeechT5SinusoidalPositionalEmbedding.get_embedding1s !A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#CQ vve--/00r.r!past_key_values_lengthcUR5up4URXRU5RUR5nURS-U-nX`R RS5:a3UR X`R-URUR5 UR RSURS55RX4S5R5$)Nrrr%) size"create_position_ids_from_input_idsrrrrrrr index_selectrrM)r}r!rbszseq_len position_idsmax_poss r,r-SpeechT5SinusoidalPositionalEmbedding.forwardCs ~~' >>yJZJZ\rsvv    ""Q&0 \\&&q) )   g 3T5G5GIYIY Z||((L,=,=b,ABGGVXY``bbr.cURU5R5n[R"USS9R U5U-U-nUR 5U-$)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rr)ner>rcumsumtype_aslong)r}r!rrmaskincremental_indicess r,rHSpeechT5SinusoidalPositionalEmbedding.create_position_ids_from_input_idsRsW||K(,,.$||Da8@@FI__cgg"'')K77r.)rrrrr)rrrr__doc__r>rrqr staticmethodrrno_gradTensorrrrrrs@r,rrsNScS#SHUXMSSG3GsGQYZ]Q^G1c1#1HUXM11" ]]_ c cs c cbc88478QYZ]Q^88r.rc.^\rSrSrU4SjrSrSrU=r$)SpeechT5PositionalConvEmbeddingidc>[TU]5 [R"URURUR UR S-UR S9Ul[RRn[[RRS5(a$[RRRn[5(Ga%SSK nURRURR SS9 U"URSSS9UlSSS5 [URS5(aUURRR R"nURRR R$nO,URR&nURR(nURR+X5 URR+X5 OU"URSSS9Ul[-UR 5Ul[0UR2Ulg!,(df  GN,=f) Nr)rmpaddinggroups weight_normr) modifier_rankweight)namerparametrizations)rprqrru hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsryutilsrrrr deepspeedzeroGatheredParametersr original0 original1weight_gweight_vregister_external_parameterSpeechT5SamePadLayerrr rzr{)r}r~rrrrrs r,rq(SpeechT5PositionalConvEmbedding.__init__es II      6622a777   hh** 288,,m < <((33??K % ' ' 224993C3CST2U' aH Vtyy"4559955<<FF9955<<FF99--99-- NN 6 6t F NN 6 6t F#DIIH!DDI+F,J,JK  !?!?@VUs I IcURSS5nURU5nURU5nURU5nURSS5nU$Nrr)rryrr{rs r,r'SpeechT5PositionalConvEmbedding.forwardsV%//15  -0  ]3  6 %//15 r.)r{ryrrrs@r,rrdsABr.rc6^\rSrSrSrSU4SjjrSrSrU=r$) SpeechT5ScaledPositionalEncodingiuS Scaled positional encoding, see §3.2 in https://huggingface.co/papers/1809.08895 c >[R"X25n[R"SU5RS5n[R"[R"SUS[R S9R 5[R"S5U- *-5n[R"UR 5U-5USS2SSS24'[R"UR 5U-5USS2SSS24'URS5n[TU]15 URSUSS9 [R"US 9UlX l[R$"[R&"S 55Ulg) NrrrrFg@peFrp?)rrQrTrrrrrrrrrprqrrDropoutdropoutr Parametertensoralpha)r}r rmax_lenrpositiondiv_termrs r,rq)SpeechT5ScaledPositionalEncoding.__init__s [[ &<<7+55a899U\\!S!5;;GMMOTXT\T\]dTehkTkRllmii 08 ;<1add7 ii 08 ;<1add7 \\!_  T2%8zzG, \\%,,s"34 r.cXRURSS2SURS524--nURU5nU$)Nr)rrrr )r}rs r,r(SpeechT5ScaledPositionalEncoding.forwards@JJMchhqkM)9!:::ll3 r.)rrr )i) rrrrrrqrrrrs@r,rrs 5r.rc2^\rSrSrSU4SjjrSrSrU=r$)"SpeechT5RelativePositionalEncodingic>[TU]5 XlX l[R R SU-U5Ulgr)rprqr max_lengthrr Embeddingpe_k)r}rrrs r,rq+SpeechT5RelativePositionalEncoding.__init__s4 $HH&&q:~s; r.ctURSn[R"SU5RUR[R S9nUSS2S4USSS24- nUR *X3UR *:'UR S- X3UR :'X0R -nURU5$)NrrrrG)r'rrTrrrrr)r}rrpos_seqs r,r*SpeechT5RelativePositionalEncoding.forwards%%a(,,q'*--]5I5IQVQ[Q[-\!T'"WT1W%55/3.>4??**+.2oo.A4??*+OO+yy!!r.)rrr)irrs@r,rrs< " "r.rc.^\rSrSrU4SjrSrSrU=r$)ricR>[TU]5 US-S:XaSUlgSUlg)Nrrr)rprqnum_pad_remove)r}rrs r,rqSpeechT5SamePadLayer.__init__s) #:Q#>!#Car.cXURS:aUSS2SS2SUR*24nU$Nrr"rs r,rSpeechT5SamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr.r&rrs@r,rrsKr.rc8^\rSrSrSrU4SjrSrSrSrU=r $)SpeechT5FeatureEncoderiz.Construct the features from raw audio waveformc >[TU]5 URS:Xa@[USS9/[ UR S- 5Vs/sHn[ XS-S9PM sn-nOVURS:Xa-[ UR 5Vs/sH n[XS9PM nnO[SURS35e[R"U5Ul SUl S Ul gs snfs snf) Ngroupr)rrlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rprqfeat_extract_normrrPnum_feat_extract_layersrirr)r ModuleList conv_layersgradient_checkpointing_requires_grad)r}r~ir0rs r,rqSpeechT5FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNNw,V!eDNwNK % % 0HMfNlNlHmHm1*6>Hm K01I1I0JJst ==5&+#"Ns C C%cNUR5H nSUlM SUlg)NF) parameters requires_gradr2)r}params r,_freeze_parameters)SpeechT5FeatureEncoder._freeze_parameterss#__&E"'E '#r.cUSS2S4nUR(aUR(aSUlURH nU"U5nM U$NT)r2trainingr7r0)r}r/r conv_layers r,rSpeechT5FeatureEncoder.forwardsK$QW-    4==*.M '**J&}5M+r.)r2r0r1) rrrrrrqr9rrrrs@r,r)r)s8#&$  r.r)c.^\rSrSrU4SjrSrSrU=r$)SpeechT5FeatureProjectionic4>[TU]5 [R"URSUR S9Ul[R"URSUR5Ul [R"UR5Ul g)Nr%eps) rprqrrrrlayer_norm_epsrLinearr projectionr feat_proj_dropoutr r}r~rs r,rq"SpeechT5FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r.cnURU5nURU5nURU5nX4$r)rrGr )r}rnorm_hidden_statess r,r!SpeechT5FeatureProjection.forwards7!__];(:;  ]3 00r.)r rrGrrs@r,rArAs< 11r.rAcL^\rSrSrU4SjrSrSS\RS\\RS\\R4Sjjr S\ S\R4S jr S \\R\ 44S jrSS \RS\\RS\\R4S jjrSrU=r$)SpeechT5SpeechEncoderPrenetic>[TU]5 Xl[U5Ul[ U5UlURS:dURS:aG[R"[R"UR5R55Ul[!U5Ul[%UR&UR(-S-URUR(5Ulg)Nr4r)rprqr~r)feature_encoderrAfeature_projectionmask_time_probmask_feature_probrr rrruniform_masked_spec_embedrpos_conv_embedrmax_speech_positionsr"pos_sinusoidal_embedrIs r,rq$SpeechT5SpeechEncoderPrenet.__init__s  5f=";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "=fE$I  ' '&*=*= = A      % !r.c8URR5 gr)rQr9r}s r,freeze_feature_encoder2SpeechT5SpeechEncoderPrenet.freeze_feature_encoders //1r.r/r1mask_time_indicescURU5nURSS5nUbURURSU5nUR U5upTUR XSUS9nUR U5nXV-nUb URS5R5nO;[R"URSS[RURS9nURU5nXX-nXR4$)Nrr)r_r1r) rQr"_get_feature_vector_attention_maskr'rR_mask_hidden_statesrWrrrrQrrY) r}r/r1r_extract_featuresrpositional_conv_embedding padding_mask positional_sinusoidal_embeddingss r,r#SpeechT5SpeechEncoderPrenet.forwards  // =+55a;  %!DD &&q)N +/*A*ABR*S' 00 ~1 %)$7$7 $F!%A  %),,Q/446L ;;}':':2A'>ejjYfYmYmnL+/+D+D\+R(%H ,,r.feature_vector_lengthcURSS9SS2S4nURU5R[R5nUR Sn[R "XQ4URURS9nSU[R"UR SURS9US- 4'URS/5RS5RS/5R5nU$)Nr%rrrrr) r _get_feat_extract_output_lengthsrrrr'rQrGrrTfliprR)r}rhr1non_padded_lengthsoutput_lengthsr^s r,ra>SpeechT5SpeechEncoderPrenet._get_feature_vector_attention_mask9s,22r2:1b5A>>?QRUUV[V`V`a#))!,   /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr.r`cSn[URRURR5Hup4U"XU5nM U$)z8 Computes the output length of the convolutional layers c8[R"X- USS9S-$)Nfloor) rounding_moder)rdiv)r@rmrns r,_conv_out_lengthVSpeechT5SpeechEncoderPrenet._get_feat_extract_output_lengths.._conv_out_lengthNs 99\7wWZ[[ [r.)zipr~rvrw)r}r`rurmrns r,rk'> @W@W#X K,]PM$Yr.rc[URSS5(dU$UR5upEnUb(URR UR 5X'OURR S:aUR(a[XE4URR URRUURRS9n[R"X!R[RS9nURR UR 5X'URRS:aUR(a[XF4URRURR URR"S9n[R"XqR[RS9nUSS2S4R%SUS5nSX'U$) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r7r8r1r9r)r7r8r9r%)getattrr~rrVrrGrSr=rgmask_time_lengthmask_time_min_masksrr rrRrTmask_feature_lengthmask_feature_min_masksexpand)r}rr_r1r^rCrmask_feature_indicess r,rb/SpeechT5SpeechEncoderPrenet._mask_hidden_statesYst{{$8$?? 4A3E3E3G0 [  (/3/E/E/H/HI\I\/]M , [[ ' '! + 5-++44 KK88-++99 ! !& ->G[G[chcmcm n /3/E/E/H/HI\I\/]M , ;; ( (1 ,#8)++77 KK;;++<< $ $)<<0DMaMainisis#t #74#@#G#GO]_#` 23M /r.)r~rQrRrVrWrYNN)rrrrrqr]rrr LongTensor FloatTensorrr>rarrkrbrrrs@r,rOrOs "2 6:9= -ll -!!1!12 -$E$5$56 -F  ]b]m]m  eEDTDTVYDY>Z &:>59 ,((,$E$5$56,!!1!12 ,,r.rOct^\rSrSrU4SjrSrSS\RS\\R4Sjjr Sr U=r $) SpeechT5SpeechDecoderPrenetic f>[TU]5 Xl[R"[ UR 5Vs/sH@n[R"US:Xa URO URUR5PMB sn5Ul [R"URUR5Ul [URURUR5Ul[R"UR"UR-UR5Ulgs snfr%)rprqr~rr/rPspeech_decoder_prenet_layersrF num_mel_binsspeech_decoder_prenet_unitslayersr final_layerrpositional_dropoutrXencode_positionsspeaker_embedding_dimspeaker_embeds_layerr}r~r3rs r,rq$SpeechT5SpeechDecoderPrenet.__init__s  mm vBBC   DA  +,6F''v7Y7Y66D    99V%G%GI[I[\ @  % %     ' '!  %'IIf.J.JVM_M_._agasas$t! sAD.c[R"USUS9nURS5RUR S5SS5n[R "US:HUS5S-SU- - $)Nrrr)r bernoullirrepeatrwhere)r} inputs_embedsrr all_maskss r,_consistent_dropout/SpeechT5SpeechDecoderPrenet._consistent_dropoutsd}Q/15NN1%,,]-?-?-BAqI {{9>=![TU]5 US:Xa URnO URnX!RS- :Xa URnO URn[ R "UUURSURS- S-SS9Ul[ R"U5Ul X!RS- :a[ R"5Ul OSUl [ R"UR5Ulg)NrrrF)rmrnrro)rprqrspeech_decoder_postnet_unitsspeech_decoder_postnet_layersrruspeech_decoder_postnet_kernelry BatchNorm1d batch_normTanhr{r speech_decoder_postnet_dropoutr )r}r~rrsrtrs r,rq#SpeechT5BatchNormConvLayer.__init__s  q= --K ==K ;;a? ?!..L!>>LII  <<99A=!C   ..6 ::Q> > ggiDO"DOzz&"G"GH r.cURU5nURU5nURbURU5nURU5nU$r)ryrr{r rs r,r"SpeechT5BatchNormConvLayer.forwardsJ -0  6 ?? & OOM:M ]3 r.)r{rryr rrrs@r,rrsI<r.rcl^\rSrSrU4SjrS\R 4SjrS\R 4SjrSr U=r $)SpeechT5SpeechDecoderPostnetic >[TU]5 Xl[R"UR UR UR-5Ul[R"UR UR5Ul [R"[UR5Vs/sHn[X5PM sn5Ulgs snfr)rprqr~rrFrrr0feat_outprob_outr/rPrrrrs r,rq%SpeechT5SpeechDecoderPostnet.__init__s   &"4"4f6I6IFLcLc6cd  &"4"4f6M6MN mm[TU]5 Xl[R"UR UR UR5Ul[URUR UR5Ul gr) rprqr~rr vocab_sizerr" embed_tokensrrmax_text_positionsrrIs r,rq"SpeechT5TextEncoderPrenet.__init__sc  LL):):F[TU]5 Xl[R"UR 5UlUR(a [R"UR5OSUl [R"URURUR5Ul[!UR"UR-S-URUR5Ulg)Nr r)rprqr~rr rr scale_embeddingrsqrtr embed_scalerrr"rrrembed_positionsrIs r,rq"SpeechT5TextDecoderPrenet.__init__s  zz&";";< [TU]5 Xl[R"UR UR SS9Ulg)NFro)rprqr~rrFrrlm_headrIs r,rq#SpeechT5TextDecoderPostnet.__init__;s3  yy!3!3V5F5FUS r.rc$URU5$rrrs r,r"SpeechT5TextDecoderPostnet.forward@s||M**r.cUR$rrr\s r,get_output_embeddings0SpeechT5TextDecoderPostnet.get_output_embeddingsCs||r.cXlgrrr}new_embeddingss r,set_output_embeddings0SpeechT5TextDecoderPostnet.set_output_embeddingsHs% r.)r~r) rrrrrqrrrrrrrrs@r,rr:s(T +U\\+ &&r.rc^\rSrSrSrSS\S\S\\S\\S\\S\\4 U4S jjjr \ "S S S S 9SS\ RS\\ RS \\ S\\ RS\\ RS\\ RS\S\\ RS\\ R\\ R\\ 44Sjj5rSrU=r$)SpeechT5AttentioniLz Multi-headed attention from 'Attention Is All You Need' paper with relative position bias (see https://aclanthology.org/N18-2074.pdf) embed_dim num_headsr  is_decoderro layer_idxc>[TU]5 XlX lX0lX-UlUR U-UR:wa[ SURSUS35eUR S-UlX@lX`l [R"XUS9Ul [R"XUS9Ul [R"XUS9Ul[R"XUS9Ulg)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gr)rprqrrr head_dimr)scalingrrrrFk_projv_projq_projout_proj)r}rrr rrorrs r,rqSpeechT5Attention.__init__Rs "" !. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 4@ ii 4@ ii 4@  )TB r.past_key_valuer4.58new_nameversionrkey_value_statesr1layer_head_mask position_biasoutput_attentionscache_positionr:c  USLn UR5upn URU5UR-n SnUb][U[5(aFUR R UR5nU (a URnOURnOUnU (aUOUnU (aQUbNU(aGWRURRnURURRnOURU5nURU5nURU SUR UR"5R%SS5nURU SUR UR"5R%SS5nUbcU (dUOSnWR'UUURSU05unnU (a.[U[5(aSUR UR'XR -SUR"4nU RXUR UR"5R%SS5n U R("U6n UR("U6nUR("U6nURS5n[*R,"U UR%SS55nUR5XR -U U4:wa.[/SXR -U U4S UR535eUbU R15RXR -SUR"5R%S S5n[*R2"UUR%S S55nUR%S S5RXR -URS 5URS55nUU- nUbzUR5U SU U4:wa#[/S U SU U4S UR535eURXR U U5U-nURXR -U U5n[4R6R9USS 9nUbUR5UR 4:wa*[/SUR 4S UR535eURSSSS5URXR U U5-nURXR -U U5nU(a=URXR U U5nURXR -U U5nOSn[4R6R;UUR:UR<S9n[*R,"UU5nUR5XR -XR"4:wa5[/SXR XR"4S UR535eURXR XR"5nUR%SS5nUR)XUR>5nURAU5nUU4$)z#Input shape: Batch x Time x ChannelNFr%rrrTz$Attention weights should be of size z , but is rrz!Attention mask should be of size rz/Head mask for a single layer should be of size )rr=z `attn_output` should be of size )!rrrrr is_updatedgetrcross_attention_cacheself_attention_cacherkeysvaluesrrrrrrupdater\rbmmr) contiguousmatmulrrsoftmaxr r=rr)r}rrrr1rrrris_cross_attentionrtgt_lenr_ query_statesrcurr_past_key_valuecurrent_states key_states value_states proj_shapesrc_len attn_weights reshape_q rel_pos_biasattn_weights_reshaped attn_probs attn_outputs r,rSpeechT5Attention.forwardos .T9',,.a{{=1DLL@   &/+>??,77;;DNNK %*9*O*O'*9*N*N'&5#-?)] /"=*,33DNNCHHJ.55dnnELLL^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*7It+>+E+E dnn?OQ_>`,( L&*_FY*Z*ZAEO..t~~>NN*B > #((t~~t}}U__`acde #++Z8 ''4 #++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6nn8LgW^7_6`a %%'(*   $$//166s^^7KRQUQ^Q^_iijkmnoI << =3J3J2r3RSL'11!Q7<<nn$m&8&8&;]=O=OPQ=RL L (L  %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S..'7SVddL',,S>>-A7GTL}},,\r,B  &##%$..):: Et~~FWEXY',,./1+//2q!<|?P?PQTVdVdfmov?wwL',,S>>-A7GTL  %1$5$5c>>7T[$\ !055cNN6JGU\]L$( !]]**<4<rrrRrqrrrr tuplerrrrs@r,rrLs^$'%*#$(CCC% C TN C tn CD>CC:%0A6R48+/152604"'152||2#5<<02"% 2 !. 2 "%,,/ 2  -2 2!.2 u||Xell3Xe_D E2S2r.rc.^\rSrSrU4SjrSrSrU=r$)SpeechT5FeedForwardic>[TU]5 [R"UR5Ul[R "URU5Ul[UR[5(a[URUl OURUl [R "X!R5Ul[R"UR5Ulgr)rprqrr activation_dropoutintermediate_dropoutrFrintermediate_denser hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropout)r}r~intermediate_sizers r,rqSpeechT5FeedForward.__init__s $&JJv/H/H$I!"$))F,>,>@Q"R f'' - -'-f.?.?'@D $'-'8'8D $II&79K9KL jj)>)>?r.cURU5nURU5nURU5nURU5nUR U5nU$r)r"r%r!r&r(rs r,rSpeechT5FeedForward.forwardsX// > 00? 11-@ ))-8 ++M: r.)r%r"r!r&r(rrs@r,rrs @r.rc ^\rSrSrS\4U4SjjrS S\RS\\RS\\RS\\RS\ 4 S jjr S r U=r $) SpeechT5EncoderLayeri r~c>[TU]5 [URURUR SS9Ul[R"UR5Ul [R"URURS9Ul [XR5Ul[R"URURS9Ulg)NF)rrr rrC)rprqrrencoder_attention_headsattention_dropout attentionrr r'r rrErrencoder_ffn_dim feed_forwardfinal_layer_normrIs r,rqSpeechT5EncoderLayer.__init__ s *((44,,   zz&"7"78 ,,v'9'9v?T?TU/8N8NO " V-?-?VEZEZ [r.rr1rrrcUnURUUUUUS9upURU5nXa-nURU5nXRU5-nUR U5nU4nU(aX4- nU$)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(config.encoder_attention_heads,)`. position_bias (`torch.FloatTensor`): relative position embeddings of size `(seq_len, seq_len, hidden_size // encoder_attention_heads)` output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rr1rrr)r2r rr4r5) r}rr1rrrresidualroutputss r,rSpeechT5EncoderLayer.forwards.!&*nn')+'/ '5' #  ]3  0  6 %(9(9-(HH --m<  "   &Gr.)r2r r4r5r)NNNF) rrrrrrqrrrrRrrrrs@r,r.r. sx \~ \ 262604"' ,||,!.,"%,,/ ,   - ,  ,,r.r.cT^\rSrSrSS\4U4Sjjjr\"SSSS9SS\RS \ \RS \ \RS \ \RS \ \RS \ \RS\ \ S\ \ S\ \ S\ \R4Sjj5r Sr U=r$)SpeechT5DecoderLayeriGr~ct>[TU]5 [URURUR SUS9Ul[R"UR5Ul [R"URURS9Ul [URURUR SUS9Ul[R"URURS9Ul[!XR"5Ul[R"URURS9Ulg)NT)rrr rrrC)r rr)rprqrrdecoder_attention_headsr1 self_attnrr r'r rrEself_attn_layer_norm encoder_attnencoder_attn_layer_normrdecoder_ffn_dimr4r5)r}r~rrs r,rqSpeechT5DecoderLayer.__init__Hs *((44,,  zz&"7"78 $&LL1C1CI^I^$_!-     * *,,  (*||F4F4FFLaLa'b$/8N8NO " V-?-?VEZEZ [r.rrrrrr1encoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskr use_cacherc rUn URUUUUUU S9upURU5nX-nURU5nSn Ub?Un URUUUUUUU S9upURU5nX-nUR U5nXR U5-nUR U5nU4nU(aXU 4- nU$)ay Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, hidden_size)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, hidden_size)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrr1rrrN)rrr1rrrr)r?r r@rArBr4r5)r}rr1rErFrrGrrrHrr8self_attn_weightscross_attn_weightsr9s r,rSpeechT5DecoderLayer.forward`s@!,0>>'+)+/) ,:, (  ]3  0 11-@ " ,$H040A0A+!65 : /"3-1B1 -M!LL7M$4M 88GM&(9(9-(HH --m<  "  +=> >Gr.)r rArBr4r5r?r@r) NNNNNNFTN)rrrrrrqrrrrr rRrrrrs@r,r<r<Gs\~\\0%0A6R268<9=26=A+/,1$(15I||I!.I ( 5 I !) 6 I "%,,/ I%-U\\$:I"%I$D>ID>I!.ISIr.r<cN\rSrSr%\\S'SrSrSrS\ R4Sjr Sr g ) SpeechT5PreTrainedModelir~speecht5r/Tmodulec |URRn[U[5(a[R R URRSS[R"SURRSURR-- 5-S9 [R RURRS5 GO%[U[5(a'UR R"R%S5 GO[U[&5(a[R"SUR(R*- 5n[R R-UR(RU*US9 [R R-UR(RU*US9 GO=[U[R.5(aWURR"R SUS9 URb$URR"R15 GO[U[R2[R4[R645(aKURR"R15 URR"R%S5 GO>[U[R85(a[R R;UR5 URbg[R"UR<URURS-- 5n[R R-URU*US9 O[U[R>5(abURR"R SUS9 UR@b1URR"UR@R15 [CUS 5(a*[R R-URD5 gg) zInitialize the weightsrrrmeanstdr )abr4NrV)#r~initializer_rangerrrinitnormal_ryrrrrm in_channels constant_rorrdatafill_rArG in_featuresrUrFzero_rrrrukaiming_normal_rrrrrV)r}rPrTks r, _init_weights%SpeechT5PreTrainedModel._init_weightsskk++ f= > > GGOO "" !v{{'>'>q'AFKKD[D['["\]]   GG  fkk.. 2  @ A A LL   # #C (  9 : : !f//;;; 6. / / GG  V55 6 0r.N) rrrrr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointingrModulerbrrdr.r,rNrNs) "$O&*#"7BII"7r.rNc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5Encoderizm Transformer encoder consisting of *config.encoder_layers* layers. Each layer is a [`SpeechT5EncoderLayer`]. r~c>[TU]U5 [R"URUR S9Ul[R"UR5Ul URUl [R"[UR5Vs/sHn[U5PM sn5Ul[#URUR$-UR&5UlSUlUR-5 gs snf)NrCF)rprqrrrrErr r'r encoder_layerdrop layerdropr/rPencoder_layersr.rrr0encoder_max_relative_positionrr1 post_init)r}r~r_rs r,rqSpeechT5Encoder.__init__s  ,,v'9'9v?T?TUzz&"7"78 11mm5QWQfQfKg$hKga%9&%AKg$hi A   &"@"@ @&BfBf ',# %isDrr1 head_maskroutput_hidden_states return_dictr:c UbUOURRnUbUOURRnUbUOURRnUb[ X!R 5nUR U5nURU5nURU5n[5=(d [U5nU(aSOSn U(aSOSn Ub`UR5S[UR5:wa6[S[UR5SUR5SS35e[UR5HupU(aX4-n Sn UR (a$["R$"/5nXR&:n U (aU(aU "UUUUbX;OSUS9nUSnU (aS nU(dM~U WS 4-n M U(aX4-n U(d[)S XU 455$[+UU U S 9$) aA Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the encoder prenet. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing convolution and attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nrdrz&The head_mask should be specified for  layers, but it is for .F)r1rrrrrc3.# UH ocMUv M g7frrd.0vs r, *SpeechT5Encoder.forward..Osm$[q$[s last_hidden_stater attentions)r~rrtuse_return_dictrrGrr rrrrrUrr) enumerater=rrKrnrr)r}rr1rsrrtrur synced_gpusall_hidden_statesall_self_attentionsidx encoder_layerskip_the_layerdropout_probability layer_outputss r,rSpeechT5Encoder.forwardsH2C1N-TXT_T_TqTq$8$D $++JjJj &1% f 'tn f d^f  uo% &f f r.rkc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5EncoderWithSpeechPrenetiXz Wrapper around SpeechT5Encoder that applies SpeechT5SpeechEncoderPrenet to convert the audio waveform data to hidden features. r~c>[TU]U5 [U5Ul[ U5UlUR 5 gr)rprqrOprenetrkwrapped_encoderrqrIs r,rq(SpeechT5EncoderWithSpeechPrenet.__init__^5  1&9 .v6 r.r/r1rsrrtrur:c TURX5uprURUUUUUUS9nU$N)rr1rsrrtrurr r}r/r1rsrrtrurr9s r,r'SpeechT5EncoderWithSpeechPrenet.forwardfsC)- L(Q% &&')/!5# ' r.rrrrs@r,rrXs ~26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r.rc^\rSrSrSrS\4U4SjjrSrSrSS\ RS\ \ RS \ \ RS \ \ S \ \ S \ \ S \\\44SjjrSrU=r$)SpeechT5EncoderWithTextPreneti}zt Wrapper around SpeechT5Encoder that applies SpeechT5TextEncoderPrenet to convert the input_ids to hidden features. r~c>[TU]U5 [U5Ul[ U5UlUR 5 gr)rprqrrrkrrqrIs r,rq&SpeechT5EncoderWithTextPrenet.__init__5  /7 .v6 r.c6URR5$rrget_input_embeddingsr\s r,r2SpeechT5EncoderWithTextPrenet.get_input_embeddings{{//11r.c:URRU5 grrset_input_embeddingsr}values r,r2SpeechT5EncoderWithTextPrenet.set_input_embeddings ((/r.r/r1rsrrtrur:c PURU5nURUUUUUUS9nU$rrrs r,r%SpeechT5EncoderWithTextPrenet.forwards@ L1 &&')/!5# ' r.rr)rrrrrrrqrrrrrrrRrrrrrrrs@r,rr}s~20 26,0,0/3&*''!.ELL)  $D>  'tn d^ uo% &r.rc^\rSrSrSrS\4U4SjjrSS\RS\ \RS\ \RS\ \ S \ \ S \ \ S \ \ \44S jjrS rU=r$)SpeechT5EncoderWithoutPreneti This wrapper class is a helper class to correctly load pretrained checkpoints when used in combination with [`SpeechT5Model`]. r~cd>[TU]U5 [U5UlUR 5 gr)rprqrkrrqrIs r,rq%SpeechT5EncoderWithoutPrenet.__init__)  .v6 r.r/r1rsrrtrur:c *URUUUUUUS9$rr)r}r/r1rsrrtrus r,r$SpeechT5EncoderWithoutPrenet.forwards.##&)/!5# $  r.rrrrs@r,rrs ~26,0,0/3&* '' !. ELL)  $D>  'tn  d^  uo% &  r.rcl^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\\ S\\RS\\\44SjjrSrU=r$)SpeechT5Decoderizl Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`SpeechT5DecoderLayer`] r~c  >[TU]U5 URUl[R "[ UR5Vs/sH n[XS9PM sn5Ul SUl UR5 gs snf)N)rF) rprqdecoder_layerdroprnrr/rPdecoder_layersr<rr1rqrs r,rqSpeechT5Decoder.__init__sl  11mmX]^d^s^sXt$uXtST%9&%NXt$uv &+#  %vs Brr1rErFrscross_attn_head_maskrrHrrtrurr:c ZU bU OURRn U bU OURRn UbUOURRnU bU OURRn UR 5SSn UR (a/UR(aU(a[RS5 SnU(a1Uc.[[URS9[URS95nU(a@[U[5(a+[RS5 [R"U5nUbUR5OSn[!X-X5nUbUb[#XAR$U SS9n['5=(d [)U5nU (aS OSnU (aS OSnU (aUbS OSn[+XV/S S /5HnunnUcM UR 5S[-UR.5:wdM7[1S US [-UR.5SUR 5SS35e [3UR.5HunnU (aUU4-nSnUR(a%[4R6"/5nUUR8:nU(a U(dM[U"UUUUUbUUOSUbUUOSUU UU S9 nUSnU (dMUUS4-nUcMUUS4-nM U (aUU4-nU (d[SXUUU455$[;UUUUUS9$)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, feature_size)`): Features extracted from the speech or text input by the decoder prenet. attention_mask (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr%zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r~zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.r)r rdrsrzThe `z` should be specified for rwrx)rFrrGrrrHrrrc30# UH nUcMUv M g7frrdrzs r,r}*SpeechT5Decoder.forward..yswAws )rrrrcross_attentions)r~rrtrHrrr1r=logger warning_oncer r rrfrom_legacy_cacherrrrGrrrwrUrr)rrrKrnr)r}rr1rErFrsrrrHrrtrurrrrrrall_cross_attentions attn_mask mask_namer decoder_layerrrrs r,rSpeechT5Decoder.forwardsRP2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK>#A&3t{{+;<$ {*DSEUDVW%NN,Q/03%r#,DKK"8 C#$58H$H!#N}}&+jjn#!4t~~!Ek)%'=3<3H3dI]Ii,@,Eos /"3#- M*!,M  &9]1=M 1]4D D ':KM`bvw  9+++*1   r.)r1rnr NNNNNNNNNNNNrrrrrrrqrrrrrr rRrrrrrrrs@r,rrs9 ~ 6:59=A=A,07;+/$(,0/3&*15n  1 12n !!1!12n  ((9(9: n !))9)9 : n ELL) n 'u||4n "%n D>n $D>n 'tnn d^n !.n  u?? @n n r.rc^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\RS \\ S \\ S\\ S\\ S\\ S\\RS\\\44SjjrSrU=r$)SpeechT5DecoderWithSpeechPrenetiz| Wrapper around SpeechT5Decoder that applies SpeechT5SpeechDecoderPrenet to convert log-mel filterbanks to hidden features. r~c>[TU]U5 [U5Ul[ U5UlUR 5 gr)rprqrrrwrapped_decoderrqrIs r,rq(SpeechT5DecoderWithSpeechPrenet.__init__rr.r/r1rErFrrsrrrHrrtrurr:c\URX5nURUUUUUUUU U U U U S9 nU$N) rr1rErFrsrrrHrrtrurrr)r}r/r1rErFrrsrrrHrrtrurdecoder_hidden_statesr9s r,r'SpeechT5DecoderWithSpeechPrenet.forwardsS !% L M&&/)"7#9!5+/!5#)'  r.r) NNNNNNNNNNNNNrrs@r,rrs@ ~5959=A=A59,07;+/$(,0/3&*15!u001!!!1!12! ((9(9: ! !))9)9 : ! %U\\2 !ELL)!'u||4!"%!D>!$D>!'tn!d^!!.! u?? @!!r.rcx^\rSrSrSrS\4U4SjjrSrSrSS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \S\ \S\ \S\ \S\ \S\ \ RS\\\44SjjrSrU=r$)SpeechT5DecoderWithTextPrenetizs Wrapper around SpeechT5Decoder that applies SpeechT5TextDecoderPrenet to convert input tokens to hidden features. r~c>[TU]U5 [U5Ul[ U5UlUR 5 gr)rprqrrrrrqrIs r,rq&SpeechT5DecoderWithTextPrenet.__init__rr.c6URR5$rrr\s r,r2SpeechT5DecoderWithTextPrenet.get_input_embeddingsrr.c:URRU5 grrrs r,r2SpeechT5DecoderWithTextPrenet.set_input_embeddingsrr.r/r1rErFrsrrrHrrtrurr:c bURXU5upURU UUUUUUUU U U U S9 nU$rr)r}r/r1rErFrsrrrHrrtrurrr9s r,r%SpeechT5DecoderWithTextPrenet.forwardsY15 LZi0j-&&/)"7#9!5+/!5#)'  r.rr)rrrrrrrqrrrrrrrr rRrrrrrrrs@r,rrs4~20 5959=A=A,07;+/$(,0/3&*15 u001 !!1!12  ((9(9:  !))9)9 :  ELL) 'u||4 "% D> $D> 'tn d^ !.  u?? @  r.rcl^\rSrSrSrS\4U4SjjrSS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\\ S\\ S\\RS\\\44SjjrSrU=r$)SpeechT5DecoderWithoutPrenetirr~cd>[TU]U5 [U5UlUR 5 gr)rprqrrrqrIs r,rq%SpeechT5DecoderWithoutPrenet.__init__rr.r/r1rErFrsrrrHrrtrurr:c :URUUUUUUUUU U U U S9 n U $rr)r}r/r1rErFrsrrrHrrtrurr9s r,r$SpeechT5DecoderWithoutPrenet.forwardsD&&&)"7#9!5+/!5#)'  r.rrrrs@r,rrs* ~5959=A=A,07;+/$(,0/3&*15u001!!1!12 ((9(9:  !))9)9 :  ELL) 'u||4"%D>$D>'tnd^!. u?? @r.rc^\rSrSrSrS\4U4SjjrS\RS\RS\RS\R4S jr S r \ S 5rS rU=r$) $SpeechT5GuidedMultiheadAttentionLossiz Guided attention loss from the paper [Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention](https://huggingface.co/papers/1710.08969), adapted for multi-head attention. r~cf>[TU]5 URUlURUlgr)rprqguided_attention_loss_sigmasigmaguided_attention_loss_scalescalerIs r,rq-SpeechT5GuidedMultiheadAttentionLoss.__init__#s( 77 77 r.r input_masks output_masksr:cDURX#UR5nURS5URS5-nURUR5RS5nXA-n[R "UR U55nURU-$)a Compute the attention loss. Args: attentions (`torch.FloatTensor` of shape `(batch_size, layers * heads, output_sequence_length, input_sequence_length)`): Batch of multi-head attention weights input_masks (`torch.BoolTensor` of shape `(batch_size, input_sequence_length)`): Input attention mask as booleans. output_masks (`torch.BoolTensor` of shape `(batch_size, output_sequence_length)`): Target attention mask as booleans. Returns: `torch.Tensor` with the loss value r%rr)_make_guided_attention_masksrrrrrS masked_selectr)r}rrrguided_attn_masksmaskslosseslosss r,r,SpeechT5GuidedMultiheadAttentionLoss.forward(s"!==kYcYjYjk&&r*[-B-B2-FF**+55a8"/zz&..u56zzD  r.cjURS5nURS5n[R"[U5URSURS4US9n[ [ XE55H.unupURXURU5XgSU 2SU24'M0 URS5$)Nr%rrj) rNrrQrUr'rrw_make_guided_attention_maskrr) r}rrrr`rnrrilenolens r,rASpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_masksAs#+ %))"-!KK[)9<;M;Ma;PR]RcRcdeRf(gpvw!*3}+M!N C$373S3STX`d`j`jlr3s 5D5%4%/ 0"O!**1--r.c[R"[R"XS9[R"XS9SS9upEUR5U- nUR5U- nS[R"XE- S-*SUS--- 5- $)Nrjxy)indexingr r)rmeshgridrTrr)r@ output_lengthrrgrid_ygrid_xs r,r@SpeechT5GuidedMultiheadAttentionLoss._make_guided_attention_maskLsz LL 5 LL 6  -/,.UYY&/a!78ANKLLLr.)rr)rrrrrrrqrr BoolTensorrrrrrrrrs@r,rrsj 8~8 !++!:?:J:J!Z_ZjZj! !2 .MMr.rc^\rSrSrSrS\4U4SjjrSS\RS\RS\RS\RS \RS \ \RS \R4S jjr S r U=r$)SpeechT5SpectrogramLossiXz3 Loss computation used by SpeechT5ForTextToSpeech. r~c.>[TU]5 URUlURUlURUl[ 5Ul[[R"S5S9Ul UR(a[U5Ul gg)Ng@) pos_weight) rprquse_guided_attention_lossguided_attention_loss_num_headsr0r l1_criterionrrr  bce_criterionrattn_criterionrIs r,rq SpeechT5SpectrogramLoss.__init__]ss )/)I)I&/5/U/U, & 7 7"H.%,,s:KL  ) )"Fv"ND  *r.r1rrrlabelsrr:c VUS:gnURU5nURU5nURU5nURX55URX%5-nUSS2SS2S4n [R"U )S-[R"U R S5S5R U R5/SS9n U SS2SS24RU 5n URU 5nURXJ5n X-n UR(a[R"UV s/sHoSS2SUR24PM sn SS9nUS:HnUSS2SS2S4nURS:a#USS2URS- SUR24nURXU5nU U- n U $s sn f)Nr3rr rr) rr rrrWrrrrr r r0r)r}r1rrrrrrel1_lossr stop_labelsbce_lossrxattnrr attn_losss r,rSpeechT5SpectrogramLoss.forwardis' %%l3!7!E!El!S 5 C CL Q##$9BTEVEVWmEvvQ1W%ii%#uzz%**Q-/K/N/Nu||/\ ]cde !!QR%(66u= %%e,%%f:!  ) )99TdeTdq#IT%I%I#I IJTdeklmD(A-K'1a0L$$q(+At/D/Dq/H/aDLaLa/a,ab ++D|LI I D fs%F&)rrr r r0r r)rrrrrrrqrrrrrrrrrs@r,rrXs O~ O&9=)(()!& 1 1) %00 ) !! ) !! )#5#4#45) ))r.rzv The bare SpeechT5 Encoder-Decoder Model outputting raw hidden-states without any specific pre- or post-nets.  custom_introc$T^\rSrSrSS\S\\RS\\R4U4SjjjrSr Sr Sr S r \ SS \\RS \\R S \\RS \\R S\\R"S\\R"S\\RS\\\\R"S\\S\\S\\R"S\\S\\S\\S\\RS\\\R"\44 Sjj5rSrU=r$) SpeechT5Modelir~encoderdecoderc>[TU]U5 XlUc [U5OUUlUc [ U5OUUlUR5 g)z encoder (`PreTrainedModel`, *optional*): The encoder model to use. decoder (`PreTrainedModel`, *optional*): The decoder model to use. N)rprqr~rrrrrq)r}r~rrrs r,rqSpeechT5Model.__init__sK   ?F3F;T[ ?F3F;T[  r.c[UR[5(aURR5$[UR[ 5(aURR5$[ er)rrrrrrNotImplementedErrorr\s r,r"SpeechT5Model.get_input_embeddingssR dll$A B B<<446 6 dll$A B B<<446 6!!r.c[UR[5(aURRU5 [UR[ 5(aURRU5 ggr)rrrrrrrs r,r"SpeechT5Model.set_input_embeddingssP dll$A B B LL - -e 4 dll$A B B LL - -e 4 Cr.cUR$r)rr\s r, get_encoderSpeechT5Model.get_encoders ||r.c[UR[5(a%URRR 5 ggz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rrrrr]r\s r,r]$SpeechT5Model.freeze_feature_encoders2 dll$C D D LL   6 6 8 Er.r/r1decoder_input_valuesdecoder_attention_maskrsdecoder_head_maskrencoder_outputsrrHrrrtrurr:cU bU OURRn U bU OURRn U bU OURRn UbUOURRnUcUR UUUU U US9nORU(aK[ U[5(d6[US[U5S:aUSOS[U5S:aUSOSS9nUbV[ UR [5(a7UR RRUSRSU5nOUn[ UR[5(aSU 0nO0nUR"S UUUSUUUU U U U UUS. UD6nU(dUU-$[UR UR"UR$UR&UR(UR UR$UR&S 9$) a1 input_values (`torch.Tensor` of shape `(batch_size, sequence_length)`): Depending on which encoder is being used, the `input_values` are either: float values of the input raw speech waveform, or indices of input sequence tokens in the vocabulary, or hidden states. decoder_input_values (`torch.Tensor` of shape `(batch_size, target_sequence_length)`, *optional*): Depending on which decoder is being used, the `decoder_input_values` are either: float values of log-mel filterbank features extracted from the raw speech waveform, or indices of decoder input sequence tokens in the vocabulary, or hidden states. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. N)r/r1rsrrtrurrrrr) r/r1rErFrsrrrHrrtrur)rrrdecoder_attentionsrencoder_last_hidden_staterEencoder_attentionsrd)r~rrtrHrrrrrUrrrar'rrrrrrrr)r}r/r1r-r.rsr/rr0rrHrrrtrurrF decoder_argsdecoder_outputss r,rSpeechT5Model.forwardsT2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK,0/3&*15!k u||,k !!1!12k 'u||4 k !))9)9 : k E--. k $E$5$56k 'u||4k "%e.?.?(@"ABk "%k D>k %U%6%67k $D>k 'tnk d^k !.!k " uU&&');; <#k k r.rzB SpeechT5 Model with a speech encoder and a text decoder. c$^\rSrSrS/rS\4U4SjjrSrSrSr Sr S r \ SS \ \RS \ \R S \ \R S \ \R S\ \RS\ \RS\ \R"S\ \\\RS\ \S\ \S\ \S\ \S\ \S\ \R S\ \R"S\\\44 Sjj5rSrU=r$)SpeechT5ForSpeechToTexti6z#text_decoder_postnet.lm_head.weightr~c>[TU]U5 URc[SURS35e[ U5n[ U5n[XU5Ul[U5Ul UR5 g)NYou are trying to instantiate a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForSpeechToText.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rprqrr)rrrrrOrtext_decoder_postnetrq)r}r~speech_encoder text_decoderrs r,rq SpeechT5ForSpeechToText.__init__>s}     $00@A// 9@4V< %flK $>v$F! r.c6URR5$rrOr(r\s r,r(#SpeechT5ForSpeechToText.get_encoderR}}((**r.c6URR5$rrO get_decoderr\s r,rG#SpeechT5ForSpeechToText.get_decoderUrDr.cTUR5RR5 gr+r(rr]r\s r,r].SpeechT5ForSpeechToText.freeze_feature_encoderX !!88:r.c6URR5$r)r=rr\s r,r-SpeechT5ForSpeechToText.get_output_embeddings_s((>>@@r.c:URRU5 gr)r=rrs r,r-SpeechT5ForSpeechToText.set_output_embeddingsbs !!77Gr.r/r1decoder_input_idsr.rsr/rr0rrHrrtrurrr:cU bU OURRn Ub7Uc4[XRRURR5nUR UUUUUUUUU U U U SUS9nUR US5nSnUbF[5nU"URSURR5URS55nU (dU4USS-nUbU4U-$U$[UUURURURURURUR UR"S9 $)a` input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) SpeechT5 uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Label indices can be obtained using [`SpeechT5Tokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToText >>> from datasets import load_dataset >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_asr") >>> model = SpeechT5ForSpeechToText.from_pretrained("microsoft/speecht5_asr") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> predicted_ids = model.generate(**inputs, max_length=100) >>> # transcribe speech >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> transcription[0] 'mister quilter is the apostle of the middle classes and we are glad to welcome his gospel' ``` ```python >>> inputs["labels"] = processor(text_target=dataset[0]["text"], return_tensors="pt").input_ids >>> # compute loss >>> loss = model(**inputs).loss >>> round(loss.item(), 2) 19.68 ``` NT)r/r1r-r.rsr/rr0rrHrrtrurrr%r) rrrrr2rr3rEr4)r~rr-r"r#rOr=rrrrrrr2rr3rEr4)r}r/r1rQr.rsr/rr0rrHrrtrurrr9rrloss_fctoutputs r,rSpeechT5ForSpeechToText.forwardes[v&1%H $D>H 'tnH d^H ))*H !.!H " uo% &#H H r.r:modelr threshold minlenratio maxlenratiovocoderoutput_cross_attentionsreturn_output_lengthsc  Uc [S5eUc*SXRR:HR5- n OUn UR S5n UR R UU SS9n U Rn [UR R [5(a@UR R RRU SRSU 5n [U R S5U-URR- 5n[U R S5U-URR- 5nU RU SURR5n/n/nSnSn0nUS- nUR R RUU5nUR R R#USS2SS24SU U USUSS9nU(a.UR%[&R("UR*SS95 URR-S5nUR.nUR0R3U5nUR5XRRURR5nUR%U5 USS2SSS24R5U SURR5n[&R("UU4SS9n[&R6"UR0R9U55nUU:aGMUU:a@[&R:"USS9U:n[&R<"U5SR?5nO[A[CU55nUVs/sH nUU;dM UPM nn[CU5S:ad[&RD"U5nURGSS5RISS 5nUR0RKU5nUH n UU UU 'M [CU5U :aOGM[A[CU55Vs/sHnUUPM nnU (dU S:XaUSO1[&RLRNRPRSUSS 9nUb U"U5n!OUn!U(a_[&R("US S9nU S:a@UR4"U [UR S5U - 5/UR 5S SQ76nU!U4n!U!$/n"[AU 5H&nU"R%UUR S55 M( Uc7[&RLRNRPRSUSS 9nUU"4n!Oy/n#[&RLRNRPRSUSS 9nU"U5n#U"Vs/sH,n[U#R S5[UU"5- 5U-PM. n$nU#U$4n!U(a\[&R("US S9nUR4"U [UR S5U - 5/UR 5S SQ76n/U!QUP7n!U!$s snfs snfs snf) Na`speaker_embeddings` must be specified. For example, you can use a speaker embeddings by following the code snippet provided in this link: https://huggingface.co/datasets/Matthijs/cmu-arctic-xvectors rrT)r/r1rur%)rr1rErFrrHrrurr) batch_first)+r)r~r"r>rrOrrrrrrar'r0r&rrrrYrrrsqueezerspeech_decoder_postnetrrsigmoidrrNrrOrPrUstackrflattenrrrrnn pad_sequencer?)%rWr/rr1rXrYrZr[r\r]rFr encoder_outr3maxlenminlenoutput_sequence spectrogramrrrresult_spectrogramr decoder_outlast_decoder_outputspectrumnew_spectrogramprobmeet_thresholds meet_indexesr3 spectrograms meet_indexr9spectrogram_lengths waveformswaveform_lengthss% r,_generate_speechrzs!   !"lll6O6O&O%T%T%V!V!/   A C..((!-)K !, = =%..((*IJJ!&!7!7!>!>!a!a N  #%;" *//2[@5<EEG $SY/ '3S|!q@R7RA|LS< 1$${{;7 +55a;CCAqI $;;CCLQ ".J5A*5M&z2#/%&#-i j49=O9P3QR3Qa&q)3QLR ),l1ouxx~~7I7I7V7VWcqu7V7v  k*G!G "$yy)9qA Qw#3#8#8-2215;<$?O?T?T?VWYWZ?[$  01G* N%!sA  & &|A';';A'> ? ? 88>>--::>--::[TU]U5 URc[SURS35e[ U5n[ U5n[XU5Ul[U5Ul UR5 g)Nr<a with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SpeechT5ForTextToSpeech.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) rprqrr)rrrrrOrrbrq)r}r~ text_encoderspeech_decoderrs r,rq SpeechT5ForTextToSpeech.__init__ s}     $00@A// 5V< 8@%fNK &B6&J# r.r:cgr<rd)clss r, can_generate$SpeechT5ForTextToSpeech.can_generate s r.c6URR5$rrBr\s r,r(#SpeechT5ForTextToSpeech.get_encoder rDr.c6URR5$rrFr\s r,rG#SpeechT5ForTextToSpeech.get_decoder rDr.r1r-r.rsr/rr0rrHrrtrurrrrcrU bU OURRn UbBUc"[XRRU5up4URR(aSn UR UUUUUUUUU U UU U SUS9nUR US5unnnSnUb,[UR5nU"UUUUUUR5nU (dU4USS-nUbU4U-$U$[UUURURURURURURURS9 $)aH input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`SpeechT5Tokenizer`]. See [`~PreTrainedTokenizer.encode`] and [`~PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Timesteps set to `-100.0` are ignored (masked) for the loss computation. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan, set_seed >>> import torch >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts") >>> model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> inputs = processor(text="Hello, my dog is cute", return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate(inputs["input_ids"], speaker_embeddings=speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([15872]) ``` NTr/r1r-r.rsr/rr0rrHrrrtrurrr rrlrrr2rr3rEr4)r~rr6r0r rOrbrrrrrr2r3rEr4)r}r!r1r-r.rsr/rr0rrHrrtrurrrrr9rrrr criterionrTs r,rSpeechT5ForTextToSpeech.forward s{Z&1%.2.215%D E,,-D !!1!12D 'u'8'89 D !))9)9 : D E--. D $E$5$56D 'u||4D "%e.?.?(@"ABD "%D D>D $D>D 'tnD d^D %U%6%67D **+!D "ell+#D $!.%D & u.. /'D D L ]]_6::> !'+(-&+Y ##Y !!1!12Y %U%6%67 Y  Y  Y Y "))$Y "&Y  $Y  u  %(9(95;L;L(L"MM NY Y v ]]_;?59 !'+(-&+] ##] %U%6%67] !!1!12 ]  ]  ] ] "))$] "&]  $]  u  %(9(95;L;L(L"MM N] ] r.r|zD SpeechT5 Model with a speech encoder and a speech decoder. c(^\rSrSrS\4U4SjjrSrSrSr\ S"S\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \\\ RS\ \S\ \S\ \S\ \S\ \S\ \ RS\ \ RS\ \ RS\ \ RS\\\44$Sjj5r\ R*"5S#S\ RS\ \ RS\ \ RS\S\S\S\ \R0S\S\S\ R4S jj5rS!rU=r$)$SpeechT5ForSpeechToSpeechi r~c>[TU]U5 [U5n[U5n[ XU5Ul[ U5UlUR5 gr) rprqrrrrOrrbrq)r}r~r>rrs r,rq"SpeechT5ForSpeechToSpeech.__init__ sK  8@8@%fnM &B6&J# r.c6URR5$rrBr\s r,r(%SpeechT5ForSpeechToSpeech.get_encoder rDr.c6URR5$rrFr\s r,rG%SpeechT5ForSpeechToSpeech.get_decoder rDr.cTUR5RR5 gr+rJr\s r,r]0SpeechT5ForSpeechToSpeech.freeze_feature_encoder rLr.r/r1r-r.rsr/rr0rrHrrtrurrrrr:cU bU OURRn Ub%Uc"[XRRU5up4UR UUUUUUUUU U UU U SUS9nUR US5unnnSnU (dU4USS-nUbU4U-$U$[ UUURURURURURURURS9 $)a[ input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a *.flac* or *.wav* audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`SpeechT5Processor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SpeechT5Processor.__call__`] for details. decoder_input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`): Float values of input mel spectrogram. SpeechT5 uses an all-zero spectrum as the starting token for `decoder_input_values` generation. If `past_key_values` is used, optionally only the last `decoder_input_values` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_values`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`SpeechT5Decoder._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. speaker_embeddings (`torch.FloatTensor` of shape `(batch_size, config.speaker_embedding_dim)`, *optional*): Tensor containing the speaker embeddings. labels (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.num_mel_bins)`, *optional*): Float values of target mel spectrogram. Spectrograms can be obtained using [`SpeechT5Processor`]. See [`SpeechT5Processor.__call__`] for details. stop_labels (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Binary tensor indicating the position of the stop token in the sequence. Example: ```python >>> from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan, set_seed >>> from datasets import load_dataset >>> import torch >>> dataset = load_dataset( ... "hf-internal-testing/librispeech_asr_demo", "clean", split="validation" ... ) # doctest: +IGNORE_RESULT >>> dataset = dataset.sort("id") >>> sampling_rate = dataset.features["audio"].sampling_rate >>> processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_vc") >>> model = SpeechT5ForSpeechToSpeech.from_pretrained("microsoft/speecht5_vc") >>> vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan") >>> # audio file is decoded on the fly >>> inputs = processor(audio=dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt") >>> speaker_embeddings = torch.zeros((1, 512)) # or load xvectors from a file >>> set_seed(555) # make deterministic >>> # generate speech >>> speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder) >>> speech.shape torch.Size([77824]) ``` NTrrrr)r~rr6r0rOrbrrrr2rr3rEr4)r}r/r1r-r.rsr/rr0rrHrrtrurrrrr9r_rlrrrTs r,r!SpeechT5ForSpeechToSpeech.forward s(h&1%Q>Q!R          # !  r.rrr)rrrrrrqr(rGr]rrrrrrrr rRrrrrrrrirrrrs@r,rr s ~ ++;5959<@=A159=7;EI+/$(,0/3&*:>.2.215% u001 !!1!12 'u'8'89  !))9)9 :  E--.  $E$5$56 'u||4 "%e.?.?(@"AB "% D> $D> 'tn d^ %U%6%67 **+! "ell+# $!.% & u.. /'  B ]]_;?59 !'+(-&+W ''W %U%6%67W !!1!12 W  W  W W "))$W "&W  $W    W W r.rcH^\rSrSrSU4SjjrS SjrSrSrSrSr U=r $) HifiGanResidualBlocki c>[TU]5 X@l[R"[ [ U55Vs/sH0n[R"UUUSX5URX#U5S9PM2 sn5Ul [R"[ [ U55Vs/sH,n[R"UUUSSURUS5S9PM. sn5Ul gs snfs snf)Nr)rndilationr) rprqleaky_relu_sloperr/rPrUru get_paddingconvs1convs2)r}channelsrmrrr3r_rs r,rqHifiGanResidualBlock.__init__ s  0mms8}- .A %[ ,,[1+F .    mms8}- .A  ,,[!< .      s 7C%%3C*cX-U- S-$rrd)r}rmrs r,r HifiGanResidualBlock.get_padding s&1a77r.c>[RRn[[RRS5(a$[RRRnUR H nU"U5 M UR H nU"U5 M gNr)rrrrrrrr}rr,s r,apply_weight_norm&HifiGanResidualBlock.apply_weight_norm sihh** 288,,m < <((33??K[[E  ![[E  !r.cURH"n[RRU5 M$ URH"n[RRU5 M$ gr)rrrremove_weight_normrr}r,s r,r'HifiGanResidualBlock.remove_weight_norm sB[[E HH ' ' .![[E HH ' ' .!r.c([URUR5Hmup#Un[RR XR 5nU"U5n[RR XR 5nU"U5nX-nMo U$r)rwrrrr leaky_relur)r}rconv1conv2r8s r,rHifiGanResidualBlock.forward sz T[[9LE$HMM44]DYDYZM!-0MMM44]DYDYZM!-0M)4M :r.)rrr)r )rr g?)r) rrrrrqrrrrrrrs@r,rr s! >8/ r.rz HiFi-GAN vocoder. c^\rSrSr%\\S'SrS\4U4SjjrS\R4Sjr Sr Sr \ "S S 9S\RS \R4S j5rS rU=r$)SpeechT5HifiGani' r~rlc >[TU]U5 [UR5Ul[UR 5Ul[R"URURSSSS9Ul [R"5Ul [[UR UR 55Haunup4URR#[R$"URSU--URSUS---UUXC- S-S95 Mc [R"5Ul[)[UR55HpnURSUS---n[URUR*5H4upFUR&R#[-XTXaR.55 M6 Mr [R"WSSSSS9UlUR3S[4R6"UR55 UR3S[4R8"UR55 UR;5 g)Nrr )rmrnrrrSr)rprqrUresblock_kernel_sizes num_kernelsupsample_rates num_upsamplesrru model_in_dimupsample_initial_channelconv_prer/ upsamplerrrwupsample_kernel_sizesrYConvTranspose1d resblocksrPresblock_dilation_sizesrr conv_postrrrQrWrq)r}r~r3 upsample_ratermrrrs r,rqSpeechT5HifiGan.__init__0 s  v;;< !6!67     + +   /8V=R=RTZTpTp9q/r +A+ NN ! !""331=33a!e E +((8Q>  0ss4>>*+A661Q<HH),V-I-I6KiKi)j% %%&:8RZ\s\s&tu*k, 8QAaQRS VU[[1D1D%EF Wejj1D1D&EF r.rPc8[U[R[R45(akURR R SURRS9 URb%URR R5 ggg)zInitialize the weights.r4rRN) rrrurrr\rYr~rWror_)r}rPs r,rbSpeechT5HifiGan._init_weightsV so fryy"*<*<= > > MM   & &CT[[5R5R & S{{&   &&(' ?r.c[RRn[[RRS5(a$[RRRnU"UR 5 UR H nU"U5 M URHnUR5 M U"UR5 gr) rrrrrrrrrrrs r,r!SpeechT5HifiGan.apply_weight_norm] shh** 288,,m < <((33??KDMM"^^E  $^^E  # # %$DNN#r.cR[RRUR5 URH"n[RRU5 M$ UR HnUR5 M [RRUR 5 gr)rrrrrrrrs r,r"SpeechT5HifiGan.remove_weight_normi sh ##DMM2^^E HH ' ' .$^^E  $ $ &$ ##DNN3r.a Converts a log-mel spectrogram into a speech waveform. Passing a batch of log-mel spectrograms returns a batch of speech waveforms. Passing a single, un-batched log-mel spectrogram returns a single, un-batched speech waveform. rr:cURR(aXR- UR- nUR 5S:HnU(dUR S5nUR SS5nURU5n[UR5Hn[RRX0RR5nURU"U5nURX@R -"U5n[SUR 5H)nXPRX@R -U-"U5- nM+ XPR - nM [RRU5nUR#U5n[$R&"U5nU(d2UR)S5R SS5R+S5nU$UR)S5nU$)a spectrogram (`torch.FloatTensor`): Tensor containing the log-mel spectrograms. Can be batched and of shape `(batch_size, sequence_length, config.model_in_dim)`, or un-batched and of shape `(sequence_length, config.model_in_dim)`. Returns: `torch.FloatTensor`: Tensor containing the speech waveform. If the input spectrogram is batched, will be of shape `(batch_size, num_frames,)`. If un-batched, will be of shape `(num_frames,)`. r rrrr%)r~normalize_beforerSrrrrrrPrrrrrrrrrrtanhrar)r}rl is_batchedrr3 res_statejwaveforms r,rSpeechT5HifiGan.forwardq s" ;; ' '&2djj@K __&!+ %//2K#--a3  m4 t))*AMM44]KKD`D`aM NN1-mr-r6rrrndarrayrgrirrrirrrrrr)rArOrrrrrrrrr.r<rNrkrrrrrrrrrrr:rrRrzr|rrr__all__rdr.r,rs " @@!CC)@7e9D,0I   H %!" %,, c [^ "ei0,,0250KSTYT`T`Ka0426 t c?tttU--. t  t ZZ tp#=,!;8!;2A8BIIA8J*bii*Zryy0""(299%RYY%R 1 1D"))DN1"))1h%%P<299<2 +?")- +?)-X&,@&$c2 c2L"))0:5:zc5cL(7o(7(7V| -| ~"&="J'$;'T #: @~ -~ B/&=/d3$;3l*#:*Z8M2998Mv:bii:z Y +Y  Y x s 5s  s r7;15#'$)"'L "L##L!!2!23LU--. L  L  LLbii L"L L 5  eE$5$5u7H7H$HI IJL^ e 5e  e P  t 7t  t n;299;| tot  tn r.