cCi\A S r SSKJrJr SSKrSSKJr SSKJ r J
r
J r SSK J
r
SSKJr SS KJrJr \" 5 ( a SSKr\R( " \5 r " S
S \
5 rS /rg) z%
Feature extractor class for Whisper
)OptionalUnionN )is_torch_available)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)
TensorTypeloggingc ^ \ rS rSrSrS/r SU 4S jjrS\R S\ S\R 4S jr
S S \R S\ S\R 4S
jjr \ S S \
\R S \
\R S
\S\
\R 4S jj5 r S S\\R \
\ \
\R \
\
\ 4 S\S\\ S\\\ \4 S\\ S\\ S\\ S\\ S\\ S\\ S\\ S\4S jjrSrU =r$ ) WhisperFeatureExtractor$ a/
Constructs a Whisper feature extractor.
This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains
most of the main methods. Users should refer to this superclass for more information regarding those methods.
This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the `Short Time
Fourier Transform` which should match pytorch's `torch.stft` equivalent.
Args:
feature_size (`int`, *optional*, defaults to 80):
The feature dimension of the extracted features.
sampling_rate (`int`, *optional*, defaults to 16000):
The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
hop_length (`int`, *optional*, defaults to 160):
Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients.
chunk_length (`int`, *optional*, defaults to 30):
The maximum number of chunks of `sampling_rate` samples used to trim and pad longer or shorter audio
sequences.
n_fft (`int`, *optional*, defaults to 400):
Size of the Fourier transform.
padding_value (`float`, *optional*, defaults to 0.0):
Padding value used to pad the audio. Should correspond to silences.
dither (`float`, *optional*, defaults to 0.0):
Adds dithering. In other words, adds a small Gaussian noise to each frame.
E.g. use 0.0001 to add dithering with a normal distribution centered
around 0.0 with standard deviation 0.0001 (assuming [-1,+1] range of raw_speech).
The value 0.0 means no dithering.
Dithering has similar effect as `spectrogram(mel_floor=...)`. It reduces
the high log_mel_fbank values for signals with hard-zero sections,
when VAD cutoff is present in the signal.
input_featuresc
> [ T
U ] " SUUUUS.U D6 XPl X0l X@l XB- U l U R
U- U l X l Xpl [ SUS- - USSUSSS9U l
g ) N) feature_size
sampling_rate
padding_valuereturn_attention_mask g @@slaney)num_frequency_binsnum_mel_filters
min_frequency
max_frequencyr norm mel_scale ) super__init__n_fft
hop_length chunk_length n_samples
nb_max_framesr ditherr mel_filters) selfr r r% r& r$ r r) r kwargs __class__s p/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/whisper/feature_extraction_whisper.pyr# WhisperFeatureExtractor.__init__H s
%''"7
$ (%5 !^^z 9 * * 5A:~ ( '
waveform_batchdevicereturnc US:w a [ SU S35 e/ nU H n[ U[ U R S5 U R U R SU R
U R SS9nUSS2SS 24 n[ R " XUR 5 S
-
5 nUS - S - nUR U5 M [ R " U5 nU$ ) z
Compute the log-mel spectrogram of the provided audio, gives similar results to Whisper's original torch
implementation with 1e-5 tolerance.
cpuz Got device `z` for feature extraction, but feature extraction on CUDA accelerator devices requires torch, which is not installed. Either set `device='cpu'`, or install torch according to the official instructions: https://pytorch.org/get-started/locally/hanng @log10) frame_lengthr% powerr) r* log_melN @ @)
ValueErrorr r r$ r% r) r* npmaximummaxappendarray)r+ r1 r2 log_spec_batchwaveformlog_specs r. _np_extract_fbank_features2WhisperFeatureExtractor._np_extract_fbank_featuresl s
U? vh 'q q
&H"
F3 !ZZ ??{{ , , H 3B3'Hzz(LLNS,@AH 3#-H ! !( + ' .1 r0 rE c " [ R " U5 R U[ R 5 n[ R " U R
US9nU R S:w aC XR [ R " UR UR UR S9- -
n[ R " XR
U R USS9nUSSS24 R 5 S - n[ R " U R 5 R U[ R 5 nUR U- n[ R " US
S 9R# 5 nUR% 5 S :X a> UR' S SS 9S
R' SSS 9S
n [ R( " XS-
5 nO'[ R( " XR' 5 S-
5 nUS- S- nUS:w a UR+ 5 R- 5 nUR/ 5 $ )z
Compute the log-mel spectrogram of the audio using PyTorch's GPU-accelerated STFT implementation with batching,
yielding results similar to cpu computing with 1e-5 tolerance.
)r2 r )dtyper2 T)windowreturn_complex.Nr; r g|=)min)dimkeepdimr r r< r= r5 )torch
from_numpytofloat32 hann_windowr$ r) randnshaperJ r2 stftr% absr* Tclampr7 rN rA r@ detachr5 numpy)
r+ rE r2 rK rW
magnitudesr* mel_specrF max_vals
r. _torch_extract_fbank_features5WhisperFeatureExtractor._torch_extract_fbank_features s
##H-00 G""4::f=
;;# ekk(.._g_n_n&oo oHzz(JJ_cd#ss(^'')Q.
&&t'7'78;;FEMMR ==:-;;xU399; <<>Q llq$l7:>>1d>STUVG }}X}=H }}X||~/CDH sNc) U? (,,.H~~ r0 input_valuesattention_maskr c Ub [ R " U[ R 5 n/ n[ XR S5 5 Hl u pEXDSU R 5 -
[ R " USU R 5 S- 5 - nXVR S : a X&US& UR U5 Mn U$ U Vs/ s H= owUR 5 -
[ R " UR 5 S- 5 - PM? nnU$ s snf )zK
Every array in the list is normalized to have zero mean and unit variance
Nr; gHz>r )
r? rC int32zipsummeansqrtvarrV rB )rb rc r normed_input_valuesvectorlength normed_slicexs r. zero_mean_unit_var_norm/WhisperFeatureExtractor.zero_mean_unit_var_norm s % XXnbhh ?N"$ "%l4F4Fr4J"K & )=)=)? ?2776RYSY?K^K^K`cgKgCh h . .q 11,9)#**<8 #L #" Vb"bUaPQLBGGAEEGdN4K#KUa "b"" #cs ;AD
max_length
raw_speech
truncationpad_to_multiple_ofreturn_tensorsr paddingr do_normalizereturn_token_timestampsc UbP XR :w a@ [ SU R R SU R SU R SU S3 5 eO-[ R SU R R S35 [
U[ R 5 =( a [ UR 5 S : n
U
( a' [ UR 5 S
: a [ S U 35 eU
=( dE [
U[ [ 45 =( a( [
US [ R [ [ 45 nU( a? U Vs/ s H1 n[ R " U/[ R S
9R PM3 nnOU( dC [
U[ R 5 ( d$ [ R " U[ R S
9nOo[
U[ R 5 ( aP UR [ R " [ R" 5 L a UR% [ R 5 nU( d" [ R " U/5 R /n[' SU05 nU R) UUU( a UO U R* UUU=( d U S9nU ( a? U R- US US U R. S9US' [ R0 " US S S9US' UR3 S5 R5 S
S S 5 n[7 5 ( a U R8 O U R: nU" US U
5 n[
US [ 5 ( a8 U Vs/ s H& n[ R " U[ R S
9PM( snUS' OUUS' U( aL US SS2SSU R< 24 nUS R S U R< - S :w a USS2SS24 nUUS' U bY [ R? SU R R S35 U Vs/ s H n[ U5 U R< - PM snUS' Ub URA U5 nU$ s snf s snf s snf )a
Main method to featurize and prepare for the model one or several sequence(s). Implementation uses PyTorch for
the STFT computation if available, otherwise a slower NumPy based one.
Args:
raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`):
The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float
values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not
stereo, i.e. single float per timestep.
truncation (`bool`, *optional*, default to `True`):
Activates truncation to cut input sequences longer than *max_length* to *max_length*.
pad_to_multiple_of (`int`, *optional*, defaults to None):
If set will pad the sequence to a multiple of the provided value.
This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability
`>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128.
return_attention_mask (`bool`, *optional*):
Whether to return the attention mask. If left to the default, will return the attention mask according
to the specific feature_extractor's default.
[What are attention masks?](../glossary#attention-mask)
For Whisper models, `attention_mask` should always be passed for batched inference, to avoid subtle
bugs.
return_tensors (`str` or [`~utils.TensorType`], *optional*):
If set, will return tensors instead of list of python integers. Acceptable values are:
- `'tf'`: Return TensorFlow `tf.constant` objects.
- `'pt'`: Return PyTorch `torch.Tensor` objects.
- `'np'`: Return Numpy `np.ndarray` objects.
sampling_rate (`int`, *optional*):
The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass
`sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition
pipeline.
padding_value (`float`, *optional*, defaults to 0.0):
The value that is used to fill the padding values / vectors.
do_normalize (`bool`, *optional*, defaults to `False`):
Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly
improve the performance of the model.
device (`str`, *optional*, defaults to `'cpu'`):
Specifies the device for computation of the log-mel spectrogram of audio signals in the
`_torch_extract_fbank_features` method. (e.g., "cpu", "cuda")
return_token_timestamps (`bool`, *optional*, defaults to `None`):
Deprecated. Use `return_attention_mask` instead from which the number of frames can be inferred.
Whether or not to return the number of frames of the input raw_speech.
These num_frames can be used by the model to compute word level timestamps.
Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.r r z2Only mono-channel audio is supported for input to r )rJ r )rw rr rt ru r rc )rc r )axisr; z,`return_token_timestamps` is deprecated for z~ and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.
num_frames)!r r> r- __name__loggerwarning
isinstancer? ndarraylenrV listtupleasarrayrS rY rJ float64astyper padr' rp r stackget transposer r` rG r% warning_onceconvert_to_tensors)r+ rs rt ru rv r rw rr r rx r2 ry r, is_batched_numpy
is_batchedspeechbatched_speech
padded_inputsr extract_fbank_featuresfeaturerescaled_attention_mask raw_speech_is r. __call__ WhisperFeatureExtractor.__call__ s H $ 2 22 I$..JaJaIb c))-););(< =))-););( " 0 0 /K /M* +
/1hh}EU7V]^._M* + '**+;<FFq!QO 3E2F2FD . .DLkLk 0q0A6J nQ' . .dr.sdrY`rzz'/Tdr.sM* + /=M* + &34D&EaI[DOOI[F[&\ #
-.44Q7$//IQN*A!SbS&*I'.EM* + " . >t~~?V?V>W XV W
eo*odnT`3|+<+Odn*oM, ' % ) < <^ LM E ]R /t( +ps 8Q-Q"Q)r& r) r% r* r$ r' r( r )P i> i r r F)r5 )r )
TNNNrr NNNr5 N)r~
__module__ __qualname____firstlineno____doc__model_input_namesr# r? r strrG r` staticmethodr floatrp r boolr intr r r __static_attributes__
__classcell__)r- s @r. r r $ s B ** #"
H S UWU_U_ < bjj # Z\ZdZd > be#2::r s\ # " H H I 4 ( H %w 6 w t %
%r0