cCi@' S r SSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK
J r J r \ " 5 ( a SS K
Jr OSr\ R " \5 rS
S S .r " S
S\ 5 rS/rg)z Tokenization class for model T5. N)copyfile)Optional )PreTrainedTokenizerFast)is_sentencepiece_availablelogging ) T5Tokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_filec ^ \ rS rSr% Sr\rSS/r\r / r
\ \ \
S' SU 4S jjr\S 5 rSS\S \\ S
\\ 4S jjr SS \ \ S
\\ \ S
\ \ 4S jjr SS \ \ S
\\ \ S
\ \ 4S jjrS rS rSrU =r$ )T5TokenizerFast) a
Construct a "fast" T5 tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
extra_ids (`int`, *optional*, defaults to 100):
Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as
"" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by
calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method
additional_special_tokens (`list[str]`, *optional*):
Additional special tokens used by the tokenizer.
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
from_slow (`book`, *optional*, defaults to `False`):
Whether or not the tokenizer should be converted from a slow one. If `add_prefix_space` is set, this will be set to `True`.
input_idsattention_mask
prefix_tokensc > Ub~ U V
s/ s H n
S[ U
5 ; d M U
PM n n
[ U 5 S: a$ U[ U5 V s/ s H n SU S3PM sn -
nOIUS: a! U[ U 5 :w a [ SU SU S35 eO![ U5 V s/ s H n SU S3PM n n U nUb [ R S5 S U S
' [ T
U ] " S UUUUUUUUS .U D6 Xl X`l g s sn
f s sn f s sn f )
Nz
r zBoth extra_ids (z!) and additional_special_tokens (zk) are provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids tokenszXYou set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizersT from_slow)r r eos_token unk_token pad_token extra_idsadditional_special_tokensadd_prefix_space )
strlenrange
ValueErrorlogger warning_oncesuper__init__r
_extra_ids)selfr r r r r r r r kwargsx extra_tokensi __class__s e/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/t5/tokenization_t5_fast.pyr$ T5TokenizerFast.__init__V sA % 0'@['@!LTWXYTZDZA'@L[< 1$)yIY-ZIYA
1#Q.?IY-ZZ)Q 9L0A#A &yk1RSlRm n 8=Y7GH7G!j1 -7GLH(4 % ' j
#'F;
!) &? -
%#A \-Z Is C'C'C,C1c U [ R ; aH [ R U nUb X!:w a U$ Uc( [ R " SU SU SU SU S3 [ 5 U$ )NzGThis tokenizer was incorrectly instantiated with a model max length of z which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on z( automatically truncating your input to zM when padding/encoding.
- If you want to encode/pad to sequences longer than z you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.)r max_model_input_sizeswarningswarn
FutureWarning) pretrained_model_name_or_pathmax_model_lengthinit_max_model_lengthdeprecated_max_model_lengths r, !_eventually_correct_t5_max_length1T5TokenizerFast._eventually_correct_t5_max_length s (O,Q,Q Q*9*O*OPm*n '$05J5^,,&.
34 5 66 734 5$$?#@ Ag
g " save_directoryfilename_prefixreturnc U R ( d [ S5 e[ R R U5 ( d [
R
SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w a. [ U R U5 [
R SU 35 U4$ )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r zCopy vocab file to )
can_save_slow_tokenizerr ospathisdirr! errorjoinVOCAB_FILES_NAMESabspathr r info)r& r: r; out_vocab_files r, save_vocabulary T5TokenizerFast.save_vocabulary s ++
ww}}^,, LL,^,< ? r9 token_ids_0 token_ids_1c XR /- nUc U R U- $ X R /- nU R U- U- $ )a"
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. A sequence has the following format:
- single sequence: `X `
- pair of sequences: `A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) eos_token_idr )r& rL rM s r, build_inputs_with_special_tokens0T5TokenizerFast.build_inputs_with_special_tokens sS & "%6%6$77 %% 3 3%):):(;;K%% 3kA Ar9 c r U R /nUc [ X- 5 S/- $ [ X- U- U- 5 S/- $ )aw
Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make
use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )rO r )r& rL rM eoss r, $create_token_type_ids_from_sequences4T5TokenizerFast.create_token_type_ids_from_sequences sL ! {()QC/ /;${2S89QC??r9 c T [ [ [ S U R 5 5 5 $ )Nc F [ [ R " SU 5 5 S L$ )Nz)boolresearch)r( s r, 5T5TokenizerFast.get_sentinel_tokens.. s bii0A1&E!Fd!Rr9 )listsetfilterr )r& s r, get_sentinel_tokens#T5TokenizerFast.get_sentinel_tokens s& RTXTrTrs t
r9 c j U R 5 Vs/ s H oR U5 PM sn$ s snf N)r` convert_tokens_to_ids)r& tokens r, get_sentinel_token_ids&T5TokenizerFast.get_sentinel_token_ids s. ?C?W?W?YZ?Ye**51?YZZZs 0)r% r )NNzzzd NNrc )__name__
__module__ __qualname____firstlineno____doc__rF vocab_files_namesmodel_input_namesr
slow_tokenizer_classr r] int__annotations__r$ staticmethodr7 r r tuplerJ rP rT r` rf __static_attributes__
__classcell__)r+ s @r, r r ) s
$L *$&67& !M49! "& .$` *!c !HSM !]bcf]g !* JNB 9B3;DI3FB
cB6 JN@ 9@3;DI3F@
c@,
[ [r9 r )rm rA rY r0 shutilr typingr tokenization_utils_fastr utilsr r tokenization_t5r
get_loggerri r! rF r __all__r r9 r, r~ sl ' > 8 ,K
H %#1EUV }[- }[@
r9