cCi. S r SSKrSSKJr SSKJr SSKJr SSKJ r J
r
\ " 5 ( a SS K J r OSr \
R " \5 rS
S S .r " S
S\5 rS/rg)z Tokenization classes for XGLM. N)copyfile)Optional )PreTrainedTokenizerFast)is_sentencepiece_availablelogging )
XGLMTokenizerzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_filec ^ \ rS rSrSr\rSS/r\r SU 4S jjr
SS\ \ S\
\ \ S\ \ 4S jjr SS\ \ S\
\ \ S\ \ 4S
jjrSS \S \
\ S\\ 4S
jjrSrU =r$ )XGLMTokenizerFast$ a
Construct a "fast" XGLM tokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`RobertaTokenizer`]
and [`XLNetTokenizer`]. Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
additional_special_tokens (`list[str]`, *optional*, defaults to `["NOTUSED", "NOTUSED"]`):
Additional special tokens used by the tokenizer.
input_idsattention_maskc D > SU l [ U R 5 V
s/ s H n
SU
S3PM n n
U R S/ 5 =( d / U S' U S== U V s/ s H oU S ; d M
U PM sn -
ss' [ T
U ] " U4UUUUUUUS.U D6 Xl g s sn
f s sn f )N z additional_special_tokens)r bos_token eos_token sep_token cls_token unk_token pad_token)num_madeup_wordsrangegetsuper__init__r )selfr r r r r r r r kwargsi madeup_wordsword __class__s i/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/xglm/tokenization_xglm_fast.pyr XGLMTokenizerFast.__init__X s !" 49$:O:O4PQ4Pq+aS*4P Q.4jj9TVX.Y._]_*+*+ )0
)T@[9\-\D\0
+
)
%' R0
s B B
*B
token_ids_0 token_ids_1returnc \ Uc U R /U- $ U R /nX1- U- U- U- $ )a;
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An XLM-RoBERTa sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr! r) r* seps r' build_inputs_with_special_tokens2XGLMTokenizerFast.build_inputs_with_special_tokens{ sD ( %%&4 4 ! 3&,{:: c x U R /nUc [ X1- 5 S/- $ [ X1- U- U- U- 5 S/- $ )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )r- lenr. s r' $create_token_type_ids_from_sequences6XGLMTokenizerFast.create_token_type_ids_from_sequences sQ $ ! s()QC/ /3$s*S0;>?1#EEr2 save_directoryfilename_prefixc U R ( d [ S5 e[ R R U5 ( d [
R
SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w a [ U R U5 U4$ )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory.- r ) can_save_slow_tokenizer
ValueErrorospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr r )r! r7 r8 out_vocab_files r' save_vocabulary!XGLMTokenizerFast.save_vocabulary s ++
ww}}^,, LL,^,<rJ rI zz)N)__name__
__module__ __qualname____firstlineno____doc__rD vocab_files_namesmodel_input_namesr
slow_tokenizer_classr listintr r0 r5 strtuplerG __static_attributes__
__classcell__)r& s @r' r r $ s -^ *$&67( !%H JN; 9;3;DI3F;
c;4 JNF 9F3;DI3FF
cF0!c !HSM !]bcf]g ! !r2 r )rO r> shutilr typingr tokenization_utils_fastr utilsr r tokenization_xglmr
get_loggerrK rA rD r __all__ r2 r' ra sd % > 8 0M
H %#