bCi S r SSKrSSKJr SSKJr SSKJr SSKJ r SSK
J r J r \ " 5 ( a S S
K
Jr OSr\ R " \5 rS S S
.rSr " S S\ 5 rS/rg)z&Tokenization classes for ALBERT model. N)copyfile)Optional )
AddedToken)PreTrainedTokenizerFast)is_sentencepiece_availablelogging )AlbertTokenizerz spiece.modelztokenizer.json)
vocab_filetokenizer_fileu ▁c ^ \ rS rSrSr\r\r S U 4S jjr S
S\
\ S\ \
\ S\
\ 4S jjr
S
S\S \ \ S\\ 4S
jjrS rU =r$ )AlbertTokenizerFast& a"
Construct a "fast" ALBERT tokenizer (backed by HuggingFace's *tokenizers* library). Based on
[Unigram](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=unigram#models). This
tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to
this superclass for more information regarding those methods
Args:
vocab_file (`str`):
[SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that
contains the vocabulary necessary to instantiate a tokenizer.
do_lower_case (`bool`, *optional*, defaults to `True`):
Whether or not to lowercase the input when tokenizing.
remove_space (`bool`, *optional*, defaults to `True`):
Whether or not to strip the text when tokenizing (removing excess spaces before and after the string).
keep_accents (`bool`, *optional*, defaults to `False`):
Whether or not to keep accents when tokenizing.
bos_token (`str`, *optional*, defaults to `"[CLS]"`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `"[SEP]"`):
The end of sequence token. .. note:: When building a sequence using special tokens, this is not the token
that is used for the end of sequence. The token used is the `sep_token`.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
sep_token (`str`, *optional*, defaults to `"[SEP]"`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
cls_token (`str`, *optional*, defaults to `"[CLS]"`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
mask_token (`str`, *optional*, defaults to `"[MASK]"`):
The token used for masking values. This is the token used when training this model with masked language
modeling. This is the token which the model will try to predict.
c
> [ U [ 5 ( a [ U SSSS9OU n [ TU ] " U4UUUUUUUU U
U U S. U
D6 X0l X@l XPl Xl g )NTF)lstriprstrip
normalized) r
do_lower_case remove_space keep_accents bos_token eos_token unk_token sep_token pad_token cls_token
mask_token)
isinstancestrr super__init__r r r r )selfr r
r r r r r r r r r r kwargs __class__s m/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/albert/tokenization_albert_fast.pyr" AlbertTokenizerFast.__init__X s ( *c**
z$u O
)'%% !
+ ( ($ token_ids_0 token_ids_1returnc d U R /nU R /nUc XA- U- $ XA- U- U- U- $ )a8
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An ALBERT sequence has the following format:
- single sequence: `[CLS] X [SEP]`
- pair of sequences: `[CLS] A [SEP] B [SEP]`
Args:
token_ids_0 (`List[int]`):
List of IDs to which the special tokens will be added
token_ids_1 (`List[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`List[int]`: list of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_id cls_token_id)r# r) r* sepclss r& build_inputs_with_special_tokens4AlbertTokenizerFast.build_inputs_with_special_tokens sL & ! ! $s* * 3&4s::r( save_directoryfilename_prefixc U R ( d [ S5 e[ R R U5 ( d [
R
SU S35 g [ R R X( a US- OS[ S - 5 n[ R R U R 5 [ R R U5 :w a [ U R U5 U4$ )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory- r ) can_save_slow_tokenizer
ValueErrorospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr r )r# r3 r4 out_vocab_files r& save_vocabulary#AlbertTokenizerFast.save_vocabulary s ++
ww}}^,, LL,^,<rF zrE z[MASK])N)__name__
__module__ __qualname____firstlineno____doc__r@ vocab_files_namesr slow_tokenizer_classr" listintr r1 r tuplerC __static_attributes__
__classcell__)r% s @r& r r & s ,\ ** +%\ JN; 9;3;DI3F;
c;2!c !HSM !]bcf]g ! !r( r )rK r: shutilr typingr tokenization_utilsr tokenization_utils_fastr utilsr r tokenization_albertr
get_loggerrG r= r@ SPIECE_UNDERLINEr __all__ r( r& r] sm - , > 8 4O H %#1EUV I!1 I!X !
!r(