cCiZSrSSKJr SSKJr SSKJr SSKJr SSK J r \R"\ 5r S S S S .rS S0r"SS\5rS/rg)zTokenization classes for Qwen2.)Optional) AddedToken)PreTrainedTokenizerFast)logging)Qwen2Tokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filezqwen/qwen-tokenizerict^\rSrSrSr\rSS/r\r S U4Sjjr S S\ S\ \ S\ \ 4S jjrS rU=r$) Qwen2TokenizerFast%a Construct a "fast" Qwen2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import Qwen2TokenizerFast >>> tokenizer = Qwen2TokenizerFast.from_pretrained("Qwen/Qwen-tokenizer") >>> tokenizer("Hello world")["input_ids"] [9707, 1879] >>> tokenizer(" Hello world")["input_ids"] [21927, 1879] ``` This is expected. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Not applicable to this tokenizer. bos_token (`str`, *optional*): The beginning of sequence token. Not applicable for this tokenizer. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc V>[U[5(a [USSSSS9OUn[U[5(a [USSSSS9OUn[U[5(a [USSSSS9OUn[U[5(a [USSSSS9OUn[T U]"SUUUUUUUS.UD6 g)NFT)lstriprstripspecial normalized)r r r unk_token bos_token eos_token pad_token) isinstancestrrsuper__init__) selfr r r rrrrkwargs __class__s k/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.pyrQwen2TokenizerFast.__init__Ss$)S)) yudW\ ] )S)) yudW\ ] )S)) yudW\ ] )S)) yudW\ ]   !#)  save_directoryfilename_prefixreturnc^URRRXS9n[U5$)N)name) _tokenizermodelsavetuple)r r&r'filess r#save_vocabulary"Qwen2TokenizerFast.save_vocabularys)%%**>*PU|r%r)NNN <|endoftext|>Nr2r2)N)__name__ __module__ __qualname____firstlineno____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr slow_tokenizer_classrrrr.r0__static_attributes__ __classcell__)r"s@r#rr%sj'R*$&67)!!!. bcHSM]bcf]gr%rN)r7typingrtokenization_utilsrtokenization_utils_fastrutilsrtokenization_qwen2r get_loggerr3loggerr8MAX_MODEL_INPUT_SIZESr__all__rr%r#rGse&,>.   H %&/6a0aH r%