cCiSSKrSSKJr SSKJr SSKJr SSKJr SSK J r J r \ "5(aSS K J r OSr \ R"\5rS S S .r"S S\5rS/rg)N)copyfile)Optional) processors)PreTrainedTokenizerFast)is_sentencepiece_availablelogging)GemmaTokenizerztokenizer.modelztokenizer.json) vocab_filetokenizer_filec^\rSrSrSr\r\rSr SS/r SU4Sjjr Sr \ S5r\ S 5r\R S 5r\R S 5rSS \S \\S\\4SjjrSSjrSrU=r$)GemmaTokenizerFast"u Construct a Gemma tokenizer fast. Based on byte-level Byte-Pair-Encoding. This uses notably ByteFallback and no prefix space. Normalization is applied to replace `" "` with `"▁"` ```python >>> from transformers import GemmaTokenizerFast >>> tokenizer = GemmaTokenizerFast.from_pretrained("hf-internal-testing/dummy-gemma") >>> tokenizer.encode("Hello this is a test") [2, 4521, 736, 603, 476, 2121] ``` If you want to change the `bos_token` or the `eos_token`, make sure to specify them when initializing the model, or call `tokenizer.update_post_processor()` to make sure that the post-processing is correctly done (otherwise the values of the first token and final token of an encoded sequence will not be correct). For more details, checkout [post-processors] (https://huggingface.co/docs/tokenizers/api/post-processors) documentation. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a .model extension) that contains the vocabulary necessary to instantiate a tokenizer. tokenizer_file (`str`, *optional*): [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`): Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The padding token add_bos_token (`bool`, *optional*, defaults to `True`): Whether or not to add an `bos_token` at the start of sequences. add_eos_token (`bool`, *optional*, defaults to `False`): Whether or not to add an `eos_token` at the end of sequences. left input_idsattention_maskc ~>[T U]"SUUUUUUUUU S. U D6 XlXlUR 5 Xlg)N) r r clean_up_tokenization_spaces unk_token bos_token eos_token pad_token add_bos_token add_eos_token)super__init___add_bos_token_add_eos_tokenupdate_post_processorr ) selfr r rrrrrrrkwargs __class__s k/home/james-whalen/.local/lib/python3.13/site-packages/transformers/models/gemma/tokenization_gemma_fast.pyrGemmaTokenizerFast.__init__Ws[  !))E''  ,+ ""$$cURnURnUcUR(a [S5eURnUR nUcUR (a [S5eUR(aUS-OSSUR (aSU-S-OS3nUUR(aSU-S -OSS UR (aSU-S -OS3n/nUR(aURX45 UR (aURX445 [R"XVUS 9URl g) zU Updates the underlying post processor with the current `bos_token` and `eos_token`. Nz)add_bos_token = True but bos_token = Nonez)add_eos_token = True but eos_token = Nonez:0 z$A:0 z:0z:1z $B:1)singlepairspecial_tokens) r bos_token_idr ValueErrorr eos_token_idrappendrTemplateProcessing _tokenizerpost_processor)r"bosr.eosr0r+r,r-s r%r!(GemmaTokenizerFast.update_post_processorvs6nn(( ;4--HI Inn(( ;4--HI I%)%7%7S5[R@[_[m[mcCiRVFVsuDvw0B0B39t+K5gkgygyRUX[R[^bRb@BQCD     ! !3"5 6     ! !3"5 6)3)F)F^* &r'cUR$N)r r"s r%r GemmaTokenizerFast.add_eos_token"""r'cUR$r9)rr:s r%r GemmaTokenizerFast.add_bos_tokenr<r'c0XlUR5 gr9)r r!r"values r%rr;# ""$r'c0XlUR5 gr9)rr!r@s r%rr>rBr'save_directoryfilename_prefixreturncUR(d [S5e[RR U5(d[ R SUS35 g[RRX(aUS-OS[S-5n[RRUR5[RRU5:wa[URU5 U4$)NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory-r)r ) can_save_slow_tokenizerr/ospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr r)r"rDrEout_vocab_files r%save_vocabulary"GemmaTokenizerFast.save_vocabularys++  ww}}^,, LL,^,<zzzTFr9)__name__ __module__ __qualname____firstlineno____doc__rPvocab_files_namesr slow_tokenizer_class padding_sidemodel_input_namesrr!propertyrrsetterstrrtuplerSrY__static_attributes__ __classcell__)r$s@r%rr"s-^*)L$&67%*%> 4####%%%% !c!HSM!]bcf]g!(  r'r)rJshutilrtypingr tokenizersrtokenization_utils_fastrutilsrr tokenization_gemmar get_loggerr[rMrPr__all__rr'r%rrsb !>82N   H %#4HXY^0^B r'