:i#SSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r J r SSK J r JrJr SSKJrJr \ "5(aSSKJr S \\-S \4S jrSS \\-S \S \4SjjrS \\-S \S-4Sjr"SS\5rg)N)cached_property)Path)TokenizerException)assert_sentencepiece_installedis_sentencepiece_installed)SpecialTokenPolicy TokenizerTokenizerVersion) ImageConfigMultiModalVersion)SentencePieceProcessorpathreturncb^[T[5(a [T5m[[R 5n[[ R 5S/-nUVVs/sHo2H nSUU3PM M snnS/-nTR5=(a [U4SjU55$s snnf)z1Check if the given path is a SentencePiece model.z.model.z.modelc3Z># UH nTRRU5v M" g7fN)nameendswith).0suffixrs h/home/james-whalen/.local/lib/python3.13/site-packages/mistral_common/tokens/tokenizers/sentencepiece.py #is_sentencepiece..s#!T8$))"4"4V"<"<8s(+) isinstancestrrlistr __members__r is_fileany)rinstruct_versions mm_versionsvmsuffixess` ris_sentencepiecer&s$Dz-99:(445 Tc!T8!TTTRsB+tokenizer_filenameraise_deprecatedc[U5nURS5SnUS:waURS5SnUS:Xa!U(a[SUS35e[S5$U[R;a[S U35e[U5$) z3Get the version of the tokenizer from the filename..modelr$rz4Make sure to rename your tokenizer file to end with z.v1.v1!Unrecognized tokenizer filename: )rsplitrr r)r'r( _version_strs rget_spm_versionr1 s/0%++C04Lw#))#.q1 w $'[\n[oos%tu u %%+777 #DEWDX!YZZ L ))c[U5nURS5SnUS:XdSU;agSURS5S-nU[R;a[ SU35e[U5R $)z1Get the image config from the tokenizer filename.r*r+r,r$Nr.)rr/r rrconfig)r'r0_mm_version_strs rget_image_configr65s/0%++C04Lw#\"9L..s3B77O/;;; #DEWDX!YZZ _ - 4 44r2c ^\rSrSrSrSS\\-S\S-SS4U4Sjjjr\ S\4Sj5r \ S\4S j5r S \S\ 4S jr \ S\ 4S j5rS\\4S jr\S\ 4Sj5r\S\ 4Sj5r\S\\ 4Sj5rS \S\S\S\\ 4SjrSS\\ S\S-S\4SjjrS\ S\4SjrS\\ S\S\4SjrS\\ S\4SjrS\\ S\4Sjr\ S\ 4Sj5r\ S\ 4Sj5rSr U=r!$) SentencePieceTokenizerEzC[SentencePiece](https://github.com/google/sentencepiece) tokenizer.N model_pathtokenizer_versionrc>[5 [R"URR5Ul[ RRU5(dU5e[[U[5(aUOUR5S9Ul URR5URR5:Xde[!UR"5Vs/sHo0RR%U5PM snUlU=(d [)USS9Ul[-U5Ul[0TU]e5 gs snf)zInitialize the `SentencePieceTokenizer`. Args: model_path: The path to the `SentencePiece` model. tokenizer_version: The version of the tokenizer. If not provided, it will be inferred from the model path. ) model_fileF)r(N)rlogging getLogger __class____name___loggerosrisfiler rras_posix_model vocab_sizeget_piece_sizerangen_words id_to_piece_vocabr1_versionr _file_pathsuper__init__)selfr:r;ir@s rrPSentencePieceTokenizer.__init__Hs '((()@)@A ww~~j))5:5),%/ C%@%@zjFYFYF[  {{%%'4;;+E+E+GGGG;@;NO;Na{{..q1;NO *;*rzlq?r z*  Ps($E cUR$)z The path to the tokenizer model.)rNrQs r file_path SentencePieceTokenizer.file_path`sr2cUR$)zThe version of the tokenizer.)rMrUs rversionSentencePieceTokenizer.versiones}}r2sc8URRU5$)z+Get the control token for the given string.)rF piece_to_id)rQr[s rget_control_token(SentencePieceTokenizer.get_control_tokenjs{{&&q))r2c6URR5$)z!Vocabulary size of the tokenizer.)rFrGrUs rrJSentencePieceTokenizer.n_wordsns{{%%''r2cUR$)z(All tokens in the vocabulary as strings.)rLrUs rvocabSentencePieceTokenizer.vocabss {{r2c6URR5$)z#The beginning of sentence token id.)rFbos_idrUs rrfSentencePieceTokenizer.bos_idw{{!!##r2c6URR5$)zThe end of sentence token id.)rFeos_idrUs rrjSentencePieceTokenizer.eos_id|rhr2c[UR5Vs1sH&oRRU5(dM$UiM( sn$s snfr)rIrJrF IsControl)rQtoks r_control_tokens&SentencePieceTokenizer._control_tokenss4$T\\2Q2kk6K6KC6P2QQQs #AAboseosc[U[5(deURRU5nU(aUR/UQnU(a/UQUR PnU$)zEncode the given string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. )rrrFencoderfrj)rQr[rqrrts rrtSentencePieceTokenizer.encodesZ!S!!!!{{))!, !q!A !!!T[[!Ar2tokensspecial_token_policyc^Ub-[U[5(d[S[U5S35eUc+[R "S[ 5 [RnU[R[R4;aURX5$URRU5$)a Decode the given list of token ids into a string. Note: Using `special_token_policy=SpecialTokenPolicy.KEEP` will keep the special tokens and the normal tokens as SentencePiece pieces. Args: tokens: The list of token ids. special_token_policy: The policy to use for special tokens. If `None`, the default policy is `SpecialTokenPolicy.IGNORE`. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. zFExpected `special_token_policy` to be None or SpecialTokenPolicy, got r*zUsing the tokenizer's special token policy `None` is deprecated. It will be removed in 1.10.0. Please pass a special token policy explicitly. Future default will be SpecialTokenPolicy.IGNORE.) rr ValueErrortypewarningswarn FutureWarningIGNOREKEEPRAISE_decode_with_special_tokensrFdecode)rQrwrxs rrSentencePieceTokenizer.decodes +J?SUg4h4hXY]^rYsXttuv  ' MMH  $6#<#< $6$;$;=O=U=U#V V33FQ Q{{!!&))r2token_idc8URRU5$)z,Convert the given token id to a token piece.)rFrK)rQrs rrK"SentencePieceTokenizer.id_to_pieces{{&&x00r2c /n/nUHnXPR;azU[R:Xa [S5eU(a2UR UVs/sHoPR U5PM sn5 /nUR UR W55 MUR U5 M U(a0UR UVs/sHoPR U5PM sn5 SRU5$s snfs snf)NzNDecoding `tokens` that contain special tokens with special_token_policy=RAISE.r)rorrrzextendrKappendjoin)rQrwrx text_list curr_tokensrns rr2SentencePieceTokenizer._decode_with_special_tokenss !# C***'+=+C+CC$%uvv$${%S{&6&6s&;{%ST"$K  !1!1#!67""3'    {K{..s3{K Lwwy!!&TLs C0 >C5cZ[R"S[5 URU5$)z[DEPRECATED] Converts a list of token ids into a string, keeping special tokens. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead. This is a convenient method for debugging. z`to_string` is deprecated and will be removed in 1.10.0. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead.)r|r}r~ _to_stringrQrws r to_string SentencePieceTokenizer.to_strings,  \   v&&r2c>URU[RS9$)N)rx)rrrrs rr!SentencePieceTokenizer._to_strings{{68J8O8O{PPr2c6URR5$)zThe padding token id.)rFpad_idrUs rrSentencePieceTokenizer.pad_idrhr2c6URR5$)zThe unknown token id.)rFunk_idrUs rrSentencePieceTokenizer.unk_idrhr2)rNrBrFrMrLr)"rA __module__ __qualname____firstlineno____doc__rrr rPpropertyrVrYintr^rJrrcrrfrjsetroboolrtrrrKrrrrr__static_attributes__ __classcell__)r@s@rr8r8EsN3:BRUYBYei04)*3*3*(((tCy$$$$$$RSRR$Td3i&$*T#Y$*>PSW>W$*cf$*L1C1C1"$s)"Se"jm"*'S 'c' QcQsQ$$$$$$r2r8)F)r>rCr| functoolsrpathlibrmistral_common.exceptionsrmistral_common.importsrr%mistral_common.tokens.tokenizers.baserr r &mistral_common.tokens.tokenizers.imager r sentencepiecer rrr&r1r6r8r2rrs %8] R4 U3: U$ U*d *d*Wg** 5t 5 d8J 5 m$Ym$r2