:iXV lSSKrSSKrSSKrSSKrSSKJr SSKJr SSKJ r SSK J r SSK r SSK JrJr SSKJrJrJrJr SSKJr \R."\5rS \\ -S \4S jr"S S \ 5r"SS\ 5r"SS\ 5r"SS\ 5r "SS\5r!SS\"\S\#S-S \$\%\#44Sjjr&g)N)cached_property)groupby)Path) TypedDict) AudioConfigAudioSpectrogramConfig)SpecialTokenPolicy SpecialTokens TokenizerTokenizerVersion) ImageConfigpathreturnc[U[5(a [U5nUR5=(a% SUR;=(a UR S:H$)z3Check if the given path is a tekken tokenizer file.tekken.json) isinstancestrris_filenamesuffix)rs a/home/james-whalen/.local/lib/python3.13/site-packages/mistral_common/tokens/tokenizers/tekken.py is_tekkenrs@$Dz <<> Nh$))3 N w8NNc>\rSrSr%Sr\\S'\\S'\S-\S'Srg) TokenInfo zToken information in the JSON file. Attributes: rank: The rank of the token. token_bytes: The token in bytes, base64 encoded. token_str: The token in string format. rank token_bytesN token_str) __name__ __module__ __qualname____firstlineno____doc__int__annotations__r__static_attributes__r!rrrr s ITzrrc8\rSrSr%Sr\\S'\\S'\\S'Sr g)SpecialTokenInfo.zSpecial token information in the JSON file. Attributes: rank: The rank of the token. token_str: The token in string format. is_control: Whether the token is a control token. rr is_controlr!N) r"r#r$r%r&r'r(rboolr)r!rrr+r+.s INrr+cL\rSrSr%Sr\\S'\\S'\\S'\\S'\\S'Srg ) TekkenConfig<a<Tekken configuration in the JSON file. Attributes: pattern: The pattern of the tokenizer. num_vocab_tokens: The number of vocabulary tokens. default_vocab_size: The default vocabulary size. default_num_special_tokens: The default number of special tokens. version: The version of the tokenizer. patternnum_vocab_tokensdefault_vocab_sizedefault_num_special_tokensversionr!N) r"r#r$r%r&rr(r'r)r!rrr0r0<s%L ## Lrr0cr\rSrSr%Sr\\\S'\\S-\S'\ \S'\ \S'\ \S'\ \S '\ \S 'S rg) ModelDataNaThe data of the tekken tokenizer model. Attributes: vocab: The vocabulary of the tokenizer. config: The configuration of the tokenizer. version: The version of the tokenizer. type: The type of the tokenizer. image: The image configuration of the tokenizer. vocabNspecial_tokensconfigr6typeimageaudior!)r"r#r$r%r&listrr(r+r0r'rr rr)r!rrr8r8Ns@  ?)*T11  L I  rr8c\rSrSrSr\"S\RSS9\"S\RSS9\"S\RSS9\"S\RSS9\"S \RSS9\"S \RSS9\"S \RSS9\"S \RSS9\"S \RSS9\"S\R SS9\"S\R"SS9\"S\R$SS9\"S\R&SS9\"S\R(SS9\"S\R*SS9\"S\R,SS9\"S\R.SS9\"S\R0SS9\"S\R2SS9\"S\R4SS94rSrSSSSS.S\\S\\S\S \ S!\ S"\!S#\S$\\"-S-S%\#S-S&\$S-4S'jjr%\&S(\"4S)j5r'\(S*\)SS+\\"-S(S4S,j5r*\&S(\#S-4S-j5r+\+RXS.\#S(S4S/j5r+\&S(\$S-4S0j5r-\-RXS.\$S(S4S1j5r-\&S(\ 4S2j5r.\&S(\ 4S3j5r/\&S(\!4S4j5r0\&S(\14S5j5r2\2RXS6\1S(S4S7j5r2\3S(\ 4S8j5r4\3S(\ 4S9j5r5\3S(\ 4S:j5r6\3S(\ 4S;j5r7S(\\4S<jr8S=\S>\9S?\9S(\\ 4S@jr:SA\\ SB\1S(\\4SCjr;SD\ S(\94SEjrSA\\ S(\4SHjr?SA\\ S(\4SIjr@SD\ S(\4SJjrASMSD\ SB\1S-S(\B4SKjjrCSLrDg)N TekkenizerbzTekken tokenizer. This tokenizer is based on the [tiktoken](https://github.com/openai/tiktoken) library. It fastens the tokenization for multiple languages. rTrr r- z tekkenizerN)r_path image_config audio_configr:r;r2 vocab_sizenum_special_tokensr6rrYrZr[c U[U5U-::dU[U5U45eX@l[[UV s/sHoSPM sn 55n [U5U :Xd SU35e[U5U::de[[U5U5V s/sH$n [ XR R U S9SS9PM& nn U(a'[RSUSSSUS S35 X.-n[[UV s/sHoSPM sn 55[U5s=:Xa U:XdU5e U5eXE- n[RS US US 35 [XS 9Ul [[U55[URR55:XdUUR45e[R"UUUR0S9UlX`lXlXlX lUV s0sH oSU S_M sn Ul[U5V s/sHoR)U 5PM sn Ul[,R.UlUb[3U5UlgSUlgs sn fs sn fs sn fs sn fs sn f)aInitialize the tekken tokenizer. Args: vocab: The vocabulary of the tokenizer. special_tokens: The special tokens of the tokenizer. pattern: The pattern of the tokenizer. vocab_size: The vocabulary size of the tokenizer. num_special_tokens: The number of special tokens of the tokenizer. version: The version of the tokenizer. name: The name of the tokenizer. image_config: The image configuration of the tokenizer. r zSpecial tokens must be unique: )idTrDzAdding special tokens rz, ..., zNon special vocabulary size is z with z special tokens.) max_vocab)rpat_strmergeable_ranksr;rN)len _vocab_sizesetranger+SPECIAL_TOKEN_TEMPLATEformatloggerinfo_reload_mergeable_ranks_tekken_token2id_nospecialvaluestiktokenEncoding_model_version _image_config _audio_config_all_special_tokens_special_tokens_reverse_vocab id_to_piece_vocabr IGNORE_special_token_policyr _file_path)selfr:r;r2r\r]r6rrYrZr[tnum_defined_special_tokensispecial_fillerinner_vocab_sizes r__init__Tekkenizer.__init__s4SZ*<<<  J ?  < &&)n-Un nn-U)V%W">"&@@tDcdrcsBtt@>"&88883~.0BC C !/J/J/Q/QUV/Q/Wdh iC    KK():;)G(HP^_aPbcnPoOpq (83?1+?@ASEXn\nn  n  n&:  56F5GvN`Maaqrs*A%*d'5)*+s43R3R3Y3Y3[/\\   + +_  \'' ;;    ))#1 Q_-`Q_A nai.GQ_-`*49*4EF4Eq''*4EF %7%>%>").):$u+W.V  @2.aFsI3 +I8I= 9J JrcJURc [S5eUR$)zThe path to the tokenizer file.z)The tokenizer was not loaded from a file.)r{ ValueErrorr|s r file_pathTekkenizer.file_paths$ ?? "HI Irclsrc [U[5(a [U5nUR5(dU5e[ USSS9n[ R "U5nSSS5 WSRS5nU[R;a+[SUSUS [[R535eUce[U5nURS S5nUc8U[S 5:a[S US 35e[[R5nOUVs/sHoPM nnXsS 'URS5=n (a:U[S5:a[SUSURS35e[S0U D6US'O&URS5=n (a[S0U D6US'URS5=n (a,U R!S5n [#S0U D6n [%SSU 0U D6US'Un U"U SUU SSU SSU SSUUR&R)SS5U RS5U RS5US9 $!,(df  GN=fs snf)z|Load the tekken tokenizer from a file. Args: path: The path to the tokenizer file. Returns: The tekken tokenizer. rutf-8)encodingNr<r6zUnknown version: z in z+. Make sure to use a valid version string: r;v7zSpecial tokens not found in zL. Please update your tokenizer file and include all special tokens you need. multimodalv11z-The image config has to be called 'image' in z for tokenizers of version .r>r?audio_encoding_configencoding_configr:r2r4r5r) r:r;r2r\r]r6rrZr[rYr!)rrrexistsopenjsonloadgetr __members__rr@rBDEPRECATED_SPECIAL_TOKENSvaluer poprrrreplace)rrfuntyped _version_strr6special_tokens_dictsr;tokenmmr>r?r model_datas r from_fileTekkenizer.from_files dC :D{{}}"d"} $g .!iilG/x(,,Y7 /;; ;#L>dV<<<@AQA]A]<^;_a  '''"<0>EkkJZ\`>a  ')$// 24&9aa "&j&J&J!K1EF1Ee1ENF$2 !\* *2 *)%00 CD6Idelererdsstu +0R0GG kk'* *U **3U3GG KK( (5 (#ii(?@O4GGO*T?TeTGG  ' W%)x(3!(+,@A)(34PQ""7B/#0#0  ]/ .0GsI I IcUR$)z)The image configuration of the tokenizer.)rsrs rr>Tekkenizer.image"s!!!rrc[S5e)Nz!Can only set Image config at initrr|rs rr>r'<==rcUR$)zvThe audio configuration of the tokenizer. Returns: The audio configuration object if it exists, otherwise None. )rtrs rr?Tekkenizer.audio+s!!!rc[S5e)Nz!Can only set Audio config at initrrs rr?r4rrc,[UR5$)z.The number of special tokens of the tokenizer.)rdrurs rr]Tekkenizer.num_special_tokens8s4++,,rcUR$)z!Vocabulary size of the tokenizer.)rers rn_wordsTekkenizer.n_words=srcUR$)zThe version of the tokenizer.)rrrs rr6Tekkenizer.versionBs}}rcUR$)z'The policy for handling special tokens.)rzrs rspecial_token_policyTekkenizer.special_token_policyGs)))rpolicyc[U[5(d[S[U5S35e[R "S[ 5 Xlg)z+Set the policy for handling special tokens.z!Expected SpecialTokenPolicy, got rzThe attributed `special_token_policy` is deprecated and will be removed in 1.10.0. Please pass a special token policy explicitly to the relevant methods.N)rr rr=warningswarn FutureWarningrz)r|rs rrrLsJ&"455@faPQ Q Y   &,"rc$URS5$)z#The beginning of sentence token id.zget_control_tokenrs rbos_idTekkenizer.bos_id\s%%e,,rc$URS5$)zThe end of sentence token id.zrrs reos_idTekkenizer.eos_idas%%f--rc$URS5$)zThe padding token id.zrrs rpad_idTekkenizer.pad_idf%%g..rc$URS5$)zThe unknown token id.zrrs runk_idTekkenizer.unk_idkrrcUR$)aAll tokens in the vocabulary as strings. Note: This will collapse all tokens for which we have a decoding error into the string. This is bad and results in things like len(set(vocab)) != len(vocab)). Returns: The vocabulary of the tokenizer. )rxrs rr:Tekkenizer.vocabps{{rsboseoscURRU5nUVs/sHoUUR-PM nnU(aUR/UQnU(a/UQURPnU$s snf)zEncode a string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. )rqencoder]rr)r|rrrtokensr}s rrTekkenizer.encodesi!KK..q17=>v!d---v> kk+F+F +v+t{{+F ?sA)rrc ^/n[UU4Sj5HupEU(auU[R:Xa[S[ U5S35eU[R :XaUR U4SjU55 MiU[R:XaMMURTRRUVs/sHofTR- PM sn55 M U$s snf)Nc">UTR:$Nr])r}r|s r(Tekkenizer._decode_all..s1t?V?V;Vrz/Decoding `tokens` that contain special tokens (a) is not allowed. Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: ``` from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy tokenizer = MistralTokenizer.v3(is_tekken=True) tekken = tokenizer.instruct_tokenizer.tokenizer tekken.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP ```c3H># UHnTRUSv M g7f)r N)ru).0r}r|s r )Tekkenizer._decode_all..s$"[UZPQ4#;#;A#>{#KUZs") rr RAISErr@KEEPextendryappendrqdecoder])r|rrdecoded is_specialgroupr}s` r _decode_allTekkenizer._decode_alls!(1V!W J'+=+C+CC$I$u+ W    *-?-D-DDNN"[UZ"[[)-?-F-FFG t{{11X]2^X]STt7N7N3NX]2^_`-"X.3_s9C"token_idcDSXR- s=:*=(a S:$s $)z$Check if a token id is a byte token.rrr|rs ris_byteTekkenizer.is_bytes$H666<<<<<URU[RS9$)Nrrr rrs rrTekkenizer._to_strings{{68J8O8O{PPrc@URU/[RS9$)z0Convert a token id to its string representation.rrrs rrwTekkenizer.id_to_pieces{{H:'>>//9+FMMgVV%);)A)AA H:-@!ABB%);)B)BB #@AU@V!WXX{{44X@W@W5WXXr) rurtr{rsrqrzrvrmrrrxrer)Er"r#r$r%r&r+r unkrr begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddler begin_system end_systembegin_tool_contentrrhr@rrr'r rr rrpropertyr classmethodr=rr>setterr?r]rr6r rrrrrrr:r.rrrrrrrrwbytesrr)r!rrrBrBbs4 a=+<+<Na=+<+<Na=+<+<Na=+C+CPTUa=+A+AdSa=+D+DQUVa=+B+BtTa=+K+KX\]a=+I+IVZ[a=+C+CPTUbM,=,=$ObM,=,=$ObM,C,CPTUbM,A,AdSbM,@,@TRbM,@,@TRbM,@,@TRbM,F,FSWXbM,D,DQUVbM,L,LY]^)!..!#'+/+/MEIME-.ME ME  ME  ME"MEMETzD ME"D(ME"D(ME^4 E tL)E tE E E N"{T)"" \\>;>4>>"{T)"" \\>;>4>>-C--   )*&8**   ,+= ,$ ,! ,---...//////tCy"$Td3i&$s)CUZ^_bZc8===;3;3;\T#Y\>PSW>W\cf\@'S 'c' QcQsQUCUCU#Y#YDVY]D]#Yin#Y#YrrBr:racPUbW[U5U:d[U5U45e[U5U:a'USUn[RS[U5S35 0n[U5Hfup4UR 51Sk:XdeUSU:Xde[ R "US5nUS:dU[U/5:XdX545eUSX%'Mh [U5[U5:Xde[UR55[[[U555:XdeU$)zAReload our tokenizer JSON file and convert it to Tiktoken format.Nz(Cutting non special vocabulary to first z tokens.>rr rrrr) rdrjrk enumeratekeysbase64 b64decoderrfrnrg)r:raranksrxmerges rrlrls 5zY&?UY(??& u: !*9%E KKB3u:,hW X!E% vvx????yA~~  =!12Cx5E1#J.: :.y ! u:U ## # u||~ #eCJ&7"8 88 8 Lrr)'r rloggingr functoolsr itertoolsrpathlibrtypingrro&mistral_common.tokens.tokenizers.audiorr%mistral_common.tokens.tokenizers.baser r r r &mistral_common.tokens.tokenizers.imager getLoggerr"rjrr.rrr+r0r8rBr@r'dictrrlr!rrr/s %V ?   8 $OC$JO4O   y 9$ (sYsYp !  ?Tz %*r