:iG %SSKrSSKJr SSKJrJrJr SSKJr SSK J r SSK J r J r JrJrJr SSKJrJr SSKJr SS KJrJrJrJrJr SS KJr SS KJrJ r J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r(J)r) SS K*J+r+J,r,J-r- SSK.J/r/J0r0J1r1J2r2J3r3J4r4 SSK5J6r6J7r7J8r8 SSK9J:r:J;r; SSKS\S\:S\ 4Sjr?"SS\\\ \\\(45r@0SS_S\@R_SS_S\@R_S \@R_S!\@R_S"\@R_S#S$_S%\@R_S&\@R_S'\@R_S(\@R_S)S*_S+\@R_S,\@R_S-S._S/\@R_\@R\@R\@R\@R\@R\@R\@R\@RS0S1S2S3. ErE\F\G\/\@44\HS4'g)5N)Path)AnyCallableGeneric)TokenizerException) FIMRequest)UATSAssistantMessageTypeSystemMessageTypeToolMessageTypeUserMessageType)InstructRequestNormalizer normalizer_for_tokenizer_version)ChatCompletionRequest)MistralRequestValidatorMistralRequestValidatorV3MistralRequestValidatorV5MistralRequestValidatorV13ValidationMode)TranscriptionRequest) AudioConfig AudioEncoderSpecialAudioIDs)InstructRequestInstructRequestTypeInstructTokenizerSpecialTokenPolicy SpecialTokens TokenizedTypeTokenizerVersion) ImageConfig ImageEncoderSpecialImageIDs)InstructTokenizerV1InstructTokenizerV2InstructTokenizerV3InstructTokenizerV7InstructTokenizerV11InstructTokenizerV13)SentencePieceTokenizerget_image_configis_sentencepiece) Tekkenizer is_tekken)download_tokenizer_from_hf_hub image_config tokenizerreturnc[UR[RR5UR[R R5UR[R R5S9n[X5$)zLoad a image encoder from a config and a tokenizer. Args: image_config: The image config. tokenizer: The tokenizer. Returns: The image encoder. )img img_breakimg_end)r#get_control_tokenrr4valuer5r6r")r0r1 special_idss b/home/james-whalen/.local/lib/python3.13/site-packages/mistral_common/tokens/tokenizers/mistral.pyload_image_encoderr;:si"  ' ' (9(9(?(? @--m.E.E.K.KL++M,A,A,G,GHK  22 audio_configc[UR[RR5UR[R R5S9n[ X5$)zLoad a audio encoder from a config and a tokenizer. Args: audio_config: The audio config. tokenizer: The tokenizer. Returns: The audio encoder. )audio begin_audio)rr7rr?r8r@r)r=r1r9s r:load_audio_encoderrALsO"))-*=*=*C*CD// 0I0I0O0OPK  22r<cR\rSrSrSrS\\\\\ 4S\ \ \ \ \ 4S\\ \ \ \ \44SjrS\\\\S444S jr\S\4S j5r\S)S j5r\S)S j5r\S*S\S\SS4Sjj5r\S+S\SS4Sjj5r\S+S\S\SS4Sjj5r\SSS S \ RB4S\S\\-S-S\S-S\S\S\ SS4Sjj5r"\\ RB4S\\-S\ SS4Sjj5r#S,S\$\%S \&S-S\4S!jjr'S\(S\4S"jr)S\S\4S#jr*S,S$\+\&S%\,S-S\4S&jjr-S$\+\&S\4S'jr.S(r/g)-MistralTokenizer]aGMistral tokenizer. This class is a wrapper around a [InstructTokenizer][mistral_common.tokens.tokenizers.base.InstructTokenizer], a [MistralRequestValidator][mistral_common.protocol.instruct.validator.MistralRequestValidator] and a [InstructRequestNormalizer][mistral_common.protocol.instruct.normalize.InstructRequestNormalizer]. It provides a convenient interface to tokenize, validate ad normalize Mistral requests. Attributes: instruct_tokenizer: The instruct tokenizer to use. See [InstructTokenizer][mistral_common.tokens.tokenizers.instruct.InstructTokenizer]. instruct_tokenizer validatorrequest_normalizerc*X lX0lUUlg)zInitializes a `MistralTokenizer`. Args: instruct_tokenizer: The instruct tokenizer to use. validator: The request validator to use. request_normalizer: The request normalizer to use. N"_chat_completion_request_validator_instruct_request_normalizerrE)selfrErFrGs r:__init__MistralTokenizer.__init__ms3</,>)  r<r2.c[RURRRUR R 44$)z Provides a recipe for pickling (serializing) this object, which is necessary for use with multiprocessing. Returns: A tuple of the factory function and the arguments to reconstruct the object from its source file. )rC from_filerEr1 file_pathrJ_mode)rLs r: __reduce__MistralTokenizer.__reduce__s@ ))  # # - - 7 7  3 3 9 9,   r<c@[[5RSS- $)Ndata)r__file__parentsclss r: _data_pathMistralTokenizer._data_pathsH~%%a(611r<crUR[UR5S- 5[RS9$)zGet the Mistral tokenizer v1.ztokenizer.model.v1moderPstrr\rtestrZs r:v1MistralTokenizer.v1s0}}S!14H!HIP^PcPc}ddr<crUR[UR5S- 5[RS9$)zGet the Mistral tokenizer v2.z*mistral_instruct_tokenizer_240216.model.v2r_rarZs r:v2MistralTokenizer.v2s8}}  #OO PWeWjWj  r<Fr.is_mmcU(a U(aSnO,U(a U(dSnOU(dU(a [S5eSnUR[UR5U- 5[R S9$)aGet the Mistral tokenizer v3. Args: is_tekken: Whether the tokenizer is a tekken tokenizer. See [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer]. is_mm: Whether to load image tokenizer. Returns: The Mistral tokenizer v3. ztekken_240911.jsonztekken_240718.jsonz;Multimodal tokenizer is currently only supported for tekkenz*mistral_instruct_tokenizer_240323.model.v3r_) ValueErrorrPrbr\rrc)r[r.ritokenizer_names r:v3MistralTokenizer.v3sX 1N u1NuZ[ [IN}}S!1N!BC.J]J]}^^r<cU(a8UR[UR5S- 5[RS9$UR[UR5S- 5[RS9$)z}Get the Mistral tokenizer v7. Args: is_mm: Whether to load the image tokenizer. Returns: The Mistral tokenizer v7. z,mistral_instruct_tokenizer_241114.model.v7m1r_z*mistral_instruct_tokenizer_241114.model.v7ra)r[ris r:v7MistralTokenizer.v7sq ==CNN$'UUV]k]p]p! ==CNN$'SST[i[n[n! r<modelstrictcU(dT[R"S[5 [R 5H!up4X1R 5;dMU"5s $ U[;a[ SU35e[U"5$)a@Get the Mistral tokenizer for a given model. Args: model: The model name. strict: Whether to use strict model name matching. If `False`, the model name is matched as a substring. This is deprecated and will be removed in `mistral_common=1.10.0`. Returns: The Mistral tokenizer for the given model. aCalling `MistralTokenizer.from_model(..., strict=False)` is deprecated as it can lead to incorrect tokenizers. It is strongly recommended to use MistralTokenizer.from_model(..., strict=True)` which will become the default in `mistral_common=1.10.0`.If you are using `mistral_common` for open-sourced model weights, we recommend using `MistralTokenizer.from_file('')` instead.zUnrecognized model: )warningswarn FutureWarningMODEL_NAME_TO_TOKENIZER_CLSitemslowerr)r[rrrs model_name tokenizer_clss r: from_modelMistralTokenizer.from_modelsx MMT   .I-N-N-P) .(?*.Q 3 3$';E7%CD D*5133r<Nrepo_idtokenrevisionforce_downloadlocal_files_onlyr`cB[UUUUUS9n[RXeS9$)aDownload the Mistral tokenizer for a given Hugging Face repository ID. See [here](https://huggingface.co/mistralai/models) for a list of our OSS models. Args: repo_id: The Hugging Face repo ID. token: The Hugging Face token to use to download the tokenizer. revision: The revision of the model to use. If `None`, the latest revision will be used. mode: The validation mode to use. force_download: Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded even if it is already cached. local_files_only: Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is already cached. Returns: The Mistral tokenizer for the given model. )rrrrrr_)r/rCrP)rrrrrr`tokenizer_paths r: from_hf_hubMistralTokenizer.from_hf_hubs348)-   )).)DDr<tokenizer_filenamec[U5(a/[R"U5nURnURnO7[ U5(a[ U5n[U5nSnO[SU35eUb [XC5OSnSnUb'[U[5(dS5e[XS5n[UR5nUR[R:Xa/UbS5eUbS5e[![#U5[%US9US9$UR[R&:Xa/UbS5eUbS5e[![)U5[%US9US9$UR[R*:Xa#UbS5e[![-X6S9[/US9US9$UR[R0:Xa[![3X6US 9[5US9US9$UR[R6:Xa[![9X6US 9[5US9US9$UR[R::Xa[![=X6S9[?US9US9$[S U35e) zLoads a tokenizer from a file. Args: tokenizer_filename: The path to the tokenizer file. mode: The validation mode to use. Returns: The loaded tokenizer. NzUnrecognized tokenizer file: z-Audio is only supported for tekken tokenizersz#Tokenizer version needs to be >= v3z#Tokenizer version needs to be >= v7r_)rFrG) image_encoder)r audio_encoderz!Unrecognized tokenizer filename: ) r.r-rPimager?r,r*r+rr; isinstancerArversionr rdrCr$rrgr%rmr&rrpr'rv11r(v13r)r) r[rr`r1r0r=rrrGs r:rPMistralTokenizer.from_file s" ' ( (",,-?@I$??L$??L 0 1 1./ABI+,>?LL$'DEWDX%YZ ZGSG_*<Cei   #i44 e6e e4.|GM=i>O>OP    0 3 3 3 ( O*O O( ( O*O O(##I.1t<#5    "2"5"5 5 ( O*O O( ( O*O O(##I.1t<#5    "2"5"5 5 ( O*O O(##IK3>#5    "2"5"5 5##IZgh3>#5    "2"6"6 6#$Y[hi3>#5    "2"6"6 6#$YL4$?#5  !#DEWDX!YZZr<requestmax_model_input_lencURRU5nUcUR(a [S5eURR U5nUR(aX$lURRU5$)a Encodes a chat completion request. Args: request: The chat completion request to encode. max_model_input_len: The maximum length of the input to the model. If `None`, the input will not be truncated. Returns: The encoded chat completion request. zUencoding a chat completion request with truncation, but no max model len was provided) rJvalidate_requesttruncate_for_context_lengthrrKfrom_chat_completion_requesttruncate_at_max_tokensrEencode_instruct)rLrrvalidated_requestinstruct_requests r:encode_chat_completion'MistralTokenizer.encode_chat_completion^s}!CCTTU\]  &7+N+N%g  <<YYZkl  . .6I 3&&667GHHr<c8URRU5$)zEncodes a transcription request. Args: request: The transcription request to encode. Returns: The encoded transcription request. )rEencode_transcriptionrLrs r:r%MistralTokenizer.encode_transcription}s&&;;GDDr<c8URRU5$)zEncodes a fill in the middle request. Args: request: The fill in the middle request to encode. Returns: The encoded fill in the middle request. )rE encode_fimrs r:rMistralTokenizer.encode_fims&&11'::r<tokensspecial_token_policyc4URRXS9$)a'Decodes a list of tokens into a string. Args: tokens: The tokens to decode. special_token_policy: The policy to use for special tokens. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. )r)rEdecode)rLrrs r:rMistralTokenizer.decodes&&--f-``r<c8URRU5$N)rE _to_string)rLrs r:rMistralTokenizer._to_strings&&11&99r<rI)r2rC)FF)Fr)0__name__ __module__ __qualname____firstlineno____doc__rrrrr rr r r rrrMtuplerrrS classmethodrr\rdrgboolrmrprbr} staticmethodrrcrrPrr intrrrrlistrrr__static_attributes__r<r:rCrC]s  -oz=Zn.no +? E,@ E] E ;* ; ; aT#Y a>PSW>W acf a:c:s:r<rCzministral-8b-2410c([RSS9$NT)r.rCrmrr<r:rs!1!4!4t!4!Dr<zmistral-tiny-2312zopen-mistral-nemo-2407c([RSS9$rrrr<r:rrs&6&9&9D&9&Ir<zmistral-tiny-2407zmistral-small-2312zopen-mixtral-8x22b-2404zmistral-small-2402zmistral-small-2409c([RSS9$rrrr<r:rrs"2"5"5"5"Er<zmistral-medium-2312zmistral-large-2402zmistral-large-2407zmistral-large-2411zpixtral-large-2411c([RSS9$NT)rirCrprr<r:rrs"2"5"5D"5"Ar<zcodestral-2405zcodestral-mamba-2407zpixtral-12b-2409c*[RSSS9$NT)r.rirrr<r:rrs 0 3 3d$ 3 Or<zopen-mistral-7bc([RSS9$rrrr<r:rrs,//$/?r<c*[RSSS9$rrrr<r:rrs'**T*Fr<c([RSS9$rrrr<r:rrs-00t0rs));sJO]]  JQ3[3ZJ`=`3eq3$3[3Z3L3"D: O1?DUWd deD:N JDJ),,JIJ),, J *-- J /22 J*--JEJ+..J*--J*--J*--JAJ&))J,//J O!J$'**%J&*,,%(((++(++%((%((*--%((?F<;JT#x4D0D'E"EFr<