:i5 SSKrSSKJrJr SSKJr SSKJr SSKJ r J r SSK r SSK JrJr SSKJr SSKJr SS KJr SS KJr SS KJrJr SS KJr SS KJr SSKJ r SSK!J"r" SSK#J$r$ "SS\%\5r&"SS\%\5r'"SS\(\5r)"SS\%\5r*"SS\5r+"SS\5r,\ "S\S9r-\ "S\S9r.\ "S \+S9r/"S!S"\ \-\.\/\45r0g)#N)ABCabstractmethod)Enum)Path)GenericTypeVar) ConfigDictField)Audio) MistralBase) FIMRequest)UserContentChunk)AssistantMessageType UserMessage)InstructRequest)Tool)TranscriptionRequest) AudioEncoder) ImageEncoderc \rSrSrSrSrSrSrg)UserMessagePositionzWhere to encode available toolsfirstlastN)__name__ __module__ __qualname____firstlineno____doc__rr__static_attributes__r_/home/james-whalen/.local/lib/python3.13/site-packages/mistral_common/tokens/tokenizers/base.pyrrs) E Dr"rc\rSrSrSrSrSrSrSrSr Sr S r S r S r S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSr g) SpecialTokens aEnum of special tokens used in the tokenizer. Attributes: unk: The unknown token. bos: The beginning of string token. eos: The end of string token. begin_inst: The beginning of instruction token. end_inst: The end of instruction token. begin_tools: The beginning of tools token. end_tools: The end of tools token. begin_tool_results: The beginning of tool results token. end_tool_results: The end of tool results token. tool_calls: The tool calls token. img: The image token. pad: The pad token. img_break: The image break token. img_end: The image end token. prefix: The prefix token for FIM. middle: The middle token for FIM. suffix: The suffix token for FIM. begin_system: The beginning of system prompt token. end_system: The end of system prompt token. begin_tool_content: The beginning of tool content token. args: The args token. call_id: The call id token. audio: The audio token. begin_audio: The beginning of audio token. transcribe: The transcribe token. begin_think: The beginning of think token. end_think: The end of think token. Examples: >>> unk = SpecialTokens.unk zzzz[INST]z[/INST]z[AVAILABLE_TOOLS]z[/AVAILABLE_TOOLS]z[TOOL_RESULTS]z[/TOOL_RESULTS]z [TOOL_CALLS]z[IMG]zz [IMG_BREAK]z [IMG_END]z[PREFIX]z[MIDDLE]z[SUFFIX]z[SYSTEM_PROMPT]z[/SYSTEM_PROMPT]z[TOOL_CONTENT]z[ARGS]z [CALL_ID]z[AUDIO]z [BEGIN_AUDIO]z [TRANSCRIBE]z[THINK]z[/THINK]rN)!rrrrr unkboseos begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddlesuffix begin_system end_systembegin_tool_contentargscall_idaudio begin_audio transcribe begin_think end_thinkr!rr"r#r%r% s!F C C CJH%K$I)(J C CIG F F F$L#J) DG E!KJKIr"r%c$\rSrSrSrSrSrSrSrg)SpecialTokenPolicyazWhat to do with special tokens when encoding/decoding. Attributes: IGNORE: Ignore special tokens. KEEP: Keep special tokens. RAISE: Raise an error if special tokens are found. rrN) rrrrr IGNOREKEEPRAISEr!rr"r#rCrCasF D Er"rCc\rSrSrSrS\SS4Sjr\S\4Sj5r SSS\ 4S jr SSS\ 4S jr SSS\ 4S jr SSS\ 4S jrS rSrSrSrSrSrSrg)TokenizerVersionoaEnum of tokenizer versions. Allow to distinguish between different versions of the tokenizer and maintain backward compatibility. Attributes: v1: The first version of the tokenizer. v2: The second version of the tokenizer that includes special control tokens [INST], [\INST]. v3: The third version of the tokenizer that includes improved function calling. v7: The seventh version of the tokenizer that includes improved system prompt and function calling. v11: The eleventh version of the tokenizer that includes improved function calling. v13: The thirteenth version of the tokenizer that includes no call id tokenization and better prompt caching. Examples: >>> version = TokenizerVersion.v1 valuereturnc[R"SU5(d[SUS35e[R X5nXlU$)Nz^v\d+$zInvalid version format: z#. Must be 'v' followed by a number.)rematch ValueErrorstr__new___value_)clsrMobjs r#rTTokenizerVersion.__new__sAxx 5))7w>abc ckk#%  r"c2[URSS5$)NrE)intrMselfs r# _version_numTokenizerVersion._version_nums4::ab>""r"otherzstr | TokenizerVersionct[U[5(a [U5nURUR:$N isinstancerSrKr]r\r_s r#__lt__TokenizerVersion.__lt__s0 eS ! !$U+E  5#5#555r"cv[U[5(a$[U5nURUR:*$grarbrds r#__le__TokenizerVersion.__le__5 eS ! !$U+E$$(:(:: : "r"cv[U[5(a$[U5nURUR:$grarbrds r#__gt__TokenizerVersion.__gt__s5 eS ! !$U+E$$u'9'99 9 "r"cv[U[5(a$[U5nURUR:$grarbrds r#__ge__TokenizerVersion.__ge__rjr"v1v2v3v7v11v13rN)rrrrr rSrTpropertyrZr]boolrerhrlrorqrrrsrtrurvr!rr"r#rKrKos C$6#c##6466 ;4;; :4:: ;4;; B B B B C Cr"rKc\rSrSr%Sr\"SS9r\\\ S'Sr \ S-\ S'Sr \\S-\ S'\ "\S 9r\\R \ S '\ "\S 9r\\\ S 'S rg) TokenizedanA tokenized [`InstructRequest`][mistral_common.tokens.instruct.request]. Attributes: tokens: The token ids. text: The text representation of the tokens. prefix_ids: The prefix ids for FIM. images: The loaded images associated with the tokens. Examples: >>> tokenized = Tokenized(tokens=[1, 2, 3], text="Hello world", prefix_ids=[1], images=[]) T)arbitrary_types_allowedtokensNtext prefix_ids)default_factoryimagesaudiosr)rrrrr r model_configlistrZ__annotations__r~rSrr rnpndarrayrr r!rr"r#rzrzsi d;L ID#*#'JS D '$T:FD :5FDK5r"rzc \rSrSr\\S\4Sj55r\S\\ 4Sj5r \S\S\ 4Sj5r \\S\4Sj55r \\S\4Sj55r \\S\4S j55r\\S\4S j55r\S \ S \S \S\\4Sj5r\SS\\S\S-S\ 4Sjj5r\S \ S\4Sj5r\\S\4Sj55r\S\\S\ 4Sj5r\S\\S\ 4Sj5r\\S\4Sj55rSrg) TokenizerrNcg)z!Vocabulary size of the tokenizer.Nrr[s r#n_wordsTokenizer.n_wordsr"cg)z(All tokens in the vocabulary as strings.Nrr[s r#vocabTokenizer.vocabrr"token_idcg)z$Convert a token id to the token str.Nr)r\rs r# id_to_pieceTokenizer.id_to_piecerr"cg)z$id of the Beginning of String token.Nrr[s r#bos_idTokenizer.bos_idrr"cg)zid of the End of String token.Nrr[s r#eos_idTokenizer.eos_idrr"cg)zid of the Pad token.Nrr[s r#pad_idTokenizer.pad_idrr"cg)zid of the Unk token.Nrr[s r#unk_idTokenizer.unk_idrr"sr(r)cg)z(Convert a string to a list of token ids.Nr)r\rr(r)s r#encodeTokenizer.encoderr"Nr}special_token_policycg)amDecode the token ids to a string. Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Passing `None` will default to `self._special_token_policy` for [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] and `SpecialTokenPolicy.IGNORE` for [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]. Note that passing `None` will be deprecated and `special_token_policy` will default to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. Nrr\r}rs r#decodeTokenizer.decoderr"cg)zGet the id of a control token.Nr)r\rs r#get_control_tokenTokenizer.get_control_tokenrr"cg)z!Get the version of the tokenizer.Nrr[s r#versionTokenizer.versionrr"cg)z[DEPRECATED] Converts a list of token ids into a string, keeping special tokens. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead. This is a convenient method for debugging. Nrr\r}s r# to_stringTokenizer.to_strings r"cgrarrs r# _to_stringTokenizer._to_string47r"cg)zThe file path of the tokenizer.Nrr[s r# file_pathTokenizer.file_path s r"rra)rrrrrwrrZrrrSrrrrrrrxrrCrrrKrrrrrr!rr"r#rrs 1118tCy884C4C44444...$$$$$$77$7T7d3i77 T#Y >PSW>W cf  .3.3..1)11 S  c  7c7s77  4  r"rInstructRequestType)boundFIMRequestType TokenizedTypec\rSrSr%Sr\\S'\S-\S'\S-\S'S\S\S-S\S-SS4Sjr \ S \ S\ 4S j5r \ S \S\ 4S j5r\ SS \\S \S-S\4Sjj5r\ S \S\ 4Sj5r\ SS\S\\S-S\S\S\S-S\S\\\\\R8\\44Sjj5r\ SS\\\-S\S\S-S\S\\\\\R8\\44 Sjj5r \ S \\S\4Sj5r!Sr"g)InstructTokenizerizBase class for instruct tokenizers. Attributes: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. tokenizerN image_encoder audio_encoderrNcg)zInitialize the instruct tokenizer. Args: tokenizer: The tokenizer to use. image_encoder: The image encoder to use if any. audio_encoder: The audio encoder to use if any. Nr)r\rrrs r#__init__InstructTokenizer.__init__!rr"requestcg)zInstruct request to Tokenized object Args: request: The instruct request to encode. Returns: The tokenized instruct request. Nrr\rs r#encode_instruct!InstructTokenizer.encode_instruct,rr"cg)a Encodes an audio transcription request into a tokenized format. This method processes a transcription request containing audio data, encodes the user message, and returns the tokenized output. Args: request: The transcription request object containing the audio data to be encoded. Returns: Tokenized: The tokenized representation of the audio data, including processed audio and tokens Nrrs r#encode_transcription&InstructTokenizer.encode_transcription7s r"r}rcg)agConvert token ids to string Args: tokens: The token ids to decode. special_token_policy: The policy to use for special tokens. Passing `None` will default to `self._special_token_policy` for [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer] and `SpecialTokenPolicy.IGNORE` for [SentencePieceTokenizer][mistral_common.tokens.tokenizers.sentencepiece.SentencePieceTokenizer]. Note that passing `None` will be deprecated and `special_token_policy` will default to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. Nrrs r#rInstructTokenizer.decodeHrr"cg)zxFIM request to Tokenized object Args: request: The FIM request to encode. Returns: The tokenized FIM request. Nrrs r# encode_fimInstructTokenizer.encode_fimYrr"messageavailable_toolsis_lastis_first system_promptforce_img_firstcg)agEncode a user message. Args: message: The user message to encode. available_tools: The available tools. is_last: Whether the message is the last one. is_first: Whether the message is the first one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)r\rrrrrrs r#encode_user_message%InstructTokenizer.encode_user_messageds. r"contentcg)a Encode a user content. Args: content: The user content to encode. is_last: Whether the content is the last one. system_prompt: The system prompt. force_img_first: Whether to force the image to be first. Returns: The encoded tokens and images. Nr)r\rrrrs r#encode_user_content%InstructTokenizer.encode_user_content}s& r"cgrarrs r#rInstructTokenizer._to_stringrr"rra)NF)#rrrrr rrrrrrrrrrrrrZrCrSrrrrrrxtuplerrr rrrrr!rr"r#rrs$&&$&&  "  3?$3F  WcfjWj     ': }   ,@ ]   T#Y >PSW>W cf   . ]  %) %  dd*     Tz    tCy$rzz*DK7 8  0 %) %  t,--  Tz    tCy$rzz*DK7 8   (7c7s77r"r)1rPabcrrenumrpathlibrtypingrrnumpyrpydanticr r mistral_common.audior mistral_common.baser #mistral_common.protocol.fim.requestr &mistral_common.protocol.instruct.chunkr)mistral_common.protocol.instruct.messagesrr(mistral_common.protocol.instruct.requestr+mistral_common.protocol.instruct.tool_callsr-mistral_common.protocol.transcription.requestr&mistral_common.tokens.tokenizers.audior&mistral_common.tokens.tokenizers.imagerrSrr%rZrCrKrzrrrrrrr"r#rs ##&&+:CE<N??#t>C>B d 5sD5p6 6*Q Q h3?K)<y9 ~8 3^]Th hi~8r"