:i^%SSKrSSKrSSKJr SSKJr SSKrSSKJr SSK J r \ \ S'SSK r SSKr Sr\R""\5rSS jrS \\S \S \\\4S jrS\S\S-S \\4SjrS\\S \\\\44SjrS\\S \4SjrSS\S\\-S-S\ \-S-S\S-S\ S\ S \4Sjjrg!\a SrNf=f)N)Path)Iterator)TokenizerVersion)MultiModalVersion_hub_installedTFreturnc0[(d [S5eg)NzxPlease install the `huggingface_hub` package to use this method. Run `pip install mistral-common[hf-hub]` to install it.)r ImportError`/home/james-whalen/.local/lib/python3.13/site-packages/mistral_common/tokens/tokenizers/utils.py_assert_hub_installedrs > F   r lst chunk_sizec#V# [S[U5U5H nXX!-v M g7f)zChunk a list into smaller lists of a given size. Args: lst: The list to chunk. chunk_size: The size of each chunk. Returns: An iterator over the chunks. Examples: >>> all_chunks = list(chunks([1, 2, 3, 4, 5], 2)) rN)rangelen)rris r chunksrs+1c#h +an%%,s')repo_idrevisionc2[5 [[RR5[RR R S/URS5Q5- nUc_US- [RR- nUR5(a*URS5nUR5nSSS5 U(a3US- U- nUR5(a[R"U5$/$!,(df  NJ=f)zlist the files of a local Hugging Face repo. Args: repo_id: The Hugging Face repo ID. revision: The revision of the model to use. If `None`, the latest revision will be used. models/Nrefsr snapshots)rrhuggingface_hub constants HF_HUB_CACHEREPO_ID_SEPARATORjoinsplitDEFAULT_REVISIONis_fileopenreadis_diroslistdir)rr repo_cache revision_filefile revision_dirs r list_local_hf_repo_filesr/0so//<<=@Y@Y@k@k@p@p 'GMM#&'AJ"V+o.G.G.X.XX  " "##C(D99;)!K/(:    ::l+ + I)(s 3D Dfilesc/n[[R5n[[R5S/-nUVVs/sHoCH nSUU3PM M snnS/-nUHhn[ U5nUR n SR UR5n U S:XaURX45 MOX;dMVURX45 Mj U$s snnf)zFilter the valid tokenizer files from a list of files. Args: files: The list of files to filter. Returns: The list of tuples of file names and paths to the valid tokenizer files. z.model.z.model tekken.json) listr __members__rrnamer"suffixesappend) r0valid_tokenizer_filesinstruct_versions mm_versionsvmsentencepiece_suffixesr- pathlib_file file_namesuffixs r _filter_valid_tokenizer_filesrBKs-99:(445.get_one_valid_tokenizer_file..xs TUVWTXr )keyz,Multiple valid tokenizer files found. Using .)rBr ValueErrorsortedloggerwarning)r0$valid_tokenizer_file_names_and_filesr@tokenizer_files r get_one_valid_tokenizer_filerRfs,I+O( /0A5344 /014)M %I )%%*N D.YZ\]^_`EnEUUVWX >a@C r cache_dirtokenforce_downloadlocal_files_onlyc [5 U(aU(a [S5eU(d)[R"5nUR XUS9nSnO*[XS9n[U5S:Xa[S US US 35e[US9n [R"UUU UUUUS9n U $![ R [ R[ R4aUnU(aUe[XS9nSn[RS5 [U5S:Xa[SUS US 35UeS nANS nAff=f![a [SUS35ef=f)aDownload the tokenizer file of a Mistral model from the Hugging Face Hub. See [here](https://huggingface.co/mistralai/models) for a list of our OSS models. Note: You need to install the `huggingface_hub` package to use this method. Please run `pip install mistral-common[hf-hub]` to install it. Args: repo_id: The Hugging Face repo ID. cache_dir: The directory where the tokenizer will be cached. token: The Hugging Face token to use to download the tokenizer. revision: The revision of the model to use. If `None`, the latest revision will be used. force_download: Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded even if it is already cached. local_files_only: Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is already cached. Returns: The downloaded tokenizer local path for the given model ID. zSYou cannot force the download of the tokenizer if you only want to use local files.)rrTF)rrTzBCould not connect to the Hugging Face Hub. Using local files only.rzXCould not connect to the Hugging Face Hub and no local files were found for the repo ID z and revision z6. Please check your internet connection and try again.Nz%No local files found for the repo ID zz. Please check the repo ID and the revision or try to download the tokenizer without setting `local_files_only` to `True`.)r0z*No valid tokenizer file found in the repo rK)rrSfilenamerTrrVrU)rrLrHfApilist_repo_filesrequestsConnectionError HTTPErrorTimeoutr/rNinforFileNotFoundErrorrRhf_hub_download) rrSrTrrUrVhf_api repo_fileserQtokenizer_paths r download_tokenizer_from_hf_hubrfs<*noo  $**,F//RW/XJ$  .gQ z?a #7yxjYoo  R5JG%44)%N I(((*<*N>NO 1'UJ#  KK\ ]:!#'novnw$XJ.df$ 0 REgYaPQQRs$'B$? D/$3D,A D''D,/E )rN)NNNFF)loggingr)pathlibrtypingrr[%mistral_common.tokens.tokenizers.baser&mistral_common.tokens.tokenizers.imagerbool__annotations__rhuggingface_hub.constantsrr getLogger__name__rNrr4strintrr/tuplerBrRrfr r r rtsX BD$N   8 $ &S &s&xS /B&"cS4ZDI6!c!tE#s(O7L!6S c8$(# " L LTzD L #: LDj L  L  L L_Ns B<<CC