+ht4SSKrSSKrSSKrSSKJrJrJrJr SSKJ r SSK J r SSK J r Jr SSKJr "SS5r"S S \ 5rg) N)DictListOptionalUnion)SentencePieceProcessor)PreTrainedTokenizer) BatchEncoding EncodedInput)PaddingStrategyc \rSrSrS\4SjrSS\4SjjrSS\S\S\S\\ 4S jjr S \\ S\4S jr S \\S\4S jr Sr SrSrg) SPTokenizer model_pathc.[RRU5(dU5e[US9UlURR 5UlURR5UlURR5UlURR5Ul URR 5URR5:Xde/SQn/SQU-n0Ul 0Ul UHInUR URU'X@RUR 'U=R S- slMK SRUVs/sHn[R "U5PM sn5Ulgs snf)N) model_file)z <|system|>z<|user|> <|assistant|>z<|observation|>)z[MASK][gMASK]z[sMASK]sopeop|)ospathisfilersp_model vocab_sizen_wordsbos_ideos_idunk_idpad_idget_piece_sizespecial_tokensindex_special_tokensjoinreescaperole_special_token_expression)selfrrole_special_tokensr#tokens ^/home/james-whalen/.local/lib/python3.13/site-packages/diffusers/pipelines/kolors/tokenizer.py__init__SPTokenizer.__init__s6ww~~j))5:5).*E !MM446 ==//1 ==//1 ==//1 }}'')T]]-I-I-KKKK\GJ]] $&!#E)-D   &6; % %dll 3 LLA L$.1XXUh6iUhEryy7GUh6i-j*6is% Fsc 0U(aSn/n[R"URU5HnX5R5:a:UR UR R XUR555 URXR5UR55 UR5nM U[U5:a,UR UR R XS55 U$UR R U5$)Nr) r&finditerr(startextendrEncodeAsPiecesappendendlen)r)r/encode_special_tokens last_indextmatchs r,tokenizeSPTokenizer.tokenize1s JAT%G%GK -HHT]]99!:WXY;;=599;78"YY[ L CF"55a nEFH==//2 2boseosreturnc[U[5(deURRU5nU(aUR/U-nU(aX@R /-nU$N) isinstancestrrencoderr)r)r/r?r@r:s r,rFSPTokenizer.encode@sR!S!!!! MM  #  !A [[M!Ar>r:cS/p2UH\nX@R;a9U(aX RRU5- n/nX RU- nMKURU5 M^ U(aX RRU5- nU$)N)r$rdecoder5)r)r:textbufferr+s r,rJSPTokenizer.decodeIs~2fE111MM0088DF11%88 e$  MM((0 0D r>tokensc<URRU5nU$rC)r DecodePieces)r)rNrKs r, decode_tokensSPTokenizer.decode_tokensWs}}))&1 r>ctXR;aURU$URRU5$z0Converts a token (str) in an id using the vocab.)r#r PieceToIdr)r+s r,convert_token_to_idSPTokenizer.convert_token_to_id[s4 '' '&&u- -}}&&u--r>cXR;aURU$XRURUR4;dUS:agURR U5$)=Converts an index (integer) in a token (str) using the vocab.rrI)r$rrr!r IdToPiecer)indexs r,convert_id_to_tokenSPTokenizer.convert_id_to_tokenasV -- -,,U3 3 [[$++t{{; ;uqy}}&&u--r>)rrr$rr!r(rr#N)F)FF)__name__ __module__ __qualname____firstlineno__rEr-r<boolrintrFrJrQrWr^__static_attributes__r>r,r r syk3k, 3# 3$Td3i S c DI#. .r>r c<^\rSrSrSS0r/SQrS(U4SjjrSr\S\ 4Sj5r \ RS \ 4S j5r \S\ 4S j5r \ RS \ 4S j5r \S 5r \S\ 4Sj5r\RS \ 4Sj5r\S5r\S5rSrSrSrSrS\\ S\ 4SjrS)SjrSrSrS*SjrS)S\\S\\\S\\4SjjrS\R>SSS4S \ \!\ \"4\#4S!\\S"\S#\\S$\\$S%\\$S\%4S&jjr&S'r'U=r($)+ChatGLMTokenizerj vocab_fileztokenizer.model) input_idsattention_mask position_idsc >SUlXl[U5UlURRURR URR S.UlX@l[TU]("SUUUS.UD6 g)N GLMTokenizer)z) padding_sideclean_up_tokenization_spacesr8rg) namerkr tokenizerrrr!r#r8superr-)r)rkrsrtr8kwargs __class__s r,r-ChatGLMTokenizer.__init__os{# $$Z0^^**^^**^^**  &;"  %)E"7   r>cXR;aURU$XRR;dUSUR35eURRU$)Nz is not a special token for )r#rvrurVs r, get_commandChatGLMTokenizer.get_commandsc '' '&&u- -555h%@\]a]f]f\g7hh5~~,,U33r>rAcgNzrgr)s r, unk_tokenChatGLMTokenizer.unk_tokenr>valuecXlgrC) _unk_tokenr)rs r,rrr>cgrrgrs r, pad_tokenChatGLMTokenizer.pad_tokenrr>cXlgrC) _pad_tokenrs r,rrrr>c$URS5$)Nrrr|rs r, pad_token_idChatGLMTokenizer.pad_token_id((r>cg)Nzrgrs r, eos_tokenChatGLMTokenizer.eos_tokensr>cXlgrC) _eos_tokenrs r,rrrr>c$URS5$)Nrqrrs r, eos_token_idChatGLMTokenizer.eos_token_idrr>c.URR$rC)rvrrs r,rChatGLMTokenizer.vocab_sizes~~%%%r>c[UR5Vs0sHoRU5U_M nnURUR5 U$s snf)zReturns vocab as a dict)ranger_convert_id_to_tokenupdateadded_tokens_encoder)r)ivocabs r, get_vocabChatGLMTokenizer.get_vocabsL:?:PQ:PQ**1-q0:PQ T../ RsAc HURRXRS9$)N)r8)rvr<r8)r)rKrxs r, _tokenizeChatGLMTokenizer._tokenizes ~~&&tC]C]&^^r>c8URRU5$rT)rvrWrVs r,_convert_token_to_id%ChatGLMTokenizer._convert_token_to_id~~11%88r>c8URRU5$)rZ)rvr^r\s r,r%ChatGLMTokenizer._convert_id_to_tokenrr>rNc8URRU5$rC)rvrQ)r)rNs r,convert_tokens_to_string)ChatGLMTokenizer.convert_tokens_to_strings~~++F33r>Nc[RRU5(a-[RRXRS5nOUn[ UR S5nUR5nSSS5 [ US5nURW5 SSS5 U4$!,(df  N7=f!,(df  U4$=f)a9 Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. rkrbNwb) rrisdirr%vocab_files_namesopenrkreadwrite)r)save_directoryfilename_prefixrkfin proto_strwriters r,save_vocabulary ChatGLMTokenizer.save_vocabularys 77== ( (n6L6L\6Z[J'J $//4 (C I)*d #v LL #$} ) ($ #}s*B,B=, B:= C cJURS5URS5/nU$)Nrrr)r) prefix_tokenss r,get_prefix_tokens"ChatGLMTokenizer.get_prefix_tokenss()))4d6F6Fu6MN r>cUS;dU5eURSUS35/URRUS35-nURRU5nXE-nU$)N)systemuser assistant observationz<|z|> )r|rvrF)r)rolemetadatamessage role_tokensmessage_tokensrNs r,build_single_message%ChatGLMTokenizer.build_single_messagespEEKtKE''"TF" 67$..:O:OS[R\\^P_:`` ..w7- r>c Uc/n/nUHknUSnUSS:Xa%SU;aUS-[R"USSSS9-nURURUSUR S S 5U55 Mm URURUS U55 URUR S 5/5 UR U/S S S9$)NcontentrrtoolsrF)indent ensure_asciirrIrptT)return_tensorsis_split_into_words)jsondumpsr3rgetr|batch_encode_plus)r)queryhistoryrrlitemrs r,build_chat_input!ChatGLMTokenizer.build_chat_inputs ?G D9oGF|x'GtO!D.4::d7mA\a+bb   T66tF|TXXjZ\E]_fg h  224UCD$**?;<=%%yk$\`%aar> token_ids_0 token_ids_1cbUR5nX1-nUbX-URS5/-nU$)a6 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. rq)rr|)r)rrrs r, build_inputs_with_special_tokens1ChatGLMTokenizer.build_inputs_with_special_tokenssA&..0 #1  "%3t7G7G7P6QQKr>encoded_inputs max_lengthpadding_strategypad_to_multiple_ofreturn_attention_maskrsc:URS:XdeXRSn[U5nU[R:Xa [U5nUbUbX$-S:wa X$-S-U-nU[R :g=(a [U5U:gn SU;a S/U-US'SU;a[ [U55US'U (aZU[U5- n SU;aS/U -US-US'SU;aS/U -US-US'UR/U -U-XRS'U$)a Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) leftrrrmrn) rsmodel_input_namesr7r LONGEST DO_NOT_PADlistrr) r)rrrrrrsrequired_input seq_lengthneeds_to_be_padded differences r,_padChatGLMTokenizer._pads[H  F***'(>(>q(AB( 66 6^,J  !&8&D*JimnJn%;q@DVVJ-1K1KKqPSTbPcgqPq > 101sZ/?N+ ,  /-1% 2C-DN> * #c.&99J>14533CnUeFf3f/0/23z1ANSaDb1b~.9=9J9J8Kj8X[i8iN11!4 5r>)rrrr8rur#rvrk)rFFrC)Nr))r`rarbrcrrr-r|propertyrErsetterrrrrrrrrrrrrrrrrerrr rrrr r rddictrrf __classcell__)rys@r,ririjs"%'89G %*#  24 3 s  3 s  ))3 s  ))&& _994tCy4S44 bJN93;DI3F c8%),;,F,F,004'+Ad3 #45}DEASMA* A %SM A (~ AtnA AAr>ri)rrr&typingrrrr sentencepiecer transformersr$transformers.tokenization_utils_baser r transformers.utilsr r rirgr>r,rs> ..0,L.M.M.`h*hr>