6biXSrSSKrSSKrSSKrSSKrSSKrSSKJr \"S5SSj5r \"S5SSj5r \"S5SS j5r \"S 5"S S \ 55r \"S 5S5rg)aUtilities for text input preprocessing. Deprecated: `tf.keras.preprocessing.text` APIs are not recommended for new code. Prefer `tf.keras.utils.text_dataset_from_directory` and `tf.keras.layers.TextVectorization` which provide a more efficient approach for preprocessing text input. For an introduction to these APIs, see the [text loading tutorial] (https://www.tensorflow.org/tutorials/load_data/text) and [preprocessing layer guide] (https://www.tensorflow.org/guide/tf_keras/preprocessing_layers). N) keras_exportz.keras.preprocessing.text.text_to_word_sequencec U(aUR5nUVs0sHoDU_M nn[RU5nURU5nUR U5nUVs/sH o(dM UPM sn$s snfs snf)alConverts a text to a sequence of words (or tokens). Deprecated: `tf.keras.preprocessing.text.text_to_word_sequence` does not operate on tensors and is not recommended for new code. Prefer `tf.strings.regex_replace` and `tf.strings.split` which provide equivalent functionality and accept `tf.Tensor` input. For an overview of text handling in Tensorflow, see the [text loading tutorial] (https://www.tensorflow.org/tutorials/load_data/text). This function transforms a string of text into a list of words while ignoring `filters` which include punctuations by default. >>> sample_text = 'This is a sample sentence.' >>> tf.keras.preprocessing.text.text_to_word_sequence(sample_text) ['this', 'is', 'a', 'sample', 'sentence'] Args: input_text: Input text (string). filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``'!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\\t\\n'``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to convert the input to lowercase. split: str. Separator for word splitting. Returns: A list of words (or tokens). )lowerstr maketrans translatesplit) input_textfiltersrr ctranslate_dict translate_mapseqis Y/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/preprocessing/text.pytext_to_word_sequencer's{D %%' (/01hN0MM.1M%%m4J   5 !C s!aAs  1 !s A<% B3Bz keras.preprocessing.text.one_hotc ([UU[UUUUS9$)aOne-hot encodes a text into a list of word indexes of size `n`. Deprecated: `tf.keras.text.preprocessing.one_hot` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.Hashing` with `output_mode='one_hot'` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [preprocessing layer guide] (https://www.tensorflow.org/guide/tf_keras/preprocessing_layers) for an overview of preprocessing layers. This function receives as input a string of text and returns a list of encoded integers each corresponding to a word (or token) in the given input string. Args: input_text: Input text (string). n: int. Size of vocabulary. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``` '!"#$%&()*+,-./:;<=>?@[\]^_`{|}~\t\n ```, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. analyzer: function. Custom analyzer to split the text Returns: List of integers in `[1, n]`. Each integer encodes a word (unicity non-guaranteed). ) hash_functionr rr analyzer) hashing_trickhash)r nr rr rs rone_hotrTs'P   z&keras.preprocessing.text.hashing_trickcUc[nO US:XaSnUc [XXES9nOU"U5nUVs/sHo"U5US- -S-PM sn$s snf)aConverts a text to a sequence of indexes in a fixed-size hashing space. Deprecated: `tf.keras.text.preprocessing.hashing_trick` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.Hashing` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [preprocessing layer guide]( https://www.tensorflow.org/guide/tf_keras/preprocessing_layers) for an overview of preprocessing layers. Args: text: Input text (string). n: Dimension of the hashing space. hash_function: When `None` uses a python `hash` function. Can be 'md5' or any function that takes in input a string and returns a int. Note that 'hash' is not a stable hashing function, so it is not consistent across different runs, while 'md5' is a stable hashing function. Defaults to `None`. filters: list (or concatenation) of characters to filter out, such as punctuation. Default: ``!"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\\t\\n``, includes basic punctuation, tabs, and newlines. lower: boolean. Whether to set the text to lowercase. split: str. Separator for word splitting. analyzer: function. Custom analyzer to split the text Returns: A list of integer word indices (unicity non-guaranteed). `0` is a reserved index that won't be assigned to any word. Two or more words may be assigned to the same index, due to possible collisions by the hashing function. The [probability]( https://en.wikipedia.org/wiki/Birthday_problem#Probability_table) of a collision is in relation to the dimension of the hashing space and the number of distinct objects. md5cz[[R"UR55R 5S5$)N)inthashlibrencode hexdigest)ws rhashing_trick..s##gkk!((*&=&G&G&I2"Nrr rr )rr) textrrr rr rrr#s rrrshX % N #  tn69 :c]1 Q '! +c :: :sA z"keras.preprocessing.text.Tokenizerct\rSrSrSrSSjrSrSrSrSr S r S r SS jr SS jr S rSrSrg) TokenizeraText tokenization utility class. Deprecated: `tf.keras.preprocessing.text.Tokenizer` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.TextVectorization` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text) for an overview of the layer and text handling in tensorflow. This class allows to vectorize a text corpus, by turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf... By default, all punctuation is removed, turning the texts into space-separated sequences of words (words may include the `'` character). These sequences are then split into lists of tokens. They will then be indexed or vectorized. `0` is a reserved index that won't be assigned to any word. Args: num_words: the maximum number of words to keep, based on word frequency. Only the most common `num_words-1` words will be kept. filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, plus tabs and line breaks, minus the `'` character. lower: boolean. Whether to convert the texts to lowercase. split: str. Separator for word splitting. char_level: if True, every character will be treated as a token. oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls analyzer: function. Custom analyzer to split the text. The default analyzer is text_to_word_sequence Nc SU;a'[R"S5 URS5nURSS5n U(a[S[ U5-5e[ R "5Ul[ R"[5Ul X l X@l X0l XlXlXPlX`l[ R"[5Ul0Ul0UlXplg)Nnb_wordszDThe `nb_words` argument in `Tokenizer` has been renamed `num_words`.document_countrz Unrecognized keyword arguments: )warningswarnpop TypeErrorr collections OrderedDict word_counts defaultdictr word_docsr r r num_wordsr. char_level oov_token index_docs word_index index_wordr) selfr8r rr r9r:rkwargsr.s r__init__Tokenizer.__init__s   MM0  :.I$4a8 >VLM M&224$005   ",$"%11#6 rc UGHJnU=RS- slUR(d[U[5(aYUR(aE[U[5(a UVs/sHo3R 5PM nnOUR 5nUnOIUR c+[ UURURURS9nOUR U5nUH:nXPR;aURU==S- ss'M+SURU'M< [U5HnURU==S- ss'M GMM [URR55nURSSS9 URc/nO UR/nURSU55 [![#U[[%S['U5S-5555UlUR(R5VVs0sHupXX_M snnUl[URR55H upXXR,UR(U'M" gs snfs snnf)a\Updates internal vocabulary based on a list of texts. In the case where texts contains lists, we assume each entry of the lists to be a token. Required before using `texts_to_sequences` or `texts_to_matrix`. Args: texts: can be a list of strings, a generator of strings (for memory-efficiency), or a list of list of strings. r'Nr&c US$)Nr')xs rr$(Tokenizer.fit_on_texts..7s1Q4rT)keyreversec3*# UH oSv M g7f)rNrD).0wcs r )Tokenizer.fit_on_texts..=s2'BQ%'s)r.r9 isinstancelistrrrr r r5setr7itemssortr:extenddictziprangelenr<r=r;) r>textsr( text_elemrr#wcounts sorted_vocr s r fit_on_textsTokenizer.fit_on_texts sD   1 $ *T4"8"8::!$--CGH4i 14H#zz|==(/ $ "jj"jj C---C((($$Q'1,'*+D$$Q'  Xq!Q&!18t''--/0  6 >> !J..)J2'22  Dq#j/A*=!>? @ -1OO,A,A,CD,CDA14,CD--/0DA23OODOOA. /1O IJEs ,I0I5cU=R[U5- slUH.n[U5nUHnURU==S- ss'M M0 g)zUpdates internal vocabulary based on a list of sequences. Required before using `sequences_to_matrix` (if `fit_on_texts` was never called). Args: sequences: A list of sequence. A "sequence" is a list of integer word indices. r'N)r.rWrPr;)r> sequencesrrs rfit_on_sequencesTokenizer.fit_on_sequencesIsJ s9~-Cc(C"a'"rc6[URU55$)aTransforms each text in texts to a sequence of integers. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: texts: A list of texts (strings). Returns: A list of sequences. )rOtexts_to_sequences_generator)r>rXs rtexts_to_sequencesTokenizer.texts_to_sequencesYsD55e<==rc#&# URnURRUR5nUGHPnUR(d[ U[ 5(aYUR(aE[ U[ 5(a UVs/sHoUR5PM nnOUR5nUnOIURc+[UURURURS9nOURU5n/nUHxnURRU5n U b7U(aX:aUbURU5 MCMEURU 5 MXURcMgURU5 Mz Uv GMS gs snf7f)a~Transforms each text in `texts` to a sequence of integers. Each item in texts can also be a list, in which case we assume each item of that list to be a token. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: texts: A list of texts (strings). Yields: Yields individual sequences. Nr&) r8r<getr:r9rNrOrrrr r append) r>rXr8oov_token_indexr(rYrvectr#rs rrc&Tokenizer.texts_to_sequences_generatorgs6NN //--dnn=D*T4"8"8::!$--CGH4i 14H#zz|==(/ $ "jj"jj C---CDOO''*= Q^*6 KK87 A^^/KK0J; IsBF F #CF-$Fc6[URU55$)aTransforms each sequence into a list of text. Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: sequences: A list of sequences (list of integers). Returns: A list of texts (strings) )rOsequences_to_texts_generator)r>r_s rsequences_to_textsTokenizer.sequences_to_textssD55i@AArc## URnURRUR5nUHn/nUHnURRU5nUbDU(a*Xb:a%Ub UR URU5 MPMRUR U5 MeURcMtUR URU5 M SR U5nUv M g7f)a~Transforms each sequence in `sequences` to a list of texts(strings). Each sequence has to a list of integers. In other words, sequences should be a list of sequences Only top `num_words-1` most frequent words will be taken into account. Only words known by the tokenizer will be taken into account. Args: sequences: A list of sequences. Yields: Yields individual texts. N )r8r<rgr:r=rhjoin)r>r_r8rirrjnumwords rrm&Tokenizer.sequences_to_texts_generatorsNN //--dnn=CD**3/# S%5*6 KK(HI7 D)^^/KK @A88D>DJs B+C-1rXrwr_s rtexts_to_matrixTokenizer.texts_to_matrixs(++E2 '' '==rc ~UR(d5UR(a[UR5S-nO[S5eURnUS:XaUR(d [S5e[ R "[U5U45n[U5GHupVU(dM[R"[5nUHnX:aM Xx==S- ss'M [UR55HupUS:Xa XUU'MUS:XaU [U5- XEU'M/US:Xa SXEU'M>US:XagS[ R"U 5-n [ R"SURSURRUS5-- -5n X-XEU'M[S U5e GM U$) a`Converts a list of sequences into a Numpy matrix. Args: sequences: list of sequences (a sequence is a list of integer word indices). mode: one of "binary", "count", "tfidf", "freq" Returns: A Numpy matrix. Raises: ValueError: In case of invalid `mode` argument, or if the Tokenizer requires to be fit to sample data. r'zKSpecify a dimension (`num_words` argument), or fit on some text data first.tfidfz7Fit the Tokenizer on some data before using tfidf mode.countfreqbinaryrzUnknown vectorization mode:)r8r<rW ValueErrorr.npzeros enumerater3r6rrOrQlogr;rg) r>r_rwr8rErrcountsjr tfidfs rrxTokenizer.sequences_to_matrixs~~014  6 I 7?4#6#6I  HHc)ni0 1 *FA ,,S1F> Q V\\^,7?aDGV^#c(lADGX%ADGW_RVVAYB&&--T__5H5HA5N1NOPC!hADG$%BDII#-+4rc [R"UR5n[R"UR5n[R"UR5n[R"UR 5n[R"UR 5nURURURURURURURUUUUUS. $)a Returns the tokenizer configuration as Python dictionary. The word count dictionaries used by the tokenizer get serialized into plain JSON, so that the configuration can be read by other projects. Returns: A Python dictionary with the tokenizer configuration. ) r8r rr r9r:r.r5r7r;r=r<)jsondumpsr5r7r;r<r=r8r rr r9r:r.)r>json_word_countsjson_word_docsjson_index_docsjson_word_indexjson_index_words r get_configTokenizer.get_configs ::d&6&67DNN3**T__5**T__5**T__5||ZZZZ//"11+')))  rc UR5nURRUS.n[R"U40UD6$)aFReturns a JSON string containing the tokenizer configuration. To load a tokenizer from a JSON string, use `keras.preprocessing.text.tokenizer_from_json(json_string)`. Args: **kwargs: Additional keyword arguments to be passed to `json.dumps()`. Returns: A JSON string containing the tokenizer configuration. ) class_nameconfig)r __class____name__rr)r>r?rtokenizer_configs rto_jsonTokenizer.to_json-s?"..11 zz*5f55r) rr9r.r r;r=rr8r:r r5r7r<)N!!"#$%&()*+,-./:;<=>?@[\]^_`{|}~ TrqFNN)r)r __module__ __qualname____firstlineno____doc__r@r\r`rdrcrnrmryrxrr__static_attributes__rDrrr*r*s\#N6"!H:4x( >.` B@ >:x >6rr*z,keras.preprocessing.text.tokenizer_from_jsonc[R"U5nURS5n[R"URS55n[R"URS55n[R"URS55nUR 5VVs0sHupg[ U5U_M nnn[R"URS55nUR 5VVs0sHupg[ U5U_M nnn[R"URS55n [ S0UD6n X:lXJlXZl Xl Xl U $s snnfs snnf)aIParses a JSON tokenizer configuration and returns a tokenizer instance. Deprecated: `tf.keras.preprocessing.text.Tokenizer` does not operate on tensors and is not recommended for new code. Prefer `tf.keras.layers.TextVectorization` which provides equivalent functionality through a layer which accepts `tf.Tensor` input. See the [text loading tutorial](https://www.tensorflow.org/tutorials/load_data/text) for an overview of the layer and text handling in tensorflow. Args: json_string: JSON string encoding a tokenizer configuration. Returns: A TF-Keras Tokenizer instance rr5r7r;r=r<rD) rloadsrgr1rQrr*r5r7r;r<r=) json_stringrrr5r7r;kvr=r< tokenizers rtokenizer_from_jsonrBs!"zz+.  ! !( +F**VZZ 67K 6::k23IFJJ|45J(2(8(8(:;(:#a&!)(:J;FJJ|45J(2(8(8(:;(:#a&!)(:J;FJJ|45J#F#I'#%%% <;s *E(=E.)rTrq)rTrqN)NrTrqN)rr3r rr/numpyr tensorflow.python.util.tf_exportrrrrobjectr*rrDrrrs  :>? 3  )!@)!X01 3   /2/d67 2  7;87;t23|6|64|6~ <=">"r