6biTSrSSKrSSKrSSKJs Jr SSKJ r SSK J r SSK J r SSK Jr SSKJr SSKJr SS KJr SS KJr \R0r\R2r\R4r\R6r\R8rS rS r"S S\R>R@5r!"SS\ RD5r#"SS\ RH5r%g)z'Keras index lookup preprocessing layer.N)backend)base_layer_utils)base_preprocessing_layer)preprocessing_utils)layer_serialization) layer_utils)tf_utils) tf_loggingvocab idf_weightscD\rSrSrSrSr\S5r\S5rSr Sr g) NullInitializer,zEA placeholder initializer for restoring this layer from a SavedModel.cXlX lg)z{Construct a table initializer object. Args: key_dtype: Type of the table keys. value_dtype: Type of the table values. N _key_dtype _value_dtype)self key_dtype value_dtypes h/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/layers/preprocessing/index_lookup.py__init__NullInitializer.__init__/s$'cUR$)zThe expected table key dtype.)rrs rrNullInitializer.key_dtype9srcUR$)zThe expected table value dtype.)rrs rrNullInitializer.value_dtype>s   rcg)z$Returns the table initialization op.N)rtables r initializeNullInitializer.initializeCs rrN) __name__ __module__ __qualname____firstlineno____doc__rpropertyrrr#__static_attributes__r!rrrr,s7O(!! rrc:\rSrSrSrSr\S5rSrSr Sr g) VocabWeightHandlerHz;Adds the vocabulary as a layer weight during serialization.cxXlURUl[RR 5UlgN)_layervocabulary_dtype_dtypetf distribute get_strategy_distribute_strategy)r lookup_layers rrVocabWeightHandler.__init__Ks+ # "33 $&MM$>$>$@!rcg)Nr!rs r num_tensorsVocabWeightHandler.num_tensorsTsrc[R"USUR5nURR U5URlgNr)r4convert_to_tensorr3r1_lookup_table_from_tokens lookup_table)rweightstokenss r set_weightsVocabWeightHandler.set_weightsXs7%%gaj$++>#';;#H#H#P  rczURRSS9n[R"XR5nU/$)NFinclude_special_tokens)r1get_vocabularyr4r@r3)rrDs r get_tensorsVocabWeightHandler.get_tensors\s4++5+I%%fkk:xr)r7r3r1N) r%r&r'r(r)rr*r<rErKr+r!rrr-r-Hs*EAQrr-c^\rSrSrSrS$U4SjjrSrSrS%SjrSr Sr U4S jr S r S&S jr S rS rSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSr Sr!S r"\#S!5r$S"r%S#r&U=r'$)' IndexLookupcaMaps values from a vocabulary to integer indices. This layer translates a set of arbitrary hashables into an integer output via a table-based lookup, with optional out-of-vocabulary handling. This is the basis layer for both IntegerLookup and StringLookup; it holds the common logic but is not intended to be exported as part of the TF-Keras API. Args: max_tokens: The maximum size of the vocabulary for this layer. If None, there is no cap on the size of the vocabulary. Note that this size includes the OOV and mask tokens. num_oov_indices: The number of out-of-vocabulary tokens to use. If this value is more than 1, OOV inputs are hashed to determine their OOV value. If this value is 0, OOV inputs will cause an error when calling the layer. mask_token: A token that represents masked inputs. When `output_mode` is `"int"`, the token is included in vocabulary and mapped to index 0. In other output modes, the token will not appear in the vocabulary and instances of the mask token in the input will be dropped. If set to None, no mask term will be added. oov_token: Only used when `invert` is True. The token to return for OOV indices. vocabulary: Optional. Either an array or a string path to a text file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D tensor containing the vocbulary terms. If passing a file path, the file should contain one line per term in the vocabulary. If this argument is set, there is no need to `adapt` the layer. vocabulary_dtype: The dtype of the vocabulary terms. For example, `"int64"` or `"string"`. idf_weights: Only valid when `output_mode` is `"tf_idf"`. A tuple, list, 1D numpy array, or 1D tensor or the same length as the vocabulary, containing the floating point inverse document frequency weights, which will be multiplied by per sample term counts for the final `tf_idf` weight. If the `vocabulary` argument is set, and `output_mode` is `"tf_idf"`, this argument must be supplied. invert: Only valid when `output_mode` is `"int"`. If True, this layer will map indices to vocabulary items instead of mapping vocabulary items to indices. Defaults to `False`. output_mode: Specification for the output of the layer. Values can be `"int"`, `"one_hot"`, `"multi_hot"`, `"count"`, or `"tf_idf"` configuring the layer as follows: - `"int"`: Return the raw integer indices of the input tokens. - `"one_hot"`: Encodes each individual element in the input into an array the same size as the vocabulary, containing a 1 at the element index. If the last dimension is size 1, will encode on that dimension. If the last dimension is not size 1, will append a new dimension for the encoded output. - `"multi_hot"`: Encodes each sample in the input into a single array the same size as the vocabulary, containing a 1 for each vocabulary term present in the sample. Treats the last dimension as the sample dimension, if input shape is (..., sample_length), output shape will be (..., num_tokens). - `"count"`: As `"multi_hot"`, but the int array contains a count of the number of times the token at that index appeared in the sample. - `"tf_idf"`: As `"multi_hot"`, but the TF-IDF algorithm is applied to find the value in each token slot. Defaults to `"int"`. pad_to_max_tokens: Only valid when `output_mode` is `"multi_hot"`, `"count"`, or `"tf_idf"`. If True, the output will have its feature axis padded to `max_tokens` even if the number of unique tokens in the vocabulary is less than max_tokens, resulting in a tensor of shape [batch_size, max_tokens] regardless of vocabulary size. Defaults to False. sparse: Boolean. Only applicable to `"one_hot"`, `"multi_hot"`, `"count"` and `"tf-idf"` output modes. If True, returns a `SparseTensor` instead of a dense `Tensor`. Defaults to `False`. c  0 >UbUS::a[SU35eU (aUc[SU35eUS:a[SU35eU S:Xa[n U S:Xa[n [R"U [ [ [[[4URRSS 9 U(aU [ :wa[S U 35eU (aU [ :Xa[S U S U 35eUbU [:wa[S US U 35eXl Xl X l X0l X@lXlXlXlXPlU R'SS5UlX`lXplU R'SUSL5UlU R'SS5 SU ;a2U [ :Xa[0R2O[4R6"5U S'[8TU]t"S0U D6 U [ :XaB[0R<"UR>5R@(dU Sn [SU 35eU(afU [ :Xa URBO[0R2Ul"[0R<"UR$5Ul#SnUnURUl$O[0R<"UR$5Ul"U [ :Xa URBO[0R2Ul#UnUR[ :XaSOURFRJnURS:XaSUl$O-URS:XaURM5Ul$OSUl$URbJ[0RN"XRD5Ul([0RN"XRF5Ul)UR[:XaW[0RT"S/URW5-SUR>SS9Ul,URXR[5Ul.UbUR_Xg5 OURa5Ul1UR.(dURe[gU5S5 [0RhRjRmU[0R2SS9Ul7UR[:Xaf[0RhRjRmU[0R2SS9Ul8[0RT"S[0R2SS9Ul9ggg)Nr;zBIf set, `max_tokens` must be greater than 1. Received: max_tokens=zJIf pad_to_max_tokens is True, must set `max_tokens`. Received: max_tokens=rzP`num_oov_indices` must be greater than or equal to 0. Received: num_oov_indices=binaryztf-idf output_mode)allowable_strings layer_namearg_namezK`output_mode` must be `'int'` when `invert` is true. Received: output_mode=zt`sparse` may only be true if `output_mode` is `'one_hot'`, `'multi_hot'`, `'count'` or `'tf_idf'`. Received: sparse=z and output_mode=zW`idf_weights` should only be set if `output_mode` is `'tf_idf'`. Received: idf_weights=vocabulary_sizehas_input_vocabularyhas_static_tabledtypezMWhen `output_mode='int'`, `dtype` should be an integer type. Received: dtype=r0F)shaperY trainable)rr default_value)rYr\r!): ValueError MULTI_HOTTF_IDFrvalidate_string_argINTONE_HOTCOUNT __class__r%invert max_tokensnum_oov_indices mask_token oov_tokenrRsparsepad_to_max_tokensr2pop_frozen_vocab_sizeinput_vocabularyinput_idf_weights_has_input_vocabularyr4int64rfloatxsuperras_dtype compute_dtype is_integerrYrr_default_valuemax_oov_start_indexr@ _mask_key _mask_valueVariable_token_start_indexr valueidf_weights_constset_vocabulary_uninitialized_lookup_tablerB_add_trackabler-lookup experimentalMutableHashTable token_countstoken_document_counts num_documents)rrgrhrirjr2 vocabularyr rfrRrkrlkwargs input_dtypemask_key mask_valueres rrIndexLookup.__init__s"  !jAo((2|5  !3((2|5  Q --<,=?  ( "#K ( " K'' "GYvF~~.."   kS())4 7  kS($$*8,*m-   "{f'<55@MB*m-   $.$"& !2 0"(**->"E *!, &,ZZ "Zt%;& "  %t, & '3.GNN4D 7O "6" 3 KK 2 23>> /K))4 7  ,73,>djjBHHDO " D,A,A BD H#J"&..D  kk$*?*?@DO.9S.@ bhhD !H#..#54;L;L;P;PJ##q(')#%%*'+&;&;&=# ')# ?? &11(OOLDN!33-- D    v %!{{d--//((  D  &*%5%5%;%;%=D "  !    8 !% @ @ BD ))    24 8% @ " 6 6 G G*HH!H!D  6)II**;;"2$&HH&'<*&([[RXX&"**rcUR[:XaU$UR(a URO URn[ R "USU/5$r?)rRrbrlrgrnr4 TensorShape)r input_shapedepths rcompute_output_shape IndexLookup.compute_output_shapeasR   s " %% OO((  ~~{1~u566rcURURR55nUR(a URO UR n[ R"X#S9$)N)r[rY)rr[as_listrfr2rvr4 TensorSpec)r input_spec output_shape output_dtypes rcompute_output_signature$IndexLookup.compute_output_signatureksN001A1A1I1I1KL %)[[D ! !d6H6H }}<DDrc2^TRR5S:Xa//p2OUTRR5upETR(aXT4OXE4up#TR U5UR 5p2[ R"U4Sj[X255n[TR55Vs/sHovUPM nnTRb#TR[:XaTRUS'U(dUTR5SnU$s snf)aAReturns the current vocabulary of the layer. Args: include_special_tokens: If True, the returned vocabulary will include mask and OOV tokens, and a term's index in the vocabulary will equal the term's index when calling the layer. If False, the returned vocabulary will not include any mask or OOV tokens. rc>TR$r0)rjrsr,IndexLookup.get_vocabulary..s DNNrN)rBsizeexportrf_tensor_vocab_to_numpynumpy collections defaultdictziprangerVrirRrbr~)rrIr indiceskeysvaluesrxs` rrJIndexLookup.get_vocabularyrs    ! ! #q (7,,335LD/3{{f^NE++E2 (( "C$7 %*$*>*>*@$AB$Aq$AB ?? &4+;+;s+BE!H%$11356E Cs8Dc[R"5(aB[URR 5R 55UR 5-$URR 5UR 5-$)zGets the current size of the layer's vocabulary. Returns: The integer size of the vocabulary, including optional mask and oov indices. )r4executing_eagerlyintrBrrr~rs rrVIndexLookup.vocabulary_sizesn    ! !D%%**,2245))+,  $$))+d.E.E.GG GrcN[R"S5 UR5$)Nz5vocab_size is deprecated, please use vocabulary_size.)loggingwarningrVrs r vocab_sizeIndexLookup.vocab_sizesOP##%%rc >URURURURURUR UR URUR[R"UR5[R"UR5URS. n[TU]=5n[![#UR%55[#UR%55-5$)N) rfrgrhrjrirRrkrlr2r rrV)rfrgrhrjrirRrkrlr2utilslistify_tensorsrprornrt get_configdictlistitems)rconfig base_configres rrIndexLookup.get_configskk//#33//++kk!%!7!7 $ 5 5 001G1GH//0E0EF#66  g(* D**,-V\\^0DDEErcUR5 [R"5 UR5UlSSS5 g!,(df  g=fr0)_ensure_vocab_size_unchangedr4 init_scoperVrnrs r_record_vocabulary_size#IndexLookup._record_vocabulary_sizes2 ))+ ]]_&*&:&:&"XQSU5nU(aXSnOUnURAU5nU(a[S RU55eUR8bSUR8U;aC[,RB"XR8:H5S n [SUSUR8SU 35eUR:bdURD(aSUR:U;aC[,RB"XR::H5S n [SUSUR:SU 35eU[GU5-n URHb3XRH:a$[SRXRH55eURKU5Ul UR5 UR[:XGaeUSLGa^[GU5[GU5:wa-[SR[GU5[GU555eURMU5nURNS:wa#[SR[QU555eU(aS n S n OUn [,RR"U5n S nURT(a)URHbURHU - [GU5- nOS n[,RV"UX4SX4S9n[ RX"UURZS9nUR\R_U5 UR\Ra5Ul1ggg)aSets vocabulary (and optionally document frequency) for this layer. This method sets the vocabulary and idf weights for this layer directly, instead of analyzing a dataset through `adapt`. It should be used whenever the vocab (and optionally document frequency) information is already known. If vocabulary data is already present in the layer, this method will replace it. Args: vocabulary: Either an array or a string path to a text file. If passing an array, can pass a tuple, list, 1D numpy array, or 1D tensor containing the vocbulary terms. If passing a file path, the file should contain one line per term in the vocabulary. idf_weights: A tuple, list, 1D numpy array, or 1D tensor of inverse document frequency weights with equal length to vocabulary. Must be set if `output_mode` is `"tf_idf"`. Should not be set otherwise. Raises: ValueError: If there are too many inputs, the inputs do not match, or input data is missing. RuntimeError: If the vocabulary cannot be set when this function is called. This happens when `"multi_hot"`, `"count"`, and `"tf_idf"` modes, if `pad_to_max_tokens` is False and the layer itself has already been called. RuntimeError: If a tensor vocabulary is passed outside of eager execution. Nz2`idf_weights` must be set if output_mode is TF_IDFzU`idf_weights` should only be set if output_mode is `'tf_idf'`. Received: output_mode=z and idf_weights=zVocabulary file z does not exist.zGoutput_mode `'tf_idf'` does not support loading a vocabulary from file.zCannot set a tensor vocabulary on {} layer {} when not executing eagerly. Create this layer or call `set_vocabulary` outside of any `tf.function`s and with eager execution enabled.rz+Cannot set an empty vocabulary, you passed .zmThe passed vocabulary has at least one repeated term. Please uniquify your dataset. The repeated terms are {}rZaFound reserved mask token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: z. Received: mask_token=z at vocabulary index aFound reserved OOV token at unexpected location in `vocabulary`. Note that passed `vocabulary` does not need to include the OOV and mask tokens. Either remove all mask and OOV tokens, or include them only at the start of the vocabulary in precisely this order: z. Received: oov_token=zpAttempted to set a vocabulary larger than the maximum vocab size. Passed vocab size is {}, max vocab size is {}.Fzb`idf_weights` must be the same length as vocabulary. len(idf_weights) is {}, len(vocabulary) is {}r;z4TF-IDF data must be a 1-index array, but received {}constantconstant_valuesrY)2rRr`r^ isinstancestrr4iogfileexists_lookup_table_from_filerBrr is_tensor RuntimeErrorformatrer%namerrtuplenparrayrrrzr~rirjrh array_equal_find_repeated_tokensargwhererflenrgrA_convert_to_ndarrayndimtypeaveragerlpadr@rvr assignrr)rrr oov_start token_startspecial_tokensfound_special_tokensrDrepeated_tokens mask_index oov_indexnew_vocab_size front_paddingfront_padding_valueback_padding_value back_paddingrCs rrIndexLookup.set_vocabularys;8   v %" H# $5595E5E4FG##.-1  j# & &55;;%%j11 &zl2BC6) ,!% < &?$ ! &'# + &(jj&=#"# %%$//*EOOm3c+6FF ! ff-!4 I G **7$:L:LMG    # #G ,%)%5%5%;%;%=D "W+C %rc UR(a.[SRURR55e[ R "XRS9nURRS:Xa[R"US5nURRS:Xa[R"US5nURU5up#URRX#URRU5-5 UR [":Xa[R$"SU5nURU5up%UR&RX%UR&RU5-5 [(R*"U5(a*UR,R/UR155 gUR,R/[R"U[R2S9S5 gg)Nz^Cannot adapt {} layer after setting a static vocabulary via init argument or `set_vocabulary`.rrr;c4[R"U5S$r?)r4unique)rs rr*IndexLookup.update_state..s299Q<?r)out_type)rqr^rrer%r ensure_tensorr2r[rankr4 expand_dims _num_tokensrinsertrrRr`map_fnrr is_raggedr assign_addnrowsrr)rdatarDcountsdeduped_doc_data doc_countss r update_stateIndexLookup.update_statevs  % %''-vdnn.E.E'F  ""4/D/DE ::??a >>$*D ::??a >>$*D))$/   T..55f==    v %!yy)BDI !%!1!12B!C F  & & - -T%?%?%F%Fv%NN !!$''""--djjl;""--HHTBHH5a8 &rc|UR(d4[R"URR 5S5(aDUR [ :XaURR5Ul UR5 gURbEURR[R"UR/UR55 URbEURR[R"UR/UR55 URR!5up["R$"UR'5UR'545SSS2nUR)5nUR*(aUR*U- nUSUn[R,"X5nUR/U5UlUR [ :XGa#UR2R5U5nUR7X`R85n[R:"XpR<5n[R>"UUR)5S//[R@"U5S9nURB(aGUR*b:[R>"USUR*[R"U5- //SS9nURREU5 URR5Ul URG5 UR5 g)NrrZr)$rqr4equalrrrRr`r rrrriremover@r2rjrrlexsortrr~rggatherrArBrr_inverse_document_frequencyrcastrvr reduce_meanrlr reset_state)rrDrsorted_indicesrmax_learned_tokensrr s rfinalize_stateIndexLookup.finalize_states  % %$2C2C2H2H2JA)N)N6))-)9)9)?)?)A&  ( ( *  ?? &    $ $$$doo%68M8MN  >> %    $ $$$dnn%5t7L7LM **113V\\^V\\^$DEddK--/ ??!%;!> +,?-?@N62 ::6B   v %$($>$>$E$Ef$M !::%'9'9K''+/A/ABK&&))+Q/0 "{ ;K %%$//*E ff277;+??@A$%    # #K 0%)%5%5%;%;%=D "  $$&rc`UR(agURRURR5S5 UR[ :XaRUR RUR R5S5 URRS5 ggr?) rqrrrrRr`rrrrs rr IndexLookup.reset_states  % %    !2!2!9!9!;A!>?   v %  & & - -**113A6     % %a ( &rc UR5 [R"XRS9nURnURR S:XaUR US5n[R"U5(aF[R"URURUR5UR5nOW[R"U5(a+[R R#URU5nOURU5nUR$[&:Xa)UR S:Xa[R("US5nU$UR*(a UR,O UR.nUR$[0:Xa UR2OSn[R4"UUR$UUR6UR8US9$)NrrrZ)rRrrYrkr )_ensure_known_vocab_sizerrrr[r _expand_dimsr is_sparser4 SparseTensorr _lookup_denser dense_shaperraggedmap_flat_valuesrRrbsqueezerlrgrnr`rencode_categorical_inputsrvrk)rinputsoriginal_shapelookupsrr s rcallIndexLookup.callsx %%'$$V??C <<   !&&vr2F   f % %oo""6==1""G    ' 'ii//0B0BFKG((0G   s """a'**Wb1N%% OO((  '+&6&6&&@D " "d .. (($$;;#   rcH[R"5(a:[R"U5(a[R"XR S9nOUR RU5nURbA[R"XR5n[R"X0RU5nUR(aU$/nURS:Xa[R"[R"US55n[R"X5n[R R#SU45n[R$"[R"[R&"U5S5U/5nUR)U5 OURS:aUR*R,(a*[R.R1XR5nO'[R R3XRS9nXPR55-n[R"X R65n [R"XU5n[R8"U5 [R:"U5sSSS5 $!,(df  g=f)zALookup table values for a dense Tensor, handling masking and OOV.rNrrZzwWhen `num_oov_indices=0` all inputs should be in vocabulary, found OOV values {}, consider setting `num_oov_indices=1`.r;) num_buckets)r4rris_keras_tensor zeros_likerrBrrirr{wherer|rfrh gather_ndstringsrAssertrappendrrwmathfloormodto_hash_bucket_fastrzrxcontrol_dependenciesidentity) rrr!mask_locations lookup_checks oov_indices oov_inputsmsg assertion oov_locationss rrIndexLookup._lookup_dense s    ! !g&=&=f&E&EmmF2C2CDG''..v6G ?? &XXfnn=Nhh~/?/?IG ;;N   1 $((288GR#89Kf:J**##M C  "((277;+?"CcUKI   +  ! !A %)) gg..v7K7KL  jj<<(<(<= &(=(=(??KHHW.A.ABMhh}7CG  $ $] 3;;w'4 3 3s 3J J!chUR[:XaURR5US'ggNr )rRr`rrrstores rsave_own_variablesIndexLookup.save_own_variables=s.   v %#'#9#9#?#?#AE-  &rcUR[:Xa>URRUS5 URR 5Ulggr;)rRr`r rrrr<s rload_own_variablesIndexLookup.load_own_variablesAsE   v %    # #E-$8 9%)%5%5%;%;%=D " &rc dUR(agURSS9n[RRR US5n[ US5nURSR UVs/sHn[U5PM sn55 SSS5 gs snf!,(df  g=f)NTrHvocabulary.txtw ) rorJr4rrjoinopenwriter)rdir_pathrvocabulary_filepathfrEs r save_assetsIndexLookup.save_assetsFs   (((E  eekk..x9IJ %s +q GGDIIz:z!s1vz:; <, +:, +sB!3B B!B!! B/cXUR(ag[RRR US5n[ US5nUR 5RS5n[R"UR5[R:XaUVs/sHn[U5PM nnOUVs/sHn[U5PM nnUR[:XaURUSS9 OURU5 SSS5 gs snfs snf!,(df  g=f)NrDrrFF)r )ror4rrrGrHreadsplitrur2stringrrrRr`r)rrJrKrLlinesliners r load_assetsIndexLookup.load_assetsPs    eekk..x9IJ %s +qFFHNN4(E{{4001RYY>056#d)6056#d)66)##F#>##F+, +76 , +s+ ADD4D;D8D D D)c[R"5 [URUR5n[R R XR5sSSS5 $!,(df  g=fr0)r4rrrrrStaticHashTablerx)r initializers rr'IndexLookup._uninitialized_lookup_tablebsC ]]_)$//4;L;LMK99,,[:M:MN__s A A)) A7c[R"5 UR5nU[R"U5-nUR(a UR O UR n[R"X#US9nUR(aXQ4OX4upg[RRXgUR UR 5n[RRXR5sSSS5 $!,(df  g=f)Nr) r4rr~rrfrrrrKeyValueTensorInitializerrYrx) rrDr token_end indices_dtyperrrrZs rrA%IndexLookup._lookup_table_from_tokensgs ]]_113K#bggfo5I#';;D4E4E hh{]KG%)[[!v6G D))==doot/@/@K99,,[:M:MN__s CC>> D c UR(aI[RRRn[RRR nOH[RRR n[RRRn[R "5 [RRUURUURUUR5S9n[RRX@R5sSSS5 $!,(df  g=f)N)filenamer key_indexr value_indexvalue_index_offset) rfr4r TextFileIndex LINE_NUMBER WHOLE_LINErTextFileInitializerrrr~rYrx)rrbrcrdrZs rr#IndexLookup._lookup_table_from_filews ;; //;;I))11<>&/ /rcLURbUR[:XaS$S$)Nr;r)rirRrbrs rrzIndexLookup._oov_start_indexs(,1A1AS1HA NO rc<UR5UR-$r0)rzrhrs rr~IndexLookup._token_start_indexs$$&)=)===rcUR[:XdUR(agURc2[ SURS3R UR55eg)NWhen using `output_mode=z` and `pad_to_max_tokens=False`, you must set the layer's vocabulary before calling it. Either pass a `vocabulary` argument to the layer, or call `adapt` with some sample data.)rRrbrlrnrrrs rr$IndexLookup._ensure_known_vocab_sizesc   s "d&<&<   " " **4+;+;*<=))*00@0@)A   +rcZUR[:XdUR(ag[R"5 UR 5nSSS5 UR b9WUR :wa([SURSUR SU35egg!,(df  NU=f)Nruzt` and `pad_to_max_tokens=False`, the vocabulary size cannot be changed after the layer is called. Old vocab size is z, new vocab size is )rRrbrlr4rrVrnr)rrs rr(IndexLookup._ensure_vocab_size_unchangeds   s "d&<&<  ]]_!113N  # # /$"9"99*4+;+;*<=-.2-D-D,EF%%3$4 6 : 0 _s B B*c[U5n[U5[U5:waA[R"U5R 5VVs/sHup4US:dM UPM snn$/$s snnf)z+Return all repeated tokens in a vocabulary.r;)setrrCounterr)rrvocabulary_setitemcounts rr!IndexLookup._find_repeated_tokenssgZ z?c.1 1$/#6#6z#B#H#H#J#JKD19#J  I s A'A'c$[R"U5(a URnO@[R"U5(a URnO[ R "US/5n[ R"U[ RS9up4nX54$)z?Count the number of tokens in a ragged, sparse or dense tensor.rZ)out_idx) r rrr flat_valuesr4reshapeunique_with_countsrr)rrrrD_rs rrIndexLookup._num_tokenssm   d # #++K    % %**K**TB40K11+rxxP6~rcR[RRSUSU-- -5$)a~Computes the inverse-document-frequency (IDF) component of "tf_idf". Uses the default weighting scheme described in https://en.wikipedia.org/wiki/Tf%E2%80%93idf. Args: token_document_counts: An array of the # of documents each token appears in. num_documents: An int representing the total number of documents Returns: An array of "inverse document frequency" weights. r;)r4r-log)rrrs rr 'IndexLookup._inverse_document_frequencys'ww{{1}4I0IJJKKrc.[R"U5$r0)rVocabularySavedModelSaverrs r_trackable_saved_model_saver(IndexLookup._trackable_saved_model_savers"<rArMrVrrArrrrzr~rrrrr r*rrr+ __classcell__)res@rrNrNcsBVwr7E> H&F$= y>v"H;'z )) V.(`B> =,$O O O$B0  > &  L CC""rrN)&r)rrrtensorflow.compat.v2compatv2r4 tf_keras.srcrtf_keras.src.enginerr!tf_keras.src.layers.preprocessingrr&tf_keras.src.saving.legacy.saved_modelrtf_keras.src.utilsrr tensorflow.python.platformr rrbr_rcrdr` _VOCAB_NAME_IDF_WEIGHTS_NAMErr]rTrackableWeightHandlerr-PreprocessingLayerrNr!rrrs.!! 08JF*'= ii OO --   ! bii99 8)@@6D"*==D"r