h "SSKJr SSKrSSKrSSKrSSKJr SSKJr SSK r SSK J r SSK Jr \(aSSK Jr \R "\5r"SS \5rg!\a SSK Jr NIf=f) ) annotationsN) TYPE_CHECKING)Self) AutoTokenizer) InputModule)PreTrainedTokenizerc^\rSrSr%SrS/rS\S'SSU4SjjjrSSjrSS .SS jjr \ SSS jj5r \ SSS jj5r SS jr SSjrSSSjjrSrU=r$)SparseStaticEmbeddinga SparseStaticEmbedding module for efficient sparse representations. This lightweight module computes sparse representations by mapping input tokens to static weights, such as IDF (Inverse Document Frequency) weights. It is designed to encode queries or documents into fixed-size embeddings based on the presence of tokens in the input. A common scenario is to use this module for encoding queries, and using a heavier module like SPLADE (MLMTransformer + SpladePooling) for document encoding. Args: tokenizer (PreTrainedTokenizer): PreTrainedTokenizer to tokenize input texts into input IDs. weight (torch.Tensor | None): Static weights for vocabulary tokens (e.g., IDF weights), shape should be (vocab_size,). If None, initializes weights to a vector of ones. Default is None. frozen (bool): Whether the weights should be frozen (not trainable). Default is False. frozenz list[str] config_keysc>[TU]5 XlUb([RR X#(+S9UlO][RR [R"[URR555U(+S9UlX0l UR RS5Ul URRUlg)N) requires_gradr)super__init__ tokenizertorchnn Parameterweightoneslen get_vocabr sizenum_dimensionsmodel_max_lengthmax_seq_length)selfrrr __class__s {/home/james-whalen/.local/lib/python3.13/site-packages/sentence_transformers/sparse_encoder/models/SparseStaticEmbedding.pyrSparseStaticEmbedding.__init__-s "  ((,,V:,NDK((,,UZZDNNE__Q' Z1D1DYM]M]ejepepq 9=$kk1   )#8J)3%&r"Tsafe_serializationcfURU5 URXS9 URU5 g)Nr5)save_tokenizersave_torch_weights save_config)r output_pathr6argskwargss r saveSparseStaticEmbedding.saveUs/ K(  S %r"c r[RRU5(dSSKJn U"USUUUUS9n[ U5n [R"U 5n SSS5 [W R56upUR[U 55n [R"U [RS9n [!U 5S -n[R""U[RS9n[X5H unnUUU'M U"S XS .UD6$![ a [ SUS35ef=f!,(df  N=f) a Create an SparseStaticEmbedding module from a JSON file containing token to IDF weight mappings. Args: json_path (str): Path to the JSON file containing token to IDF weight mappings. tokenizer (PreTrainedTokenizer): Tokenizer to use for converting tokens to IDs. token (bool | str | None): Token for Hugging Face authentication cache_folder (str | None): Cache folder for Hugging Face revision (str | None): Model revision local_files_only (bool): Whether to only load local files **config: Additional configuration options for the IDF model. Returns: SparseStaticEmbedding: An initialized SparseStaticEmbedding model. r)hf_hub_downloadzidf.json)repo_idfilenametoken cache_dirrevisionlocal_files_onlyzIDF JSON file not found at z. Please provide a valid path.N)r(r)rr)ospathexistshuggingface_hubrA ValueErroropenjsonloadzipitemsconvert_tokens_to_idslistrtensorfloat32maxr,)cls json_pathrrD cache_folderrFrGconfigrAfInidftokensweights token_ids max_token_idrtoken_idws r from_jsonSparseStaticEmbedding.from_jsonZs4ww~~i(( j;+%'*%%5  )_))C.Csyy{+33DLA ,,wemm<9~) \?y2KHa F8 3@&@@@ j #>ykIg!hii j_sD D( D%( D6c (URUUUUUUS9n[R"UUUUUUS9n URSS5n U b/U R S5(aUR "U U 4UUUUS.UD6$U"S SU S.UD6n UR UUUUUUU S9n U $) a/ Load the SparseStaticEmbedding module with its tokenizer. Args: model_name_or_path (str): Path to the directory containing the saved model. subfolder (str): Subfolder within the model directory token (bool | str | None): Token for Hugging Face authentication cache_folder (str | None): Cache folder for Hugging Face revision (str | None): Model revision local_files_only (bool): Whether to only load local files **kwargs: Additional keyword arguments Returns: SparseStaticEmbedding: The loaded SparseStaticEmbedding module. )model_name_or_path subfolderrDr[rFrG)rirDrErFrGrKNz.json)rDrErFrGrH)rhrirDr[rFrGmodelrI) load_configrfrom_pretrainedpopendswithreload_torch_weights) rYrhrirDr[rFrGr=r\rrKrjs r rQSparseStaticEmbedding.loads41%- ! "11 "-  zz&$'   g 6 6==&!!1  ?49??&&1%-'  r"cSURRR3nSUR5SURUS3$)Nz , tokenizer=zSparseStaticEmbedding(z, dim=))rr__name__get_config_dictr)rtokenizer_infos r __repr__SparseStaticEmbedding.__repr__sL'(@(@(I(I'JK'(<(<(>'?vdFYFYEZ[iZjjkllr"cUR$)N)r)rs r get_sentence_embedding_dimension6SparseStaticEmbedding.get_sentence_embedding_dimensions"""r"c 8[URXSSSS95$)NTptF)padding truncationreturn_tensorsadd_special_tokens)dictr)rtextsr}s r tokenizeSparseStaticEmbedding.tokenizes' NN5dSWlqN r  r")r rrrr)NF)rrrztorch.Tensor | Noner bool)r/dict[str, torch.Tensor]returnr)r;strr6rrNone)NNNF) rZrrrrDbool | str | Noner[ str | NonerFrrGr)NNNF)rhrrirrDrr[rrFrrGrrr)rr)rint)T)rz.list[str] | list[dict] | list[tuple[str, str]]r}z str | boolrr)rs __module__ __qualname____firstlineno____doc__r __annotations__rr3r> classmethodrerQrvryr__static_attributes__ __classcell__)rs@r r r sa$'ZK' '+ >&>$> >>$,HL&  $(#'#!&4A4A'4A! 4A ! 4A  4A4A4Al#'#'#!&DDD! D ! D  DD DDLm#\` C NX   r"r ) __future__rrPloggingrJtypingrr ImportErrortyping_extensionsr transformersr(sentence_transformers.models.InputModulerr getLoggerrsloggerr rIr"r rs_"  ' &@0   8 $L KL '&'sA A*)A*