hu%SSKJr SSKJr SSKrSSKJr SSKJs Jr SSK J r SSK J r S Sjr"SS\R5r"S S \R5rg) ) annotations)IterableN)"SparseMultipleNegativesRankingLoss) SparseEncodercURSS9n[R"X5[R"USSS24RUR5U5- nU$)z :param reconstruction: output of Autoencoder.decode (shape: [batch, n_inputs]) :param original_input: input of Autoencoder.encode (shape: [batch, n_inputs]) :return: normalized mean squared error (shape: [1]) r)dimN)meanFmse_loss broadcast_toshape)reconstructionoriginal_inputoriginal_input_meanlosss m/home/james-whalen/.local/lib/python3.13/site-packages/sentence_transformers/sparse_encoder/losses/CSRLoss.pynormalized_mean_squared_errorrs\ )--!-4 ::n 5 D!G$11.2F2FG9 D KcJ^\rSrSrSSU4SjjjrS SjrS SjrSrSrU=r $) CSRReconstructionLossc:>[TU]5 XlX lg)a CSRReconstructionLoss implements the reconstruction loss component for Contrastive Sparse Representation (CSR) models. This loss ensures that the sparse encoding can accurately reconstruct the original model embeddings through three components: 1. A primary reconstruction loss (L_k) that measures the error between the original embedding and its reconstruction using the top-k sparse components. 2. A secondary reconstruction loss (L_4k) that measures the error using the top-4k sparse components. 3. An auxiliary loss (L_aux) that helps to learn residual information. Args: model: SparseEncoder model with autoencoder components beta: Weight for the auxiliary loss component (L_aux) References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://arxiv.org/abs/2503.01776 Requirements: 1. The model must be configured to output the necessary reconstruction components 2. Used with SparseEncoder models that implement compositional sparse autoencoding Relations: - Used as a component within :class:`CSRLoss` combined with a contrastive loss Example: - This loss is never used standalone, but instead used within the :class:`CSRLoss` class. See that loss for more details. N)super__init__modelbeta)selfrr __class__s rrCSRReconstructionLoss.__init__s<   rc[S5e)Nz[CSRReconstructionLoss is not intended to be used standalone. Use it within CSRLoss instead.)NotImplementedError)rsentence_featuress rforwardCSRReconstructionLoss.forward?s! i  rc`SnSnSnUHpnUSnUSnUSnUSn USn [R"Xg5n [R"Xh5n [XU R5- 5n X+- nX<- nXM- nMr [ U5nUS:a X.-nX>-nXN-nUUS- UR U-S .$) z Compute the CSRReconstruction loss from embeddings. Args: outputs: List of dictionaries containing sentence embeddings and their sparse representations Returns: total_loss: The total reconstruction loss value gsentence_embedding_backbonedecoded_embedding_kdecoded_embedding_4kdecoded_embedding_auxdecoded_embedding_k_pre_biasrg @)reconstruction_loss_kreconstruction_loss_4kreconstruction_loss_aux)r r rdetachlenr)routputs total_L_k total_L_4k total_L_auxfeaturesxrecons_k recons_4k recons_auxreconsk_pre_biasL_kL_4kL_aux num_columnss rcompute_loss_from_embeddings2CSRReconstructionLoss.compute_loss_from_embeddingsDs    H67A 56H !78I!"9:J'(FG **Q)C::a+D2*BRBYBYB[>[\E  I  J  K' ,'l ?  $I  %J  &K&/&03&6'+yy;'>  rcSUR0$)d Get the configuration dictionary. Returns: Dictionary containing the configuration parameters r)rrs rget_config_dict%CSRReconstructionLoss.get_config_dictws ""r)rr)?)rrrfloatreturnNone)r"!Iterable[dict[str, torch.Tensor]]rGdict[str, torch.Tensor])r0zlist[dict[str, torch.Tensor]]rGrJ) __name__ __module__ __qualname____firstlineno__rr#r>rC__static_attributes__ __classcell__rs@rrrs#  D 1 f##rrcd^\rSrSrSSU4SjjjrS S SjjrSr\S Sj5rSr U=r $) CSRLossc>[TU]5 XlX0lX@l[ XS9UlUbX lg[US9Ulg)a  CSRLoss implements a combined loss function for Contrastive Sparse Representation (CSR) models. This loss combines two components: 1. A reconstruction loss :class:`CSRReconstructionLoss` that ensures the sparse representation can faithfully reconstruct the original embedding. 2. A main loss, which in the paper is a :class:`SparseMultipleNegativesRankingLoss` that ensures semantically similar sentences have similar representations. The total loss is linear combination of the two losses. Args: model: SparseEncoder model loss: The principal loss function to use can be any of the SparseEncoder losses except flops loss and CSRReconstruction loss. If None, the default loss is used, which is the SparseMultipleNegativesRankingLoss. beta: Weight for the L_aux component in the reconstruction loss. Default is 0.1. gamma: Weight for the main loss component (MNRL a.k.a. InfoNCE by default). Default is 1.0. References: - For more details, see the paper "Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation" https://arxiv.org/abs/2503.01776 Requirements: 1. Input requirements depend on the chosen loss 2. Uses autoencoder components of the SparseEncoder model Relations: - Uses :class:`CSRReconstructionLoss` for the reconstruction component Example: :: from datasets import Dataset from sentence_transformers.sparse_encoder import SparseEncoder, SparseEncoderTrainer, losses model = SparseEncoder("sentence-transformers/all-MiniLM-L6-v2") train_dataset = Dataset.from_dict( { "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], "negative": ["It's quite rainy, sadly.", "She walked to the store."], } ) loss = losses.CSRLoss(model, beta=0.1, gamma=1.0) trainer = SparseEncoderTrainer(model=model, train_dataset=train_dataset, loss=loss) trainer.train() )rrN)r) rrrrgammarreconstruction_lossrr)rrrrrVrs rrCSRLoss.__init__sFd    $9u#P  ,D 2T[`2a rcUVs/sHo0RU5PM nnUVs/sHoUSPM nnURRU5nURRXb5n[ U[ 5(a+UR 5HupXR-Xy'M U$XR-US'U$s snfs snf)Nsentence_embedding base_loss)rrWr>r isinstancedictitemsrV) rr"labelssentence_featurer0outputrZlossesr[keyvalues rr#CSRLoss.forwardsIZZHY4D::./HYZIPQv%9:Q))FFwOII::;MV i & &'oo/ #jj0 0  #,jj"8F;  [Qs B>CcJURURURS.$)rArrVrrgrBs rrCCSRLoss.get_config_dicts DJJ JJrcg)Na @misc{wen2025matryoshkarevisitingsparsecoding, title={Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation}, author={Tiansheng Wen and Yifei Wang and Zequn Zeng and Zhong Peng and Yudi Su and Xinyang Liu and Bo Chen and Hongwei Liu and Stefanie Jegelka and Chenyu You}, year={2025}, eprint={2503.01776}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2503.01776}, } rBs rcitationCSRLoss.citations r)rrVrrrW)Ng?rE)rrrznn.Module | NonerrFrVrF)N)r"rIr_ztorch.Tensor | NonerGrJ)rGstr) rKrLrMrNrr#rCpropertyrkrOrPrQs@rrSrSsN9b9bxcg!BL_ $K  rrS)r torch.TensorrrorGro) __future__rcollections.abcrtorchtorch.nnnntorch.nn.functional functionalr Nsentence_transformers.sparse_encoder.losses.SparseMultipleNegativesRankingLossr2sentence_transformers.sparse_encoder.SparseEncoderrrModulerrSrjrrrzsL"$ M b#BIIb#Jcbiicr