h`%jSSKJr SSKrSSKJrJr SSKJr SSKJr "SS\R5r g)) annotationsN)Tensornn) CrossEncoder)fullnamec~^\rSrSr\R "54SU4SjjjrS SjrS SjrSr \ S Sj5r Sr U=r $) MarginMSELoss c >[TU]5 XlX l[R "S0UD6Ul[UR[5(d8[URRS[UR5S35eURRS:wa9[URRSURRS35eg)a# Computes the MSE loss between ``|sim(Query, Pos) - sim(Query, Neg)|`` and ``|gold_sim(Query, Pos) - gold_sim(Query, Neg)|``. This loss is often used to distill a cross-encoder model from a teacher cross-encoder model or gold labels. In contrast to :class:`~sentence_transformers.cross_encoder.losses.MultipleNegativesRankingLoss`, the two passages do not have to be strictly positive and negative, both can be relevant or not relevant for a given query. This can be an advantage of MarginMSELoss over MultipleNegativesRankingLoss. .. note:: Be mindful of the magnitude of both the labels and what the model produces. If the teacher model produces logits with Sigmoid to bound them to [0, 1], then you may wish to use a Sigmoid activation function in the loss. Args: model (:class:`~sentence_transformers.cross_encoder.CrossEncoder`): A CrossEncoder model to be trained. activation_fn (:class:`~torch.nn.Module`): Activation function applied to the logits before computing the loss. **kwargs: Additional keyword arguments passed to the underlying :class:`torch.nn.MSELoss`. References: - Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation: https://arxiv.org/abs/2010.02666 - `Cross Encoder > Training Examples > Distillation <../../../examples/cross_encoder/training/distillation/README.html>`_ Requirements: 1. Your model must be initialized with `num_labels = 1` (a.k.a. the default) to predict one class. 2. Usually uses a finetuned CrossEncoder teacher M in a knowledge distillation setup. Inputs: +------------------------------------------------+--------------------------------------------------------------------------------------------+-------------------------------+ | Texts | Labels | Number of Model Output Labels | +================================================+============================================================================================+===============================+ | (query, passage_one, passage_two) triplets | gold_sim(query, passage_one) - gold_sim(query, passage_two) | 1 | +------------------------------------------------+--------------------------------------------------------------------------------------------+-------------------------------+ | (query, passage_one, passage_two) triplets | [gold_sim(query, passage_one), gold_sim(query, passage_two)] | 1 | +------------------------------------------------+--------------------------------------------------------------------------------------------+-------------------------------+ | (query, positive, negative_1, ..., negative_n) | [gold_sim(query, positive) - gold_sim(query, negative_i) for i in 1..n] | 1 | +------------------------------------------------+--------------------------------------------------------------------------------------------+-------------------------------+ | (query, positive, negative_1, ..., negative_n) | [gold_sim(query, positive), gold_sim(query, negative_1), ..., gold_sim(query, negative_n)] | 1 | +------------------------------------------------+--------------------------------------------------------------------------------------------+-------------------------------+ Relations: - :class:`MSELoss` is similar to this loss, but without a margin through the negative pair. Example: :: from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, losses from datasets import Dataset student_model = CrossEncoder("microsoft/mpnet-base") teacher_model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L12-v2") train_dataset = Dataset.from_dict({ "query": ["What are pandas?", "What is the capital of France?"], "positive": ["Pandas are a kind of bear.", "The capital of France is Paris."], "negative": ["Pandas are a kind of fish.", "The capital of France is Berlin."], }) def compute_labels(batch): positive_scores = teacher_model.predict(list(zip(batch["query"], batch["positive"]))) negative_scores = teacher_model.predict(list(zip(batch["query"], batch["negative"]))) return { "label": positive_scores - negative_scores } train_dataset = train_dataset.map(compute_labels, batched=True) loss = losses.MarginMSELoss(student_model) trainer = CrossEncoderTrainer( model=student_model, train_dataset=train_dataset, loss=loss, ) trainer.train() z? expects a model of type CrossEncoder, but got a model of type .z; expects a model with 1 output label, but got a model with z output labels.N)super__init__model activation_fnrMSELossloss_fct isinstancer ValueError __class____name__type num_labels)selfrrkwargsrs r/home/james-whalen/.local/lib/python3.13/site-packages/sentence_transformers/cross_encoder/losses/MarginMSELoss.pyrMarginMSELoss.__init__ sT  * ,V, $**l33>>**+,++/ +;*>**+,((, (=(='>oO  &c 0USnUSnUSSn[U5n[U[5(a[R"U5nUR U[U5S-4:Xa$USS2S4R S5USS2SS24- nUR U4:XaUR S5nUR U[U54:wa'[SUR SU[U54S35e[[X455nURU5n/n UH7n [[X:55n U RURU 55 M9 UR S5[R"U SS9- n URXR55n U $)Nrr z Labels shape z does not match expected shape z. Ensure that your dataset labels/scores are 1) lists of differences between positive scores and negatives scores (length `num_negatives`), or 2) lists of positive and negative scores (length `num_negatives + 1`).)dim) lenrlisttorchstackshape unsqueezerziplogits_from_pairsappendrfloat)rinputslabelsanchors positives negatives batch_sizepositive_pairspositive_logitsnegative_logits_listnegativenegative_pairs margin_logitslosss rforwardMarginMSELoss.forwardfs)1I 12J \  fd # #[[(F <F <(>~(N O"(11!4u{{CW]^7__ }}]LLN; rcURRUSSSS9nURURR5 UR"S0UD6SR S5nUR U5$)z Computes the logits for a list of pairs using the model. Args: pairs (list[tuple[str, str]]): A list of pairs of strings (query, passage). Returns: Tensor: The logits for the pairs. Tpt)padding truncationreturn_tensorsrr)r tokenizertodeviceviewr)rpairstokenslogitss rr*MarginMSELoss.logits_from_pairsss%%  &   $**##$%f%a(--b1!!&))rc0S[UR50$)Nr)rrrs rget_config_dictMarginMSELoss.get_config_dicts Xd&8&89  rcg)NuY @misc{hofstätter2021improving, title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation}, author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury}, year={2021}, eprint={2010.02666}, archivePrefix={arXiv}, primaryClass={cs.IR} } rrKs rcitationMarginMSELoss.citations r)rrr)rrrz nn.ModulereturnNone)r-zlist[list[str]]r.zTensor | list[Tensor]rQr)rFzlist[tuple[str, str]]rQr)rQstr)r __module__ __qualname____firstlineno__rIdentityrr:r*rLpropertyrO__static_attributes__ __classcell__)rs@rr r s=GI{{}YYv%N*(   rr ) __future__rr%rr0sentence_transformers.cross_encoder.CrossEncoderrsentence_transformers.utilrModuler rrrr_s&" I/gBIIgr