haSSKJrJr SSKJr SSKJrJrJrJ r SSK J r "SS\5r "SS5r \"S S \ 55rg ) ) dataclassfield)Enum)AnyCallableOptionalUnion)TrainingArgumentsc \rSrSrSrSrSrSrg)FDivergenceType reverse_kl js_divergencealpha_divergenceN)__name__ __module__ __qualname____firstlineno__ REVERSE_KL JS_DIVERGENCEALPHA_DIVERGENCE__static_attributes__rP/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/dpo_config.pyr r sJ#M)rr c\rSrSrSrSrSrg)FDivergenceConstantsalpha_divergence_coef?rN)rrrrALPHA_DIVERGENCE_COEF_KEYALPHA_DIVERGENCE_COEF_DEFAULTrrrrrrs 7$'!rrc:^\rSrSr%Sr\R SS/-r\"SSS0S9r\ \ S '\"S SS 0S9r \ \ S '\"S SS0S9r \ \ S'\"SSS0S9r\\ \ S'\"SSS0S9r\\\\4\ S'\"SSS0S9r\\\\4\ S'\"SSS0S9r\\\ S'\"SSS0S9r\\\ S'\"SSS0S9r\ \ S'\"S SS0S9r\ \ S'\"SSS0S9r\ \ S'\"SSS 0S9r\\\ S!'\"SSS"0S9r\\\ S#'\"S$SS%0S9r\\ S&'\"S'SS(0S9r\\\ S)'\"SSS*0S9r\\\ S+'\"S,SS-0S9r \\\ S.'\"S/S0S/S1/S2.S9r!\\ S3'\"SSS40S9r"\ \ S5'\"SSS60S9r#\ \ S7'\"SSS80S9r$\\\ S9'\"SSS:0S9r%\\&\'\\(4\ S;'\"S<SS=0S>9r)\&\\ S?'\"SSS@0S9r*\ \ SA'\"SBSSC0S9r+\\ SD'\"SESSF0S9r,\ \ SG'\"\-R\SSH0S9r/\-\ SI'\"SJSSK0S9r0\ \ SL'\"SSSM0S9r1\ \ SN'\"SOSSP0S9r2\ \ SQ'\"SSSR0S9r3\ \ SS'\"SSST0S9r4\\ \ SU'\"SSSV0S9r5\\ \ SW'\"SXSSY0S9r6\ \ SZ'\"SSS[0S9r7\\&\ \ S\'\"SSS]0S9r8\ \ S^'\"S_SS`0S9r9\ \ Sa'\"S'SSb0S9r:\\ Sc'\"SSSd0S9r;\ \ Se'U4Sfjr$)h DPOConfig!u+ Configuration class for the [`DPOTrainer`]. This class includes only the parameters that are specific to DPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model and reference model model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the [`DPOTrainer`] is provided as a string. ref_model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `ref_model` argument of the [`DPOTrainer`] is provided as a string. model_adapter_name (`str` or `None`, *optional*, defaults to `None`): Name of the train target PEFT adapter, when using LoRA with multiple adapters. ref_adapter_name (`str` or `None`, *optional*, defaults to `None`): Name of the reference PEFT adapter, when using LoRA with multiple adapters. force_use_ref_model (`bool`, *optional*, defaults to `False`): If you provide a PEFT model as the active model and wish to use a different model for the `ref_model`, set this flag to `True`. disable_dropout (`bool`, *optional*, defaults to `True`): Whether to disable dropout in the model and reference model. use_logits_to_keep (`bool`, *optional*, defaults to `False`): If `True`, only a specified number of logits are computed in the forward pass. This can be useful for saving memory and speeding up training by not computing the logits for all tokens, especially in scenarios when working with very long prompts where labels are ignored (-100). > Parameters that control the data preprocessing dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of processes to use for processing the dataset. padding_value (`int` or `None`, *optional*, defaults to `None`): Padding value to use. If `None`, the padding value of the tokenizer is used. label_pad_token_id (`int`, *optional*, defaults to `-100`): Padding value to use for labels. max_prompt_length (`int` or `None`, *optional*, defaults to `512`): Maximum length of the prompt. max_completion_length (`int` or `None`, *optional*, defaults to `None`): Maximum length of the completion. max_length (`int` or `None`, *optional*, defaults to `1024`): Maximum length of the full sequence (prompt + completion). truncation_mode (`str`, *optional*, defaults to `"keep_end"`): Truncation mode to use when the sequence exceeds `max_length`. Possible values are `"keep_end"` and `"keep_start"`. padding_free (`bool`, *optional*, defaults to `False`): Whether to perform forward passes without padding by flattening all sequences in the batch into a single continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only supported with the `flash_attention_2` attention implementation, which can efficiently handle the flattened batch structure. precompute_ref_log_probs (`bool`, *optional*, defaults to `False`): Whether to precompute the log probabilities from the reference model. Setting this to `True` allows training without needing the reference model during training, which can help reduce GPU memory usage. If set to `False` (default), the reference model will be used during training to compute log probabilities on-the-fly. precompute_ref_batch_size (`int` or `None`, *optional*, defaults to `None`): Batch size to use when precomputing reference model log probabilities. This can be set higher than the training batch size to speed up preprocessing. If `None`, defaults to `per_device_train_batch_size` for training and `per_device_eval_batch_size` for evaluation. tools (`Optional[list[Union[dict, Callable]]]`, *optional*, defaults to `None`): List of tools (callable functions) that will be accessible to the model. If the template does not support function calling, this argument will have no effect. > Parameters that control the training loss_type (`str` or `list[str]`, *optional*, defaults to `"sigmoid"`): Type of loss to use. Possible values are: - `"sigmoid"`: sigmoid loss from the original [DPO](https://huggingface.co/papers/2305.18290) paper. - `"hinge"`: hinge loss on the normalized likelihood from the [SLiC](https://huggingface.co/papers/2305.10425) paper. - `"ipo"`: IPO loss from the [IPO](https://huggingface.co/papers/2310.12036) paper. - `"exo_pair"`: pairwise EXO loss from the [EXO](https://huggingface.co/papers/2402.00856) paper. - `"nca_pair"`: pairwise NCA loss from the [NCA](https://huggingface.co/papers/2402.05369) paper. - `"robust"`: unbiased estimate of the DPO loss that is robust to preference noise from the [Robust DPO](https://huggingface.co/papers/2403.00409) paper. - `"bco_pair"`: pairwise BCO loss from the [BCO](https://huggingface.co/papers/2404.04656) paper. - `"sppo_hard"`: SPPO loss with hard label from the [SPPO](https://huggingface.co/papers/2405.00675) paper. - `"aot"`: AOT loss for paired datasets from the [AOT](https://huggingface.co/papers/2406.05882) paper. - `"aot_pair"`: AOT loss for unpaired datasets from the [AOT](https://huggingface.co/papers/2406.05882) paper. - `"discopop"`: DiscoPOP (a.k.a Log-Ratio Modulated Loss, LRML) loss from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper. - `"apo_zero"`: APO-zero loss from the [APO](https://huggingface.co/papers/2408.06266) paper. - `"apo_down"`: APO-down loss from the [APO](https://huggingface.co/papers/2408.06266) paper. - `"sft"`: Negative log-likelihood loss (standard supervised fine-tuning loss). Multiple loss types can be combined using comma separation (e.g., `["sigmoid", "bco_pair", "sft"]` for [MPO](https://huggingface.co/papers/2411.10442)). The `loss_weights` parameter can be used to specify corresponding weights for each loss type. use_liger_loss (`bool`, *optional*, defaults to `False`): Whether to use Liger loss. base_model_attribute_name (`str`, *optional*, defaults to `"model"`): Name of the attribute in the model that contains the base model. This is used to get the base model from the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`. beta (`float`, *optional*, defaults to `0.1`): Parameter controlling the deviation from the reference model. Higher β means less deviation from the reference model. For the IPO loss (`loss_type="ipo"`), β is the regularization parameter denoted by τ in the [paper](https://huggingface.co/papers/2310.12036). f_divergence_type (`str`, *optional*, defaults to `FDivergenceType.REVERSE_KL`): Type of f-divergence regularization function to compute divergence between policy and reference model. f_alpha_divergence_coef (`float`, *optional*, defaults to `1.0`): α coefficient in the α-divergence u^-α regularization function for DPO loss. reference_free (`bool`, *optional*, defaults to `False`): Whether to ignore the provided reference model and implicitly use a reference model that assigns equal probability to all responses. label_smoothing (`float`, *optional*, defaults to `0.0`): Robust DPO label smoothing parameter from the [cDPO report](https://ericmitchell.ai/cdpo.pdf) and [Robust DPO](https://huggingface.co/papers/2403.00409) paper that should be between `0.0` and `0.5`. use_weighting (`bool`, *optional*, defaults to `False`): Whether to weight the loss as done in the [WPO paper](https://huggingface.co/papers/2406.11827). rpo_alpha (`float`, *optional*, defaults to `None`): α parameter from the [RPO paper](https://huggingface.co/papers/2404.19733) (v3), which controls the weighting of the NLL term in the loss. If `None`, no weighting is applied and the loss is the same as the DPO loss. The paper recommends `rpo_alpha=1.0`. ld_alpha (`float` or `None`, *optional*, defaults to `None`): α parameter from the [LD-DPO paper](https://huggingface.co/papers/2409.06411), which controls the weighting of the verbose token log-probabilities in responses. If `None`, no weighting is applied to the verbose part, and the loss is equivalent to the standard DPO loss. The paper recommends setting `ld_alpha` between `0.0` and `1.0`. discopop_tau (`float`, *optional*, defaults to `0.05`): τ/temperature parameter from the [DiscoPOP](https://huggingface.co/papers/2406.08414) paper, which controls the shape of log ratio modulated loss. The paper recommends the default value `discopop_tau=0.05`. loss_weights (`list[float]` or `None`, *optional*, defaults to `None`): List of loss weights for multi-loss combinations. Used when combining multiple loss types. Example: `[0.8, 0.2, 1.0]` for [MPO](https://huggingface.co/papers/2411.10442). If not provided, defaults to equal weights (`1.0`) for all loss types. sync_ref_model (`bool`, *optional*, defaults to `False`): Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper. ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`): α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`. ref_model_sync_steps (`int`, *optional*, defaults to `512`): τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`. > Parameters that control the logging generate_during_eval (`bool`, *optional*, defaults to `False`): Whether to generate and log completions from both the model and the reference model to W&B or Comet during evaluation. model_init_kwargsref_model_init_kwargsgư>helpz$The initial learning rate for AdamW.)defaultmetadata learning_rate zLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps. logging_stepsTzZIf True, use gradient checkpointing to save memory at the expense of slower backward pass.gradient_checkpointingNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zKeyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the `DPOTrainer` is provided as a string.zKeyword arguments for `AutoModelForCausalLM.from_pretrained`, used when the `ref_model` argument of the `DPOTrainer` is provided as a string.zNName of the train target PEFT adapter, when using LoRA with multiple adapters.model_adapter_namezKName of the reference PEFT adapter, when using LoRA with multiple adapters.ref_adapter_nameFzIf you provide a PEFT model as the active model and wish to use a different model for the `ref_model`, set this flag to `True`.force_use_ref_modelzDPOConfig.Ls raType of loss to use. Possible values are: `'sigmoid'`, `'hinge'`, `'ipo'`, `'exo_pair'`, `'nca_pair'`, `'robust'`, `'bco_pair'`, `'sppo_hard'`, `'aot'`, `'aot_pair'`, `'discopop'`, `'apo_zero'`, `'apo_down'` and `'sft'`. Multiple loss types can be combined using comma separation (e.g., `['sigmoid', 'bco_pair', 'sft']` for MPO). The `loss_weights` parameter can be used to specify corresponding weights for each loss type.)default_factoryr* loss_typezWhether to use Liger loss.use_liger_lossmodelzName of the attribute in the model that contains the base model. This is used to get the base model from the model when the model does not have a `get_decoder` method in the case when `use_liger_loss` is `True`.base_model_attribute_nameg?uvParameter controlling the deviation from the reference model. Higher β means less deviation from the reference model.betazfType of f-divergence regularization function to compute divergence between policy and reference model.f_divergence_typer uOα coefficient in the α-divergence u^-α regularization function for DPO loss.f_alpha_divergence_coefzWhether to ignore the provided reference model and implicitly use a reference model that assigns equal probability to all responses.reference_freegzvRobust DPO label smoothing parameter from the cDPO report and Robust DPO paper that should be between `0.0` and `0.5`.label_smoothingz4Whether to weight the loss as done in the WPO paper. use_weightinguα parameter from the RPO paper (v3), which controls the weighting of the NLL term in the loss. If `None`, no weighting is applied and the loss is the same as the DPO loss. The paper recommends `rpo_alpha=1.0`. rpo_alphau#α parameter from the LD-DPO paper, which controls the weighting of the verbose token log-probabilities in responses. If `None`, no weighting is applied to the verbose part, and the loss is equivalent to the standard DPO loss. The paper recommends setting `ld_alpha` between `0.0` and `1.0`.ld_alphag?uτ/temperature parameter from the DiscoPOP paper, which controls the shape of log ratio modulated loss. The paper recommends the default value `discopop_tau=0.05`. discopop_tauzList of loss weights for multi-loss combinations. Used when combining multiple loss types. Example: `[0.8, 0.2, 1.0]` for MPO. If not provided, defaults to equal weights (`1.0`) for all loss types. loss_weightszWhether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter.sync_ref_modelg333333?u-α parameter from the TR-DPO paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.ref_model_mixup_alphauτ parameter from the TR-DPO paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.ref_model_sync_stepsz~Whether to generate and log completions from both the model and the reference model to W&B, MLFLow or Comet during evaluation.generate_during_evalc>URcUR(+O URUl[URS5(a-[ UR5S:XaURSUlUR bv[ UR[5(a URO UR/n[ UR 5[ U5:wa[SUR SUS35e[TU])5 g)N__len__rzLength of loss_weights list (z#) must match number of loss types (z).) r/fp16hasattrrHlenrU isinstancelist ValueErrorsuper __post_init__)self loss_types __class__s rrdDPOConfig.__post_init__s'+yy'8Odii  4>>9 - -#dnn2E2J!^^A.DN    (+5dnnd+K+KRVR`R`QaJ4$$%Z8 3D4E4E3FG"|2' r)r/rH)?rrrr__doc__r _VALID_DICT_FIELDSrr+float__annotations__r-r.boolr/rr&dictstrrr'r0r1r2r3r4r5intr6r7r8r9r:r>r?r@rArBrar rrHrIrKrLr rrMrNrOrPrQrRrSrTrUrVrWrXrYrdr __classcell__)rgs@rr$r$!sZx+==ATVm@nn!@AM5! D M5$) p $D ! ! D(4.38 8 3xS#X/7< ; 78DcN3).jk) ',gh'hsm!& 4 !"XYOT % `  ',RS'hsm$)hi$M8C=$<=(-9:(x},1=>,8C=!&VW!J !""L1 OS 4 L$&+ ( &d05 j 0x}49 O 4E8DtX~./ 0!+ 8  ItCy !67ND&+ * &s G D%*/**  *&+kl&U! 2 ND# * OU PQM4"'  "Ix!& t !Hhuo  P L%+0  +L(4;'! B ND$) y $5!& q !#"' * "$  rr$N) dataclassesrrenumrtypingrrrr transformersr r rr$rrrrvsK)11**d* ((  q !q  q r