h"PSSKJrJr SSKJrJr SSKJr \"SS\55rg)) dataclassfield)OptionalUnion)TrainingArgumentsc^\rSrSr%Sr\R S/-r\"SSS0S9r\ \ S'\"S SS 0S9r \ \ S '\"S SS 0S9r \ \ S'\"SSS0S9r\\ \ S'\"SSS0S9r\\\\4\ S'\"SSS0S9r\ \ S'\"SSS0S9r\\ \ S'\"SSS0S9r\\\ S'\"SSS0S9r\\\ S'\"SSS0S9r\\\ S '\"S SS!0S9r\ \ S"'\"S SS#0S9r\\ \ S$'\"SSS%0S9r\\\ S&'\"SSS'0S9r\\\ S('\"S)SS*0S9r\ \ S+'\"S)SS,0S9r\ \ S-'\"SSS.0S9r \\\ S/'\"SSS00S9r!\\ \ S1'\"SSS20S9r"\\\ S3'\"S)SS40S9r#\ \ S5'\"SSS60S9r$\ \ S7'\"SSS80S9r%\\\ S9'\"SSS:0S9r&\ \ S;'\"S'\"S?SS@0S9r(\\ SA'\"SSSB0S9r)\ \ SC'\"SSSD0S9r*\\\ SE'\"SSSF0S9r+\\\ SG'\"SHSSI0S9r,\\ SJ'\"SKSSL0S9r-\\ SM'\"SNSSO0S9r.\ \ SP'\"SQSSR0S9r/\ \ SS'\"STSSU0S9r0\\ SV'\"SWSSX0S9r1\ \ SY'\"STSSZ0S9r2\\ S['\"S\SS]0S9r3\ \ S^'\"SSS_0S9r4\\ \ S`'\"SSSa0S9r5\\ \ Sb'\"ScSSd0S9r6\\ Se'\"SSSf0S9r7\\8\ \ Sg'\"ShSSi0S9r9\\ Sj'\"SkSSl0S9r:\\ Sm'\"SSSn0S9r;\ \ So'\"SSSp0S9r<\ \ Sq'\"SrSSs0S9r=\ \ St'\"SSSu0S9r>\\ Sv'\"S)SSw0S9r?\ \ Sx'\"SSSy0S9r@\ \ Sz'\"S SS{0S9rA\ \ S|'\"S}SS~0S9rB\ \ S'\"SSS0S9rC\ \ S'\"SSS0S9rD\\\ S'\"SSS0S9rE\\ \ S'U4SjrFSrGU=rH$) GRPOConfigu@ Configuration class for the [`GRPOTrainer`]. This class includes only the parameters that are specific to GRPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model and reference model model_init_kwargs (`str`, `dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model` argument of the [`GRPOTrainer`] is provided as a string. disable_dropout (`bool`, *optional*, defaults to `False`): Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input. > Parameters that control the data preprocessing remove_unused_columns (`bool`, *optional*, defaults to `False`): Whether to only keep the column `"prompt"` in the dataset. If you use a custom reward function that requires any column other than `"prompts"` and `"completions"`, you should keep this to `False`. max_prompt_length (`int` or `None`, *optional*, defaults to `512`): Maximum length of the prompt. If the prompt is longer than this value, it will be truncated left. num_generations (`int` or `None`, *optional*, defaults to `8`): Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value. max_completion_length (`int` or `None`, *optional*, defaults to `256`): Maximum length of the generated completion. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. shuffle_dataset (`bool`, *optional*, defaults to `True`): Whether to shuffle the training dataset. > Parameters that control generation generation_batch_size: (`int` or `None`, *optional*, defaults to `None`): Batch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`. steps_per_generation: (`int` or `None`, *optional*, defaults to `None`): Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive with `generation_batch_size`. temperature (`float`, defaults to `1.0`): Temperature for sampling. The higher the temperature, the more random the completions. top_p (`float`, *optional*, defaults to `1.0`): Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to `1.0` to consider all tokens. top_k (`int` or `None`, *optional*, defaults to `None`): Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered. min_p (`float` or `None`, *optional*, defaults to `None`): Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range. repetition_penalty (`float`, *optional*, defaults to `1.0`): Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat tokens. use_transformers_paged (`bool`, *optional*, defaults to `False`): Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers` paged implementation will be used for generation instead of the default padded implementation. This parameter is only effective when `use_vllm` is set to `False`. cache_implementation (`str` or `None`, *optional*, defaults to `None`): Implementation of the cache method for faster generation when `use_vllm` is set to `False`. generation_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them. > Parameters that control generation acceleration powered by vLLM use_vllm (`bool`, *optional*, defaults to `False`): Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed. vllm_mode (`str`, *optional*, defaults to `"server"`): Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `"server"` or `"colocate"`. - `"server"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). - `"colocate"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_model_impl (`str`, *optional*, defaults to `"vllm"`): Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model implementation. vllm_guided_decoding_regex (`str` or `None`, *optional*, defaults to `None`): Regex for vLLM guided decoding. If `None` (default), guided decoding is disabled. > Parameters that control the vLLM server (only used when `vllm_mode` is `"server"`) vllm_server_base_url (`str` or `None`, *optional*, defaults to `None`): Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `vllm_server_host` and `vllm_server_port` are ignored. vllm_server_host (`str`, *optional*, defaults to `"0.0.0.0"`): Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_port (`int`, *optional*, defaults to `8000`): Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_timeout (`float`, *optional*, defaults to `240.0`): Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised. > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `"colocate"`) vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`): Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag. vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`): Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag. vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`): Whether to enable sleep mode for vLLM. If `True`, vLLM will sleep during the optimization step and woken for weight sync and generation. > Parameters that control the training beta (`float`, *optional*, defaults to `0.0`): KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving training speed. num_iterations (`int`, *optional*, defaults to `1`): Number of iterations per batch (denoted as μ in the algorithm). epsilon (`float`, *optional*, defaults to `0.2`): Epsilon value for clipping. delta (`float` or `None`, *optional*, defaults to `None`): Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291). epsilon_high (`float` or `None`, *optional*, defaults to `None`): Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`. importance_sampling_level (`str`, *optional*, defaults to `"token"`): Controls whether importance sampling ratios are computed at the `"token"` or `"sequence"` level. `"token"` keeps the raw per-token log-probability ratios (one weight per token). `"sequence"` averages the log-probability ratios across valid tokens to produce a single ratio per sequence. The [GSPO paper](https://huggingface.co/papers/2507.18071) shows that sequence-level sampling often yields more stable training and better alignment with sequence-level rewards. reward_weights (`list[float]` or `None`, *optional*, defaults to `None`): Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`. scale_rewards (`str` or `bool`, *optional*, defaults to `"group"`): Specifies the scaling strategy for rewards. Supported values are: - `True` or `"group"` (default): rewards are scaled by the standard deviation within each group, ensuring unit variance within a group. - `"batch"`: rewards are scaled by the standard deviation across the entire batch, as recommended in the [PPO Lite paper](https://huggingface.co/papers/2508.08221). - `False` or `"none"`: no scaling is applied. The [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) recommends not scaling rewards, as scaling by the standard deviation introduces a question-level difficulty bias. loss_type (`str`, *optional*, defaults to `"dapo"`): Specifies the loss formulation to use. Supported values are: - `"grpo"`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages. - `"dr_grpo"`: Aggregates token-level losses by normalizing with a global constant. This method was introduced in the [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) to eliminate length bias. The value of the constant corresponds to `max_completion_length`. - `"dapo"` (default): Aggregates token-level losses by normalizing with the number of active token in the global accumulated batch. This method was introduced in the [DAPO paper](https://huggingface.co/papers/2503.14476) to eliminate length bias. - `"bnpo"`: Aggregates token-level losses by normalizing with the number of active token in the local batch. Note that normalization is performed over the local batch only, so results may slightly vary depending on the local batch size, despite a constant effective batch size. When using `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss. mask_truncated_completions (`bool`, *optional*, defaults to `False`): When enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability. sync_ref_model (`bool`, *optional*, defaults to `False`): Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper. ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`): α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`. ref_model_sync_steps (`int`, *optional*, defaults to `512`): τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`. top_entropy_quantile (`float`, *optional*, defaults to `1.0`): ρ parameter from [Beyond the 80/20 Rule](https://huggingface.co/papers/2506.01939). Keeps in the policy loss term only the top-ρ quantile of tokens by entropy of the probability distribution at each sequence position, improving results. Range: `[0.0-1.0]`. A value of `0.0` masks all but the highest entropy token; `1.0` keeps all tokens. The paper recommends a value of `0.2`. If used with `mask_truncated_completions=True`, only tokens from non-truncated completions are considered. use_liger_loss (`bool`, *optional*, defaults to `False`): Whether to use the Liger GRPO loss. vllm_importance_sampling_correction (`bool`, *optional*, defaults to `True`): Whether to apply Truncated Importance Sampling (TIS) between vLLM completion logprobs and recomputed logprobs. [Your Efficient RL Framework Secretly Brings You Off-Policy RL Training](https://fengyao.notion.site/off-policy-rl) highlights that using a separate generation framework (such as vLLM) can introduce off-policy effects due to subtle implementation differences between generation and training backends. TIS is proposed as a remedy for this issue. vllm_importance_sampling_cap (`float`, *optional*, defaults to `2.0`): Truncation parameter C for Truncated Importance Sampling (TIS). This sets an upper bound on the importance sampling ratio, improving training stability. > Parameters that control the logging log_completions (`bool`, *optional*, defaults to `False`): Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`. num_completions_to_print (`int` or `None`, *optional*, defaults to `None`): Number of completions to print with `rich`. If `None`, all completions are logged. wandb_log_unique_prompts (`bool`, *optional*, defaults to `False`): Whether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged. model_init_kwargsgư>helpz$The initial learning rate for AdamW.)defaultmetadata learning_rate zLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps. logging_stepsTzZIf True, use gradient checkpointing to save memory at the expense of slower backward pass.gradient_checkpointingNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zKeyword arguments for `transformers.AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the `GRPOTrainer` is provided as a string.FzWhether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input.disable_dropoutzWhether to only keep the column 'prompt' in the dataset. If you use a custom reward function that requires any column other than 'prompts' and 'completions', you should keep this to `False`.remove_unused_columnsizaMaximum length of the prompt. If the prompt is longer than this value, it will be truncated left.max_prompt_lengthzNumber of generations to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value.num_generationsz+Maximum length of the generated completion.max_completion_lengthaSThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation.ds3_gather_for_generationz(Whether to shuffle the training dataset.shuffle_datasetzBatch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * steps_per_generation`.generation_batch_sizezXNumber of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`.steps_per_generationg?zVTemperature for sampling. The higher the temperature, the more random the completions. temperaturezFloat that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1.0 to consider all tokens.top_pzNumber of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered.top_kzMinimum token probability, which will be scaled by the probability of the most likely token. It must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.min_paAdditional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.generation_kwargszFloat that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model to repeat tokens.repetition_penaltyaWhether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers` paged implementation will be used for generation instead of the default padded implementation. This parameter is only effective when `use_vllm` is set to `False`.use_transformers_pagedzWImplementation of the cache method for faster generation when use_vllm is set to False.cache_implementationzWhether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed.use_vllmserveraMode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `'server'` or `'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_modevllmzModel implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for model implementation.vllm_model_implzWhether to enable sleep mode for vLLM. If `True`, vLLM will sleep during the optimization step and woken for weight sync and generation.vllm_enable_sleep_modezQRegex for vLLM guided decoding. If `None` (default), guided decoding is disabled.vllm_guided_decoding_regexzBase URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` and `vllm_server_port` are ignored.vllm_server_base_urlz0.0.0.0zSHost of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_hosti@zSPort of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_portgn@zTotal timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised.vllm_server_timeoutg333333?a Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.vllm_gpu_memory_utilizationaControl the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag.vllm_tensor_parallel_sizegzzKL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving training speed.betau@Number of iterations per batch (denoted as μ in the algorithm).num_iterationsg?zEpsilon value for clipping.epsilonuEnables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291).deltazUpper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper DAPO recommends `0.28`. epsilon_hightokenaControls whether importance sampling ratios are computed at the `'token'` or `'sequence'` level. `'token'` keeps the raw per-token log-probability ratios (one weight per token). `'sequence'` averages the log-probability ratios across valid tokens to produce a single ratio per sequence. The GSPO paper shows that sequence-level sampling often yields more stable training and better alignment with sequence-level rewards.importance_sampling_levelzWeights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`.reward_weightsgroupaSpecifies the scaling strategy for rewards. Supported values are: `True` or `group'` (default): rewards are scaled by the standard deviation within each group, ensuring unit variance within a group. `'batch'`: rewards are scaled by the standard deviation across the entire batch, as recommended in the PPO Lite paper. `False` or `'none'`: no scaling is applied. The Dr. GRPO paper recommends not scaling rewards, as scaling by the standard deviation introduces a question-level difficulty bias. scale_rewardsdapouSSpecifies the loss formulation to use. Supported values are 'grpo', 'dapo', 'bnpo', and 'dr_grpo'. 'grpo': Aggregates token-level losses by normalizing over sequence length. Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages. 'dapo' (default): Aggregates token-level losses by normalizing with the number of active token in the global accumulated batch. This method was introduced in the DAPO paper to eliminate length bias. 'dr_grpo': Aggregates token-level losses by normalizing with a global constant. This method was introduced in the Dr. GRPO paper to eliminate length bias. The value of the constant corresponds to `max_completion_length`. 'bnpo': Aggregates token-level losses by normalizing with the number of active token in the local batch. Note that normalization is performed over the local batch only, so results may slightly vary depending on the local batch size, despite a constant effective batch size. When using `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss. loss_typezWhen enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the DAPO paper, this is a good practice for training stability.mask_truncated_completionszWhether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter.sync_ref_modelg333333?u-α parameter from the TR-DPO paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.ref_model_mixup_alphauτ parameter from the TR-DPO paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.ref_model_sync_stepsuρ parameter from Beyond the 80/20 Rule. Keeps in the policy loss term only the top-ρ quantile of tokens by entropy of the probability distribution at each sequence position, improving results. Range: [0.0-1.0]. A value of `0.0` masks all but the highest entropy token; `1.0` keeps all tokens. The paper recommends a value of `0.2`. If used with `mask_truncated_completions=True`, only tokens from non-truncated completions are considered.top_entropy_quantilez#Whether to use the Liger GRPO loss.use_liger_lossaWhether to apply Truncated Importance Sampling (TIS) between vLLM completion logprobs and recomputed logprobs. Your Efficient RL Framework Secretly Brings You Off-Policy RL Training highlights that using a separate generation framework (such as vLLM) can introduce off-policy effects due to subtle implementation differences between generation and training backends. TIS is proposed as a remedy for this issue.#vllm_importance_sampling_correctiong@zTruncation parameter C for Truncated Importance Sampling (TIS). This sets an upper bound on the importance sampling ratio, improving training stability.vllm_importance_sampling_capzWhether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.log_completionszRNumber of completions to print with `rich`. If `None`, all completions are logged.num_completions_to_printzvWhether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged.wandb_log_unique_promptsc &>URcUR(+O URUl[TU] 5 SSS.R UR UR 5UlUR nURc@URc3URUlURU-UR-UlOURbxURckURURU--S:wa)[SURSURU-S35eURURU--UlOGURc/URb"URU-UR-UlO [S5eUR(aYURS :waIURU-UR-S:wa)[S URS US URS35eURUR-S:wa&[SURS URS35eURS :a[SURS35eUR bUR"(a [S5egg)Nr=none)TFrzgeneration_batch_size (z.) must be divisible by the global batch size (z).z^'generation_batch_size' and 'steps_per_generation' can not be both configured at the same timenozThe global eval batch size (z * z() must be divisible by num_generations (zZGRPO requires at least 2 generations per prompt to calculate the advantages. You provided z*, which is less than the minimum required.z4Liger loss does not support two-sided GRPO loss yet.)rfp16super __post_init__getr> world_sizerrgradient_accumulation_stepsper_device_train_batch_size ValueErrordo_eval eval_strategyper_device_eval_batch_sizerr8rF)self num_processes __class__s Q/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/grpo_config.pyrRGRPOConfig.__post_init__s'+yy'8Odii  $+F;??@R@RTXTfTfg  % % -$2K2K2S(,(H(HD %)-)I)IM)Y\`\u\u)uD &  ' ' 38Q8Q8Y))T-M-MP]-]^bcc -d.H.H-IJ88=HIM)-(B(B00=@)D % ' ' /D4M4M4Y)-)I)IM)Y\`\u\u)uD &p  <r)I__name__ __module__ __qualname____firstlineno____doc__r_VALID_DICT_FIELDSrrfloat__annotations__rrboolrrr rdictstrrrrintrrrrrrrr r!r"r#r$r%r&r'r)r+r,r-r.r/r0r1r2r4r5r6r7r8r9r;r<listr>r@rArBrCrDrErFrGrHrIrJrKrR__static_attributes__ __classcell__)r]s@r^r r s\|+==AT@UU!@AM5! D M5$) p $D ! ! D(4.5: E 5xdCi 01" [ OT-2 p -8D>(- w (x} &+ U &OXc],1GH,8C=', 6 't',DE'OXd^ ,1 T ,8C=+0tu+(3-rsK 1 E5! I E8C=# ] E8E?). t )x~!&   !$) b $D+0st+(3-  c Hd (  Is ! $ OS$) 8 $D16mn1  +0 2 +(3-"opc"opc"' @ "*/ V *&+ T &s ( D% \]NC78GU# o E8E?%* Y %L(5/&+ &  &s -2 > -NHT%[) ]  M3   Y Is&(- 6 (! B ND$) y $5!& q !##( 8  #% !?@ND16 3  1' +0 G + %" e OT/4no/hsm05 & 0htn4U4Ur`r N) dataclassesrrtypingrr transformersrr r`r^rts2)"* ] U"] U ] Ur`