h*XSSKrSSKJrJr SSKJrJr SSKJr \"SS\55r g)N) dataclassfield)OptionalUnion)TrainingArgumentsc^\rSrSr%Sr\R S/-r\"SSS0S9r\ \ S'\"S SS 0S9r \ \ S '\"S SS 0S9r \ \ S'\"SSS0S9r\\ \ S'\"SSS0S9r\\\\4\ S'\"SSS0S9r\ \ S'\"SSS0S9r\\ \ S'\"SSS0S9r\\\ S'\"SSS0S9r\\\ S'\"SSS0S9r\\\ S '\"S SS!0S9r\ \ S"'\"S SS#0S9r\\ \ S$'\"SSS%0S9r\\\ S&'\"SSS'0S9r\\\ S('\"S)SS*0S9r\ \ S+'\"S)SS,0S9r\ \ S-'\"SSS.0S9r \\\ S/'\"SSS00S9r!\\ \ S1'\"SSS20S9r"\\\ S3'\"S)SS40S9r#\ \ S5'\"SSS60S9r$\ \ S7'\"SSS80S9r%\\\ S9'\"SSS:0S9r&\ \ S;'\"S'\"S?SS@0S9r(\\ SA'\"SSSB0S9r)\\\ SC'\"SSSD0S9r*\\\ SE'\"SFSSG0S9r+\\ SH'\"SISSJ0S9r,\\ SK'\"SLSSM0S9r-\ \ SN'\"SOSSP0S9r.\ \ SQ'\"SRSSS0S9r/\\ ST'\"SUSSV0S9r0\ \ SW'\"SRSSX0S9r1\\ SY'\"SZSS[0S9r2\ \ S\'\"SSS]0S9r3\\ \ S^'\"SSS_0S9r4\\5\ \ S`'\"SSSa0S9r6\ \ Sb'\"SSSc0S9r7\\8\ \ 4\ Sd'\"SSSe0S9r9\ \ Sf'\"SSSg0S9r:\ \ Sh'\"SiSSj0S9r;\ \ Sk'\"SSSl0S9r<\\ Sm'\"SSSn0S9r=\ \ So'\"SSSp0S9r>\\\ Sq'\"SSSr0S9r?\\ \ Ss'\"SSSt0S9r@\\\ Su'\"SSSv0S9rA\\ \ Sw'\"SSSx0S9rB\\ \ Sy'\"SSSz0S9rC\\\ S{'\"SSS|0S9rD\\ \ S}'\"SSS~0S9rE\\\ S'\"SSS0S9rF\\\ S'\"SSS0S9rG\\\ S'\"SSS0S9rH\\\ S'\"SSS0S9rI\\ \ S'\"SSS0S9rJ\\\ S'\"SSS0S9rK\\\ S'\"SSS0S9rL\\\ S'\"SSS0S9rM\\\ S'\"SSS0S9rN\\\ S'\"SSS0S9rO\\ \ S'U4SjrPSrQU=rR$) RLOOConfigu. Configuration class for the [`RLOOTrainer`]. This class includes only the parameters that are specific to RLOO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model and reference model model_init_kwargs (`str`, `dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model` argument of the [`GRPOTrainer`] is provided as a string. disable_dropout (`bool`, *optional*, defaults to `False`): Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input. > Parameters that control the data preprocessing remove_unused_columns (`bool`, *optional*, defaults to `False`): Whether to only keep the column `"prompt"` in the dataset. If you use a custom reward function that requires any column other than `"prompts"` and `"completions"`, you should keep this to `False`. max_prompt_length (`int` or `None`, *optional*, defaults to `512`): Maximum length of the prompt. If the prompt is longer than this value, it will be truncated left. num_generations (`int` or `None`, *optional*, defaults to `2`): Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value. max_completion_length (`int` or `None`, *optional*, defaults to `256`): Maximum length of the generated completion. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. shuffle_dataset (`bool`, *optional*, defaults to `True`): Whether to shuffle the training dataset. > Parameters that control generation generation_batch_size: (`int` or `None`, *optional*, defaults to `None`): Batch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`. steps_per_generation: (`int` or `None`, *optional*, defaults to `None`): Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive with `generation_batch_size`. temperature (`float`, defaults to `1.0`): Temperature for sampling. The higher the temperature, the more random the completions. top_p (`float`, *optional*, defaults to `1.0`): Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to `1.0` to consider all tokens. top_k (`int` or `None`, *optional*, defaults to `None`): Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered. min_p (`float` or `None`, *optional*, defaults to `None`): Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range. repetition_penalty (`float`, *optional*, defaults to `1.0`): Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat tokens. use_transformers_paged (`bool`, *optional*, defaults to `False`): Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers` paged implementation will be used for generation instead of the default padded implementation. This parameter is only effective when `use_vllm` is set to `False`. cache_implementation (`str` or `None`, *optional*, defaults to `None`): Implementation of the cache method for faster generation when `use_vllm` is set to `False`. generation_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them. > Parameters that control generation acceleration powered by vLLM use_vllm (`bool`, *optional*, defaults to `False`): Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed. vllm_mode (`str`, *optional*, defaults to `"server"`): Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `"server"` or `"colocate"`. - `"server"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). - `"colocate"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_model_impl (`str`, *optional*, defaults to `"vllm"`): Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model implementation. vllm_guided_decoding_regex (`str` or `None`, *optional*, defaults to `None`): Regex for vLLM guided decoding. If `None` (default), guided decoding is disabled. > Parameters that control the vLLM server (only used when `vllm_mode` is `"server"`) vllm_server_base_url (`str` or `None`, *optional*, defaults to `None`): Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `vllm_server_host` and `vllm_server_port` are ignored. vllm_server_host (`str`, *optional*, defaults to `"0.0.0.0"`): Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_port (`int`, *optional*, defaults to `8000`): Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_timeout (`float`, *optional*, defaults to `240.0`): Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised. > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `"colocate"`) vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`): Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag. vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`): Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag. > Parameters that control the training beta (`float`, *optional*, defaults to `0.05`): KL coefficient. If `0.0`, the reference model is not loaded, reducing memory usage and improving training speed. num_iterations (`int`, *optional*, defaults to `1`): Number of iterations per batch (denoted as μ in the algorithm). epsilon (`float`, *optional*, defaults to `0.2`): Epsilon value for clipping. epsilon_high (`float` or `None`, *optional*, defaults to `None`): Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`. reward_weights (`list[float]` or `None`, *optional*, defaults to `None`): Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`. normalize_advantages (`bool`, *optional*, defaults to `False`): Whether to normalize advantages. Normalization is done per generation batch to have mean `0.0` and standard deviation of `1.0`. reward_clip_range (`tuple[float, float]` or `None`, *optional*, defaults to `None`): Clip range for rewards as (min, max). If `None`, no clipping is applied. mask_truncated_completions (`bool`, *optional*, defaults to `False`): When enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability. sync_ref_model (`bool`, *optional*, defaults to `False`): Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper. ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`): α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`. ref_model_sync_steps (`int`, *optional*, defaults to `512`): τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`. > Parameters that control the logging log_completions (`bool`, *optional*, defaults to `False`): Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`. num_completions_to_print (`int` or `None`, *optional*, defaults to `None`): Number of completions to print with `rich`. If `None`, all completions are logged. wandb_log_unique_prompts (`bool`, *optional*, defaults to `False`): Whether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged. model_init_kwargsgư>helpz$The initial learning rate for AdamW.)defaultmetadata learning_rate zLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps. logging_stepsTzZIf True, use gradient checkpointing to save memory at the expense of slower backward pass.gradient_checkpointingNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zKeyword arguments for `transformers.AutoModelForCausalLM.from_pretrained`, used when the `model` argument of the `GRPOTrainer` is provided as a string.FzWhether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input.disable_dropoutzWhether to only keep the column 'prompt' in the dataset. If you use a custom reward function that requires any column other than 'prompts' and 'completions', you should keep this to `False`.remove_unused_columnsizaMaximum length of the prompt. If the prompt is longer than this value, it will be truncated left.max_prompt_lengthzNumber of generations to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value.num_generationsz+Maximum length of the generated completion.max_completion_lengthaSThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation.ds3_gather_for_generationz(Whether to shuffle the training dataset.shuffle_datasetzBatch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * steps_per_generation`.generation_batch_sizezXNumber of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`.steps_per_generationg?zVTemperature for sampling. The higher the temperature, the more random the completions. temperaturezFloat that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1.0 to consider all tokens.top_pzNumber of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered.top_kzMinimum token probability, which will be scaled by the probability of the most likely token. It must be a value between 0.0 and 1.0. Typical values are in the 0.01-0.2 range.min_paAdditional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them.generation_kwargszFloat that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1.0 encourage the model to use new tokens, while values < 1.0 encourage the model to repeat tokens.repetition_penaltyaWhether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers` paged implementation will be used for generation instead of the default padded implementation. This parameter is only effective when `use_vllm` is set to `False`.use_transformers_pagedzWImplementation of the cache method for faster generation when use_vllm is set to False.cache_implementationzWhether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed.use_vllmserveraMode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `'server'` or `'colocate'`. `'server'`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). `'colocate'`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_modevllmzModel implementation to use for vLLM. Must be one of `transformers` or `vllm`. `transformers`: Use the `transformers` backend for model implementation. `vllm`: Use the `vllm` library for model implementation.vllm_model_implzQRegex for vLLM guided decoding. If `None` (default), guided decoding is disabled.vllm_guided_decoding_regexzBase URL for the vLLM server (e.g., 'http://localhost:8000'). If provided, `vllm_server_host` and `vllm_server_port` are ignored.vllm_server_base_urlz0.0.0.0zSHost of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_hosti@zSPort of the vLLM server to connect to. Ignored if vllm_server_base_url is provided.vllm_server_portgn@zTotal timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised.vllm_server_timeoutg333333?a Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag.vllm_gpu_memory_utilizationaControl the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `'colocate'`. If you are using `vllm_mode='server'`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag.vllm_tensor_parallel_sizeg?zpKL coefficient. If `0.0`, the reference model is not loaded, reducing memory usage and improving training speed.betau@Number of iterations per batch (denoted as μ in the algorithm).num_iterationsg?zEpsilon value for clipping.epsilonzUpper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper DAPO recommends `0.28`. epsilon_highzWeights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`.reward_weightszWhether to normalize advantages. Normalization is done per generation batch to have mean `0.0` and standard deviation of `1.0`.normalize_advantageszFClip range for rewards as (min, max). If None, no clipping is applied.reward_clip_rangezWhen enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the DAPO paper, this is a good practice for training stability.mask_truncated_completionszWhether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter.sync_ref_modelg333333?u-α parameter from the TR-DPO paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`.ref_model_mixup_alphauτ parameter from the TR-DPO paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`.ref_model_sync_stepszWhether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`.log_completionszRNumber of completions to print with `rich`. If `None`, all completions are logged.num_completions_to_printzvWhether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged.wandb_log_unique_promptsz*Deprecated: use `num_generations` instead.rloo_kz"Deprecated: use `epsilon` instead. cliprangezDeprecated: use `beta` instead.kl_coefz#Deprecated: use `run_name` instead.exp_namez/Deprecated: use `normalize_advantages` instead.normalize_rewardz)Deprecated: use `num_iterations` instead.num_ppo_epochsz/Deprecated: use `steps_per_generation` instead.num_mini_batcheszXDeprecated: use `max_steps=total_episodes/(gradient_accumulation_steps*rloo_k)` instead.total_episodesz0Deprecated: use `max_completion_length` instead.response_lengthz7Removed: KL is now computed only at the sequence level.token_level_klzORemoved: this parameter was unused, you can safely remove it from your scripts.dataset_num_procztRemoved: now automatically set to `per_device_train_batch_size` (or `per_device_eval_batch_size` during evaluation). local_rollout_forward_batch_sizezERemoved: use `logging_steps` to control generation logging frequency.num_sample_generationszRemoved. stop_tokenz5Removed: use `processing_class.eos_token_id` instead. stop_token_idzcRemoved: replicate with a custom reward function checking if `eos_token_id` is in `completion_ids`.missing_eos_penaltyc >URcUR(+O URUlSSSSSSSSS S . n1S knUR5Hlup4[X5cM[X5nUS :XaXPR-n[ R "S USUSUSU35 [XU5 [XS5 Mn UH=n[X5(dM[X5cM#[ R "S US35 M? [TU])5 URnURc@URc3URUl URU-UR-Ul OURbxURckURURU--S:wa)[SURSURU-S35eURURU--Ul OGURc/URb"URU-UR-Ul O [S5eUR (aYUR"S:waIUR$U-UR&-S:wa)[SUR$SUSUR&S35eURUR&-S:wa&[SURSUR&S35eUR&S:a[SUR&S35eg)Nrr6r4run_namer9r5r max_stepsr) rBrCrDrErFrGrHrIrJ>rOrPrKrLrQrNrMrIz Parameter 'zC' is deprecated and will be removed in version 0.25.0. Please use 'z' instead. We are setting =z' is deprecated and will be removed in version 0.25.0. Please refer to the migration guide: https://huggingface.co/docs/trl/en/rloo_trainer##migration-guide-from-the-old-implementation-021-and-belowrzgeneration_batch_size (z.) must be divisible by the global batch size (z).z^'generation_batch_size' and 'steps_per_generation' can not be both configured at the same timenozThe global eval batch size (z * z() must be divisible by num_generations (rzZGRPO requires at least 2 generations per prompt to calculate the advantages. You provided z*, which is less than the minimum required.)rfp16itemsgetattrgradient_accumulation_stepswarningswarnsetattrhasattrsuper __post_init__ world_sizerrper_device_train_batch_size ValueErrordo_eval eval_strategyper_device_eval_batch_sizer) self_DEPRECATED_PARAMS_REMOVED_PARAMS old_param new_param old_value removed_param num_processes __class__s Q/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/rloo_config.pyr`RLOOConfig.__post_init__IsG'+yy'8Odii ("" 6. 6)6   %7$<$<$> It'3#D4  00 )-M-M MI !)-!{"r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQr`__static_attributes__ __classcell__)ros@rpr r s iV+==AT@UU!@AM5! D M5$) p $D ! ! D(4.5: E 5xdCi 01" [ OT-2 p -8D>(- w (x} &+ U &OXc],1GH,8C=', 6 't',DE'OXd^ ,1 T ,8C=+0tu+(3-rsK 1 E5! I E8C=# ] E8E?). t )x~!&   !$) b $D+0st+(3-  c Hd (  Is ! $ OS16mn1  +0 2 +(3-"opc"opc"' @ "*/ V *&+ T &s  D% \]NC78GU%* Y %L(5/-2 > -NHT%[)"' / "$8=bc8xeUl 34(- 6 (! B ND$) y $5!& q !#" e OT/4no/hsm05 & 0htn"FGFHSM"'>?"Ix %;< GXe_$?@Hhsm(-KL(htn%*EF%NHSM',KL'hsm%*tu%NHSM&+LM&OXc]&+ST&NHTN',kl'hsm7< " 7$hsm-2ab-HSM!&*%!J $)QR$M8C=,1   ,%[[rrr ) r[ dataclassesrrtypingrr transformersrr rrrprs2("* M "M  M rr