hXSSKrSSKJrJr SSKJrJr SSKJr \"SS\55r g)N) dataclassfield)LiteralOptional)OnPolicyConfigc\rSrSr%Sr\"\RR\ 5SSSS0S9r \ \ S'\"S SS 0S9r \ \ S '\"SSS 0S9r\\ \ S '\"SSS0S9r\\ \ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SSS0S9r\\ S'\"SSS0S9r\S\ S'\"SSS0S9r\\ S'\"S SS!0S9r\\ S"'\"SSS#0S9r\\ S$'\"S%SS&0S9r\\ S''\"S(SS)0S9r\\ S*'\"S+SS,0S9r\\ S-'S.rg)/ PPOConfigaL Configuration class for the [`PPOTrainer`]. This class includes only the parameters that are specific to PPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] and [`OnPolicyConfig`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: exp_name (`str`, *optional*, defaults to `os.path.basename(__file__)[:-3]`): Name of this experiment. reward_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`): Path to the reward model. model_adapter_name (`str` or `None`, *optional*, defaults to `None`): Name of the train target PEFT adapter, when using LoRA with multiple adapters. ref_adapter_name (`str` or `None`, *optional*, defaults to `None`): Name of the reference PEFT adapter, when using LoRA with multiple adapters. num_ppo_epochs (`int`, *optional*, defaults to `4`): Number of epochs to train. whiten_rewards (`bool`, *optional*, defaults to `False`): Whether to whiten the rewards. kl_coef (`float`, *optional*, defaults to `0.05`): KL coefficient. kl_estimator (`Literal["k1", "k3"]`, *optional*, defaults to `"k1"`): Which estimator for KL-Divergence to use from [Approximating KL Divergence](http://joschu.net/blog/kl-approx.html). Defaults to "k1", a straightforward, unbiased estimator. Can be set to "k3", an unbiased estimator with lower variance which "appears to be a strictly better estimator". Cannot be set to "k2", as it is used for logging purposes. cliprange (`float`, *optional*, defaults to `0.2`): Clip range. vf_coef (`float`, *optional*, defaults to `0.1`): Value function coefficient. cliprange_value (`float`, *optional*, defaults to `0.2`): Clip range for the value function. gamma (`float`, *optional*, defaults to `1.0`): Discount factor. lam (`float`, *optional*, defaults to `0.95`): Lambda value for GAE. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. NhelpzName of this experiment.)defaultmetadataexp_namezEleutherAI/pythia-160mzPath to the reward model.reward_model_pathzNName of the train target PEFT adapter, when using LoRA with multiple adapters.model_adapter_namezKName of the reference PEFT adapter, when using LoRA with multiple adapters.ref_adapter_namezNumber of epochs to train.num_ppo_epochsFzWhether to whiten the rewards.whiten_rewardsg?zKL coefficient.kl_coefk1aWWhich estimator for KL-Divergence to use from Approximating KL Divergence (http://joschu.net/blog/kl-approx.html). Defaults to 'k1', a straightforward, unbiased estimator. Can be set to 'k3', an unbiased estimator with lower variance which 'appears to be a strictly better estimator'. Cannot be set to 'k2', as it is used for logging purposes.)rk3 kl_estimatorg?z Clip range. cliprangeg?zValue function coefficient.vf_coefz"Clip range for the value function.cliprange_valueg?zDiscount factor.gammagffffff?zLambda value for GAE.lamTaThis setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation.ds3_gather_for_generation) __name__ __module__ __qualname____firstlineno____doc__rospathbasename__file__rstr__annotations__rrrrrintrboolrfloatrrrrrrrr __static_attributes__r!P/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/ppo_config.pyr r s-^  *3B/45Hc#(56s).jk) ',gh'hsm 67NC!:;ND+,GU). U )L'*%-(Iu78GU#>?OU,-E512C', a 'tr1r ) r' dataclassesrrtypingrr trainer.utilsrr r!r1r2r6s1 ($* pp pr1