h/fSSKrSSKrSSKJrJr SSKJr SSKJr SSK J r \"SS55r g) N) dataclassfield)Optional)is_bitsandbytes_available) flatten_dictc\rSrSr%Sr\"\RR\ RS5S\ "S5*SS0S9r \ \S '\"S SS 0S9r\ \S '\"SSS 0S9r\\S'\"SSSS/S.S9r\\ \S'\"\SS0S9r\\S'\"\SS0S9r\\S'\"\SS0S9r\\S'\"SSS0S9r\ \S'\"SSS0S9r\ \S '\"S!SS"0S9r\\S#'\"S$SS%0S9r\\S&'\"S'SS(0S9r\\S)'\"S*SS+0S9r\ \S,'\"S-SS.0S9r\\S/'\"S SS00S9r \ \S1'\"S2SS30S9r!\\S4'\"S5SS60S9r"\#\S7'\"S8SS90S9r$\#\S:'\"S$SS;0S9r%\\S<'\"S=SS>0S9r&\\S?'\"S$SS@0S9r'\\SA'\"SBSSC0S9r(\\SD'\"SESSF0S9r)\#\SG'\"SHSSI0S9r*\#\SJ'\"SKSSL0S9r+\#\SM'\"SNSSO0S9r,\#\SP'\"SQSSR0S9r-\#\SS'\"S$SST0S9r.\\SU'\"S5SSV0S9r/\#\SW'\"S$SSX0S9r0\\SY'\"S-SSZ0S9r1\\S['\"S8SS\0S9r2\#\S]'\"SNSS^0S9r3\#\S_'\"S5SS`0S9r4\#\Sa'\"SBSSb0S9r5\\Sc'\"SdSSe0S9r6\\Sf'\"SdSSg0S9r7\\Sh'\"SBSSi0S9r8\\Sj'\"S=SSk0S9r9\\Sl'\"S SSm0S9r:\ \Sn'\"SBSSo0S9r;\\Sp'Sqrg)t DDPOConfiga` Configuration class for the [`DDPOTrainer`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: exp_name (`str`, *optional*, defaults to `os.path.basename(sys.argv[0])[: -len(".py")]`): Name of this experiment (by default is the file name without the extension name). run_name (`str`, *optional*, defaults to `""`): Name of this run. seed (`int`, *optional*, defaults to `0`): Random seed. log_with (`Literal["wandb", "tensorboard"]]` or `None`, *optional*, defaults to `None`): Log with either 'wandb' or 'tensorboard', check https://huggingface.co/docs/accelerate/usage_guides/tracking for more details. tracker_kwargs (`Dict`, *optional*, defaults to `{}`): Keyword arguments for the tracker (e.g. wandb_project). accelerator_kwargs (`Dict`, *optional*, defaults to `{}`): Keyword arguments for the accelerator. project_kwargs (`Dict`, *optional*, defaults to `{}`): Keyword arguments for the accelerator project config (e.g. `logging_dir`). tracker_project_name (`str`, *optional*, defaults to `"trl"`): Name of project to use for tracking. logdir (`str`, *optional*, defaults to `"logs"`): Top-level logging directory for checkpoint saving. num_epochs (`int`, *optional*, defaults to `100`): Number of epochs to train. save_freq (`int`, *optional*, defaults to `1`): Number of epochs between saving model checkpoints. num_checkpoint_limit (`int`, *optional*, defaults to `5`): Number of checkpoints to keep before overwriting old ones. mixed_precision (`str`, *optional*, defaults to `"fp16"`): Mixed precision training. allow_tf32 (`bool`, *optional*, defaults to `True`): Allow `tf32` on Ampere GPUs. resume_from (`str`, *optional*, defaults to `""`): Resume training from a checkpoint. sample_num_steps (`int`, *optional*, defaults to `50`): Number of sampler inference steps. sample_eta (`float`, *optional*, defaults to `1.0`): Eta parameter for the DDIM sampler. sample_guidance_scale (`float`, *optional*, defaults to `5.0`): Classifier-free guidance weight. sample_batch_size (`int`, *optional*, defaults to `1`): Batch size (per GPU) to use for sampling. sample_num_batches_per_epoch (`int`, *optional*, defaults to `2`): Number of batches to sample per epoch. train_batch_size (`int`, *optional*, defaults to `1`): Batch size (per GPU) to use for training. train_use_8bit_adam (`bool`, *optional*, defaults to `False`): Use 8bit Adam optimizer from bitsandbytes. train_learning_rate (`float`, *optional*, defaults to `3e-4`): Learning rate. train_adam_beta1 (`float`, *optional*, defaults to `0.9`): Adam beta1. train_adam_beta2 (`float`, *optional*, defaults to `0.999`): Adam beta2. train_adam_weight_decay (`float`, *optional*, defaults to `1e-4`): Adam weight decay. train_adam_epsilon (`float`, *optional*, defaults to `1e-8`): Adam epsilon. train_gradient_accumulation_steps (`int`, *optional*, defaults to `1`): Number of gradient accumulation steps. train_max_grad_norm (`float`, *optional*, defaults to `1.0`): Maximum gradient norm for gradient clipping. train_num_inner_epochs (`int`, *optional*, defaults to `1`): Number of inner epochs per outer epoch. train_cfg (`bool`, *optional*, defaults to `True`): Whether to use classifier-free guidance during training. train_adv_clip_max (`float`, *optional*, defaults to `5.0`): Clip advantages to the range. train_clip_range (`float`, *optional*, defaults to `1e-4`): PPO clip range. train_timestep_fraction (`float`, *optional*, defaults to `1.0`): Fraction of timesteps to train on. per_prompt_stat_tracking (`bool`, *optional*, defaults to `False`): Whether to track statistics for each prompt separately. per_prompt_stat_tracking_buffer_size (`int`, *optional*, defaults to `16`): Number of reward values to store in the buffer for each prompt. per_prompt_stat_tracking_min_count (`int`, *optional*, defaults to `16`): Minimum number of reward values to store in the buffer. async_reward_computation (`bool`, *optional*, defaults to `False`): Whether to compute rewards asynchronously. max_workers (`int`, *optional*, defaults to `2`): Maximum number of workers to use for async reward computation. negative_prompts (`str`, *optional*, defaults to `""`): Comma-separated list of prompts to use as negative examples. push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the final model checkpoint to the Hub. rNz.pyhelpzQName of this experiment (by default is the file name without the extension name).)defaultmetadataexp_namezName of this run.run_namez Random seed.seedz)Log with either 'wandb' or 'tensorboard'.wandb tensorboard)r choiceslog_withz7Keyword arguments for the tracker (e.g. wandb_project).)default_factoryrtracker_kwargsz&Keyword arguments for the accelerator.accelerator_kwargszJKeyword arguments for the accelerator project config (e.g. `logging_dir`).project_kwargstrlz$Name of project to use for tracking.tracker_project_namelogsz2Top-level logging directory for checkpoint saving.logdirdzNumber of epochs to train. num_epochsz2Number of epochs between saving model checkpoints. save_freqz:Number of checkpoints to keep before overwriting old ones.num_checkpoint_limitfp16zMixed precision training.mixed_precisionTzAllow `tf32` on Ampere GPUs. allow_tf32z"Resume training from a checkpoint. resume_from2z"Number of sampler inference steps.sample_num_stepsg?z#Eta parameter for the DDIM sampler. sample_etag@z Classifier-free guidance weight.sample_guidance_scalez)Batch size (per GPU) to use for sampling.sample_batch_sizerz&Number of batches to sample per epoch.sample_num_batches_per_epochz)Batch size (per GPU) to use for training.train_batch_sizeFz*Use 8bit Adam optimizer from bitsandbytes.train_use_8bit_adamga2U0*3?zLearning rate.train_learning_rateg?z Adam beta1.train_adam_beta1g+?z Adam beta2.train_adam_beta2g-C6?zAdam weight decay.train_adam_weight_decayg:0yE>z Adam epsilon.train_adam_epsilonz&Number of gradient accumulation steps.!train_gradient_accumulation_stepsz,Maximum gradient norm for gradient clipping.train_max_grad_normz'Number of inner epochs per outer epoch.train_num_inner_epochsz8Whether to use classifier-free guidance during training. train_cfgzClip advantages to the range.train_adv_clip_maxzPPO clip range.train_clip_rangez"Fraction of timesteps to train on.train_timestep_fractionz7Whether to track statistics for each prompt separately.per_prompt_stat_trackingz?Number of reward values to store in the buffer for each prompt.$per_prompt_stat_tracking_buffer_sizez7Minimum number of reward values to store in the buffer."per_prompt_stat_tracking_min_countz*Whether to compute rewards asynchronously.async_reward_computationz>Maximum number of workers to use for async reward computation. max_workersz?K">?c?@J$)<=$5#EFs).BC) #"EFc!&FG!"'*+"$-(e$-(e&+./&U!&/*!.3BC.%s"'HI"#(CD#CTUIt!&9:!$+,e&+>?&U&+ST&d16[\1(#/4ST/&&+FG&dZ[K"XYcRSK ) rOr ) rZr] dataclassesrrtypingr transformersrcorerr rTrOrLrks5 (2 QQ QrO