f:i SrSSKJr SSKrSSKJr SSKJr SSKJrJ r J r J r J r J r JrJr SSKJrJrJrJrJrJrJrJrJrJrJrJrJ r JrJrJrJrJ r J!r!J"r"J#r#J$r$J%r%J&r&J r J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r.J/r/J0r0J1r1J2r2J3r3J4r4J5r5J6r6J7r7J8r8J9r9J:r:J;r;Jr>J?r?J@r@JArAJBrBJCrCJDrDJrJErEJFrFJGrGJHrHJIrIJJrJJKrKJLrLJMrMJNrNJOrOJPrPJQrQJRrRJSrSJTrTJUrUJVrVJWrWJrJXrXJYrYJZrZJ[r[JrJrJ r J)r)J*r*J+r+J1r1J2r2J7r7J9r9J@r@JArAJBrBJCrCJDrDJErEJFrFJGrGJKrKJNrNJPrPJrJXrXJYrYJ[r[J0r0JFrFJGrGJPrPJRrRJrJXrXJPrPJrJ r JOrOJPrPJTrTJUrUJVrVJrJZrZJ r JrJ%r%J?r?JFrFJPrPJrJrJrJFrFJPrPJrJrJPrPJrJrJ%r%J1r1JFrFJPrPJr SSKFrFSSK7 SSK\J]r]J^r^ SS K_J`r` SSKrSSKarbSS KcJErE SSKJr SS KXJdrdJerf SS KgJhrh SSKiriSS KjJkrk SrlSSSSSS.rm\R"SS\mS9S5roS\RS\pS\pS\R4SjrqS\RS\RS\pS\pS\R4 SjrrS\RS\pS\R4SjrsS rt"S!S"\RR5rwS0S#jrx\R"SS\mS9S$5ryS%rz\]"S&S'\55r{"S(S)\%5r|"S*S+\|5r}\~"\?S,5(a4SSK@r@"S-S.\@R5r\?GR"\"S/55 gg)1z; 2025.10.10 2025.10.9 4.56.2 0.23.0 __UNSLOTH_VERSIONING__ )TensorN) functional)AnyListOptionalTupleUnionDictSetCallable)r AutoConfig"AutoModelForSequenceClassification AutoProcessor AutoTokenizer DataLoaderDatasetFSDP GRPOConfig GRPOTrainerGenerationConfigIterableDatasetrPath PeftConfigPreTrainedModelPreTrainedTokenizerBaseProcessorMixin RepeatSampler RewardFuncSamplerSyncRefModelCallbackTrainerTrainerCallbackr VLLMClient_ForwardRedirectionapply_chat_templatebroadcast_object_listcopydatasets defaultdictdequedisable_dropout_in_modelentropy_from_logitsgather gather_objectgenerate_model_cardget_comet_experiment_urlidentityinspectis_conversationalis_datasets_availableis_flash_attn_2_availableis_liger_kernel_available is_peft_modelis_rich_availableis_vllm_availableis_wandb_availableloggerloggingmaybe_apply_chat_templatenanmaxnanminnanstdnn nullcontextospadpartialprepare_deepspeed prepare_fsdpprepare_multimodal_messagesprepare_peft_modelprint_prompt_completions_sampleprofiling_contextprofiling_decoratorre seed_workerselective_log_softmaxset_seedshuffle_sequence_dictsplit_pixel_values_by_gridsplit_tensor_dicttextwraptorch transformerstruncate_with_protected_tokensunsplit_pixel_values_by_gridunwrap_model_for_generationrrr r%r&r'r-r.r3r5r<r=r>r?r@rBrCrDrHrKrMrUrVrWrYr,rCrDrMrOrUrVrMrr rLrMrQrRrSrUrXrrr!r;rCrMrUrrArCrMrrArMrUrr!r-rCrMrU)*) dataclassfield)Version)rB)DataCollatorForSeq2SeqDataCollatorForLanguageModeling) ParallelMode) MethodTypecF^[R"T5U4Sj5nU$)Nc8>[US5(a5[URS5(aURR5 T"U/UQ70UD6n[US5(a5[URS5(aURR5 U$)Nmodel for_training for_inference)hasattrrdrerf)selfargskwargsoutputfs ?/home/james-whalen/unsloth_compiled_cache/UnslothGRPOTrainer.pywrapper*prepare_for_training_mode..wrapper0sx 4 ! !gdjj.&I&I JJ # # %4)$)&) 4 ! !gdjj/&J&J JJ $ $ & ) functoolswraps)rlrns` rmprepare_for_training_moders/s%__Q NrpTF)epilogue_fusion max_autotune shape_paddingz trace.enabledztriton.cudagraphs)dynamic fullgraphoptionscd[R"URSURS5SSS9n[R"URS5SSS9n/n[ X#5HupVUR [R 5n[R"USURS5S9RS5n[R"USS9nXx- n URU 5 M [R"U5nURURSURS45nU$)Nrchunksdimrindexr) rUchunkreshapeshapeziptofloat32r- unsqueezesqueeze logsumexpappendconcat) logitsrchunked_logits chunked_indexall_per_token_logps chunk_logits chunk_indexselected_logitslogsumexp_valuesper_token_logpss rmchunked_selective_log_softmaxrEs[[FLL4D!EPQYZ[N[[r!2QaHM%(%G! #u}}5 ,,|2{G\G\]_G`aiijlm ??9CCAFG88;;J,<.J rptensorpad_idclX:gn[R"USSSS9n[R"USU5nU$)zD Moves all padding tokens in each sequence of a batch to the right. rT)r descendingstable)rUargsortr-)rrmasksorted_indices packed_tensors rmleft_pack_paddingrs8  D]]4Q4MNLLN;M rpc ^^$URSS5nURSS5n URSS5n URSS5n URSS5n URS S 5n URS S 5nURS S 5nURSS 5nURSS5nURS5nUS:waX-nUS:waX- nUS:waU[R"UU- 5-nUR [R 5nU S :waX- n[R "USUS9RS5nU[R"USS9- n[R"5 US :waUcS5eUS:waX-nUS:waX- nUS:waU[R"UU- 5-nUR [R 5nU S :waX - n[R "USUS9RS5nU[R"USS9- nUbUS:waX.-nUS:waX/- nUS:waU[R"UU- 5-nUR [R 5nU S :waX-- n[R "USUS9RS5nU[R"USS9- nSSS5 US :wa#[R"WU- 5UU- - S - nO>US:Xa"URURS5S5nO[R"U5nUbUW- nOUUR5- nUS:XaUnOZUS:XaEUT-RS5TRS5R!S S9- nURS5nO[#SUS35e[R"U5n[R "USU - SU -5nU b([R "UU S9URS5-nOUURS5-nUURS5-n[R$"UU5*nUS :waUUU--nTR [R 5mTRS5m$US:XaBUT-RS5TRS5R!S S9- R'5n OuUS:Xa2UT-R5TR5R!S S9- n O=US:Xa)UT-R5URS5U -- n O[#SU35eUU$4Sjn!U!"U5un"n#U U"U#4$!,(df  GN=f) N loss_typegrpo epsilon_low皙? epsilon_highmax_completion_length delta temperature?logit_scale_multiplylogit_scale_dividelogit_softcappingimportance_sampling_leveltokenr{rrr.ref_logits should not be None when beta != 0.0sequencermin#Unknown importance sampling level: -. Possible values are 'token' and 'sequence'.maxbnpodr_grpoUnknown loss type: c2>[R"5 TR5nURSS:XaXR54sSSS5 $UT-R S5T- nUR5nX4sSSS5 $!,(df  g=fNrrUinference_modemeanrrxcompletion_lengthmean_kl_per_rewardmean_klrn_mask_per_rewards rmmasked_batch_mean,grpo_compute_loss..masked_batch_mean|  ! ! # 1 6 6 8 wwqzQ(&&(2$ # '($h^^A%69J%J",113(1$ # #4B)B BgetrrUtanhrrr-rrno_gradexp new_zerossize zeros_likedetachrclamprrr% ref_logits new_logits old_logitsrrbeta advantagesrjrrrrrrrrrrnew_xnewref_xrefold_xoldkl_i log_ratiolog_importance_weightscoef_1coef_2loss_1loss_2loss_ilossrrrrs% ` @rmgrpo_compute_lossrsR ;/I**]C0K::nc2L"JJ'>E JJw %E**]C0K!::&P>PUX>P>Y!Y!7!A!A"!E12K1LM   ii./F [[[!l2B CF V/*2F2Fq2II*..q11 j**1- -Fii' 'F s{$+% 775== !D F$##B'$((2,*<*<*<*EEKKM f  ""$txxz'7'7C'7'@@ i  ""$ A9N(NO.yk:;; 2"34!8w "G ++W  9E U(( U7c:\rSrSr\SSj5r\S5rSrg)UnslothEfficientGRPOiNc ^^^ ^^^ ^!T c0m UU U4Sjm!URn [R"U5n [R"SU S9m[R"SU S9m[R"SU S9m UUU U!4Sjn[R"USS[ S9n[R "XSS9n[R "XSS9nUb[R "X*SS9nOS/U -nUb[R "X:SS9nOS/U -n[R "XZSS9n[R "XjSS9n[R "XzSS9nU bU R5OS n[UUUUUUU5HunnnnnnnU"UUUUUUUU5 M U RU 5 TRU 5 TRU 5 T RU 5 URU 5 TTT 4$) Nc >[R"URTR5TR 55nUSS2SS2SS24n[R "5 TS:waM[R"URTR5TR 55nUSS2SS2SS24nOSnUbM[R"URTR5TR 55n U SS2SS2SS24n OSn SSS5 [ WUW UUTU40TD6upn X-n XR5X44$!,(df  N:=f)Nr{r)rUmatmulrdtypetrrr)new_hidden_statesold_hidden_statesref_hidden_statesrrrscalingrrrrrr scaled_lossr extra_kwargslm_heads rm compute_loss2UnslothEfficientGRPO.forward..compute_loss%s5&7&:&:7==&I799;WJ#AssAI.J3;!&.?.B.B7==.QSZS\S\S^!_J!+AssAI!6J!%J$0!&.?.B.B7==.QSZS\S\S^!_J!+AssAI!6J!%J!"0A 0 0 ,DW.K0A LL L?!s #B(D?? E rrc >[RRTSSS9"XX#XEU5uunun upn TRU 5 T RU 5 TRU 5 XSS&g)N)rT)argnumshas_aux)rUfuncgrad_and_valueadd_)new_hidden_states_jold_hidden_states_jref_hidden_states_j input_ids_jmask_j advantages_jr grad_inputs_jchunk_grad_input chunk_loss unscaled_losschunk_completion_length chunk_mean_klaccumulated_completion_lengthaccumulated_lossaccumulated_mean_klrs rmaccumulate_chunk6UnslothEfficientGRPO.forward..accumulate_chunkPskpjtjtkDkDkDk"8KZ`pw ky g  !g*.f}Wd  . .} = ) . ./F G  . .} =/! rpT)rxrwryrr}r) rrU empty_likezeroscompiletorch_compile_optionsr get_scalerdiv_save_for_backward)"ctx_new_hidden_states_old_hidden_states_ref_hidden_statesr _input_ids_mask _advantagesrscalern_chunksrr grad_inputsr.grad_inputs_chunksrrrrrrrr%rr r!r"r#r$r+r,r-rs" ` ` ` @@@@rmforwardUnslothEfficientGRPO.forward!s   L" MF "))&&'9: (- A(G(- A(G%(- A(G 0 0&  == +  #[[VWX"[[);VWX  )!&-?Z[!\ !% 1   )!&-?Z[!\ !% 1 "[[VWX "[[VWX"[[VWX )/(:&""$ "$57HJ[]fhlnx y @]/1DFY\gioq} ###  z* **84**84%**84**84 k*  )   rpc 8URunUSSSSSSSSSS4 $N) saved_tensors)r7 grad_outputdcompletion_lengthdmean_kl grad_inputs rmbackwardUnslothEfficientGRPO.backwards-)) D$dD$dDRVWWrp)NrN)__name__ __module__ __qualname____firstlineno__ staticmethodrBrK__static_attributes__rMrprmr r s5@ @ B XX rpr c NURupU RSS5n U RSS5n U RSS5nU RSS5n[SU S-5Vs/sHnU U-S:XdMUPM nnUS:XaU nU[[R "UU5[ U5S- 5n[US5(dt[RRS S 5S :Xa[RO[RUl [RRS S 5S :XaSUl S [RS'URR5R nU c[#XUR$R&5n[)U5R+5n[-XR$R&5nUSS2UU-*S24n[/UUUUR$R&5R1UR25nXR$R&:gnUR1UR25nO USS2U*S24nUR4R7URSS9nUR95HDn[US5(dM[UR:S5(dM3SUR:lMF URc [?5nOF[R@RCURRDRFURS9nU U c[U"UUU U UUS9RHnUSS2UW-S-*S2SS24nUbUSS2UU-S-*S2SS24nUbUSS2UU-S-*S2SS24nOU"UUU U UUUS-S9RHnSSS5 [JRMWUUUUUUURNUR4RPUU 5 unnnS [RS'UUU4$s snf!,(df  Nl=f)N pixel_valuesimage_grid_thwpixel_attention_mask image_sizesrrr{_autocast_dtypeACCELERATE_MIXED_PRECISIONfp16UNSLOTH_FORCE_FLOAT3201UNSLOTH_RETURN_HIDDEN_STATESFkeep_fp32_wrapper_hf_hookio_same_decice device_typerrattention_maskrUrVrWrXrrgrUrVrWrXr),rrrangernp searchsortedlenrgrCenvironrUfloat16bfloat16rYrdget_output_embeddingsweightrprocessing_classrritemrrrr accelerator unwrap_modelmodulesrbrcrBampautocastrtyperr applyrr>r rr)trainerrrgrcompletion_maskrrrr?rjbszqlenrUrVrWrXifactorsrrrrunwrapped_modelmodule autocasterrrrrrrs rmgrpo_accumulated_lossrsFIC::nT2LZZ 06N!::&>B^`f3gkq3q%--w|xFxF ::>>13 73 >Z^@W14BJJ-.mm113::G%CI_f_w_w`E`E&F"56;;= %i1I1I1V1VW (nl.J,K,L)LM:;OQkmy|C|T|T|a|abeeftfzfz{#'?'?'L'LL'**>+?+?@(^O,<)<=))66w}}Z_6`O"))+ 6: & &76??DT+U+U-2FOO *, & ] YY''gmm6J6J6O6OY`YpYp'q    /%!/+!/';) !f !2!~|7STU7U5V5WZ[2[ \  ,$5a><;WXY;Y9Z9[^_6_$`! ,$5a><;WXY;Y9Z9[^_6_$`! /%!/+!/';)!/!!3!f ' :(<'A'A "" ($D W25BJJ-. "G ++m=P s( P9P8A8P P$c ^^$URSS5nURSS5n URSS5n URSS5n URSS5n URS S 5n URS S 5nURS S 5nURSS 5nURSS5nURS5nUS:waX-nUS:waX- nUS:waU[R"UU- 5-nUR [R 5nU S :waX- n[R "USUS9RS5nU[R"USS9- n[R"5 US :waUcS5eUS:waX-nUS:waX- nUS:waU[R"UU- 5-nUR [R 5nU S :waX - n[R "USUS9RS5nU[R"USS9- nUbUS:waX.-nUS:waX/- nUS:waU[R"UU- 5-nUR [R 5nU S :waX-- n[R "USUS9RS5nU[R"USS9- nSSS5 US :wa#[R"WU- 5UU- - S - nO>US:Xa"URURS5S5nO[R"U5nUbUW- nOUUR5- nUS:XaUnOZUS:XaEUT-RS5TRS5R!S S9- nURS5nO[#SUS35e[R"U5n[R "USU - SU -5nU b([R "UU S9URS5-nOUURS5-nUURS5-n[R$"UU5*nUS :waUUU--nTR [R 5mTRS5m$US:XaBUT-RS5TRS5R!S S9- R'5n OuUS:Xa2UT-R5TR5R!S S9- n O=US:Xa)UT-R5URS5U -- n O[#SU35eUU$4Sjn!U!"U5un"n#U U"U#4$!,(df  GN=f) Nrrrrrrrrrrrrrrrrr{rrrrrrrrrrrrrc2>[R"5 TR5nURSS:XaXR54sSSS5 $UT-R S5T- nUR5nX4sSSS5 $!,(df  g=frrrs rmr1grpo_compute_loss_slow..masked_batch_meanrrrrs% ` @rmgrpo_compute_loss_slowr&sR ;/I**]C0K::nc2L"JJ'>E JJw %E**]C0K!::&P>PUX>P>Y!Y!7!A!A"!E12K1LM   ii./F [[[!l2B CF V/*2F2Fq2II*..q11 j**1- -Fii' 'F s{$+% 775== !D F$##B'$((2,*<*<*<*EEKKM f  ""$txxz'7'7C'7'@@ i  ""$ A9N(NO.yk:;; 2"34!8w "G ++W rc .SSKJn U"S0UD6nXlU$)Nr)SamplingParamsrM)vllmr _set_kwargs)rjrsampling_paramss rmvLLMSamplingParamsrs#$.v.O"( rpc^\rSrSr%Sr\"SSS0S9r\\\ S'\"SSS 0S9r \\ \ S 'SSS S S S S S S SSSSSSSSSSSSSSSSSSSSSSS SS SSSSS S S S S S S S S S S S S!S"S S SSSSS S#S SSSSSSS SS SSS S#SSSSSSS$S%SS S S&SSSSSS SS S SSS'SSS SSSS SS"SSSS#S S SS(S)S SSS S SSS S S SS SSS S*S+S,SSSSSSSS0SS SS S-S.S SSS/S0S1S2SS3SS4SSS5SS6S7S S S8S*SS S S9S SS SS4U4S:jjr S;r U=r$)<UnslothGRPOConfigiu&@ Configuration class for the [`GRPOTrainer`]. This class includes only the parameters that are specific to GRPO training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model and reference model model_init_kwargs (`str`, `dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model` argument of the [`GRPOTrainer`] is provided as a string. disable_dropout (`bool`, *optional*, defaults to `False`): Whether to disable dropout in the model. This is useful for training with a reference model, as it prevents the model from generating different logprobs for the same input. > Parameters that control the data preprocessing remove_unused_columns (`bool`, *optional*, defaults to `False`): Whether to only keep the column `"prompt"` in the dataset. If you use a custom reward function that requires any column other than `"prompts"` and `"completions"`, you should keep this to `False`. max_prompt_length (`int` or `None`, *optional*, defaults to `512`): Maximum length of the prompt. If the prompt is longer than this value, it will be truncated left. num_generations (`int` or `None`, *optional*, defaults to `8`): Number of generations per prompt to sample. The effective batch size (num_processes * per_device_batch_size * gradient_accumulation_steps) must be evenly divisible by this value. max_completion_length (`int` or `None`, *optional*, defaults to `256`): Maximum length of the generated completion. ds3_gather_for_generation (`bool`, *optional*, defaults to `True`): This setting applies to DeepSpeed ZeRO-3. If enabled, the policy model weights are gathered for generation, improving generation speed. However, disabling this option allows training models that exceed the VRAM capacity of a single GPU, albeit at the cost of slower generation. Disabling this option is not compatible with vLLM generation. shuffle_dataset (`bool`, *optional*, defaults to `True`): Whether to shuffle the training dataset. > Parameters that control generation generation_batch_size: (`int` or `None`, *optional*, defaults to `None`): Batch size to use for generation. If `None`, it defaults to the effective training batch size: `per_device_train_batch_size * num_processes * steps_per_generation`. In other words, there is one generation batch processed per optimization step. Mutually exclusive with `steps_per_generation`. steps_per_generation: (`int` or `None`, *optional*, defaults to `None`): Number of steps per generation. If `None`, it defaults to `gradient_accumulation_steps`. Mutually exclusive with `generation_batch_size`. temperature (`float`, defaults to `1.0`): Temperature for sampling. The higher the temperature, the more random the completions. top_p (`float`, *optional*, defaults to `1.0`): Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to `1.0` to consider all tokens. top_k (`int` or `None`, *optional*, defaults to `None`): Number of highest probability vocabulary tokens to keep for top-k-filtering. If `None`, top-k-filtering is disabled and all tokens are considered. min_p (`float` or `None`, *optional*, defaults to `None`): Minimum token probability, which will be scaled by the probability of the most likely token. It must be a value between `0.0` and `1.0`. Typical values are in the `0.01-0.2` range. repetition_penalty (`float`, *optional*, defaults to `1.0`): Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > `1.0` encourage the model to use new tokens, while values < `1.0` encourage the model to repeat tokens. use_transformers_paged (`bool`, *optional*, defaults to `False`): Whether to use the `transformers` paged implementation for generation. If set to `True`, the `transformers` paged implementation will be used for generation instead of the default padded implementation. This parameter is only effective when `use_vllm` is set to `False`. cache_implementation (`str` or `None`, *optional*, defaults to `None`): Implementation of the cache method for faster generation when `use_vllm` is set to `False`. generation_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Additional keyword arguments to pass to `GenerationConfig` (if using transformers) or `SamplingParams` (if using vLLM) when sampling completions. This can be used to further customize the generation behavior, such as setting `suppress_tokens`, `num_beams`, etc. If it contains keys that conflict with the other generation parameters (like `min_p`, `top_p`, etc.), they will override them. > Parameters that control generation acceleration powered by vLLM use_vllm (`bool`, *optional*, defaults to `False`): Whether to use vLLM for generating completions. If set to `True`, the trainer will use vLLM for generation instead of the default model.generate(). Requires `vllm` to be installed. vllm_mode (`str`, *optional*, defaults to `"server"`): Mode to use for vLLM integration when `use_vllm` is set to `True`. Must be one of `"server"` or `"colocate"`. - `"server"`: The trainer will send generation requests to a separate vLLM server. Make sure a TRL vLLM server is running (start with `trl vllm-serve`). - `"colocate"`: vLLM will run in the same process and share the training GPUs. This avoids the need for a separate server but may cause resource contention with training. vllm_model_impl (`str`, *optional*, defaults to `"vllm"`): Model implementation to use for vLLM. Must be one of `"transformers"` or `"vllm"`. `"transformers"`: Use the `transformers` backend for model implementation. `"vllm"`: Use the `vllm` library for model implementation. vllm_guided_decoding_regex (`str` or `None`, *optional*, defaults to `None`): Regex for vLLM guided decoding. If `None` (default), guided decoding is disabled. > Parameters that control the vLLM server (only used when `vllm_mode` is `"server"`) vllm_server_base_url (`str` or `None`, *optional*, defaults to `None`): Base URL for the vLLM server (e.g., `"http://localhost:8000"`). If provided, `vllm_server_host` and `vllm_server_port` are ignored. vllm_server_host (`str`, *optional*, defaults to `"0.0.0.0"`): Host of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_port (`int`, *optional*, defaults to `8000`): Port of the vLLM server to connect to. Ignored if `vllm_server_base_url` is provided. vllm_server_timeout (`float`, *optional*, defaults to `240.0`): Total timeout duration in seconds to wait for the vLLM server to be up. If the server is not up after the timeout, a `ConnectionError` is raised. > Parameters that control colocated vLLM execution (only used when `vllm_mode` is `"colocate"`) vllm_gpu_memory_utilization (`float`, *optional*, defaults to `0.3`): Control the GPU memory utilization for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_gpu_memory_utilization` flag. vllm_tensor_parallel_size (`int`, *optional*, defaults to `1`): Control the tensor parallel size for vLLM. This setting only applies when `vllm_mode` is set to `"colocate"`. If you are using `vllm_mode="server"`, this parameter must be passed separately when launching the vLLM server via the `--vllm_tensor_parallel_size` flag. vllm_enable_sleep_mode (`bool`, *optional*, defaults to `False`): Whether to enable sleep mode for vLLM. If `True`, vLLM will sleep during the optimization step and woken for weight sync and generation. > Parameters that control the training beta (`float`, *optional*, defaults to `0.0`): KL coefficient. If `0.0` (default), the reference model is not loaded, reducing memory usage and improving training speed. num_iterations (`int`, *optional*, defaults to `1`): Number of iterations per batch (denoted as μ in the algorithm). epsilon (`float`, *optional*, defaults to `0.2`): Epsilon value for clipping. delta (`float` or `None`, *optional*, defaults to `None`): Enables the upper clipping bound in two-sided GRPO loss when set to a float. If `None` (default), standard GRPO clipping is used. Recommended to be greater than `1 + ε` when enabled. This method is introduced in the [INTELLECT-2 tech report](https://huggingface.co/papers/2505.07291). epsilon_high (`float` or `None`, *optional*, defaults to `None`): Upper-bound epsilon value for clipping. If not specified, it defaults to the same value as the lower-bound specified in argument `epsilon`. Paper [DAPO](https://huggingface.co/papers/2503.14476) recommends `0.28`. importance_sampling_level (`str`, *optional*, defaults to `"token"`): Controls whether importance sampling ratios are computed at the `"token"` or `"sequence"` level. `"token"` keeps the raw per-token log-probability ratios (one weight per token). `"sequence"` averages the log-probability ratios across valid tokens to produce a single ratio per sequence. The [GSPO paper](https://huggingface.co/papers/2507.18071) shows that sequence-level sampling often yields more stable training and better alignment with sequence-level rewards. reward_weights (`list[float]` or `None`, *optional*, defaults to `None`): Weights for each reward function. Must match the number of reward functions. If `None`, all rewards are weighted equally with weight `1.0`. scale_rewards (`str` or `bool`, *optional*, defaults to `"group"`): Specifies the scaling strategy for rewards. Supported values are: - `True` or `"group"` (default): rewards are scaled by the standard deviation within each group, ensuring unit variance within a group. - `"batch"`: rewards are scaled by the standard deviation across the entire batch, as recommended in the [PPO Lite paper](https://huggingface.co/papers/2508.08221). - `False` or `"none"`: no scaling is applied. The [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) recommends not scaling rewards, as scaling by the standard deviation introduces a question-level difficulty bias. loss_type (`str`, *optional*, defaults to `"dapo"`): Specifies the loss formulation to use. Supported values are: - `"grpo"`: Aggregates token-level losses by normalizing over sequence length. Not recommended due to length bias—this approach tends to prefer shorter completions with positive advantages and longer ones with negative advantages. - `"dr_grpo"`: Aggregates token-level losses by normalizing with a global constant. This method was introduced in the [Dr. GRPO paper](https://huggingface.co/papers/2503.20783) to eliminate length bias. The value of the constant corresponds to `max_completion_length`. - `"dapo"` (default): Aggregates token-level losses by normalizing with the number of active token in the global accumulated batch. This method was introduced in the [DAPO paper](https://huggingface.co/papers/2503.14476) to eliminate length bias. - `"bnpo"`: Aggregates token-level losses by normalizing with the number of active token in the local batch. Note that normalization is performed over the local batch only, so results may slightly vary depending on the local batch size, despite a constant effective batch size. When using `per_device_train_batch_size==1`, the loss is equivalent to the GRPO loss. mask_truncated_completions (`bool`, *optional*, defaults to `False`): When enabled, truncated completions are excluded from the loss calculation, preventing them from being incorrectly penalized and introducing noise during training. According to the [DAPO](https://huggingface.co/papers/2503.14476) paper, this is a good practice for training stability. sync_ref_model (`bool`, *optional*, defaults to `False`): Whether to synchronize the reference model with the active model every `ref_model_sync_steps` steps, using the `ref_model_mixup_alpha` parameter. This synchronization originates from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper. ref_model_mixup_alpha (`float`, *optional*, defaults to `0.6`): α parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: `π_ref = α * π_θ + (1 - α) * π_ref_prev`. To use this parameter, you must set `sync_ref_model=True`. ref_model_sync_steps (`int`, *optional*, defaults to `512`): τ parameter from the [TR-DPO](https://huggingface.co/papers/2404.09656) paper, which determines how frequently the current policy is synchronized with the reference policy. To use this parameter, you must set `sync_ref_model=True`. top_entropy_quantile (`float`, *optional*, defaults to `1.0`): ρ parameter from [Beyond the 80/20 Rule](https://huggingface.co/papers/2506.01939). Keeps in the policy loss term only the top-ρ quantile of tokens by entropy of the probability distribution at each sequence position, improving results. Range: `[0.0-1.0]`. A value of `0.0` masks all but the highest entropy token; `1.0` keeps all tokens. The paper recommends a value of `0.2`. If used with `mask_truncated_completions=True`, only tokens from non-truncated completions are considered. use_liger_loss (`bool`, *optional*, defaults to `False`): Whether to use the Liger GRPO loss. vllm_importance_sampling_correction (`bool`, *optional*, defaults to `True`): Whether to apply Truncated Importance Sampling (TIS) between vLLM completion logprobs and recomputed logprobs. [Your Efficient RL Framework Secretly Brings You Off-Policy RL Training](https://fengyao.notion.site/off-policy-rl) highlights that using a separate generation framework (such as vLLM) can introduce off-policy effects due to subtle implementation differences between generation and training backends. TIS is proposed as a remedy for this issue. vllm_importance_sampling_cap (`float`, *optional*, defaults to `2.0`): Truncation parameter C for Truncated Importance Sampling (TIS). This sets an upper bound on the importance sampling ratio, improving training stability. > Parameters that control the logging log_completions (`bool`, *optional*, defaults to `False`): Whether to log a sample of (prompt, completion) pairs every `logging_steps` steps. If `rich` is installed, it prints the sample. If `wandb` logging is enabled, it logs it to `wandb`. num_completions_to_print (`int` or `None`, *optional*, defaults to `None`): Number of completions to print with `rich`. If `None`, all completions are logged. wandb_log_unique_prompts (`bool`, *optional*, defaults to `False`): Whether to log unique prompts in wandb. If `True`, only unique prompts are logged. If `False`, all prompts are logged. NhelpzvLLM SamplingParams)defaultmetadatavllm_sampling_paramsr{z8Chunk size to reduce memory usage. -1 is most efficient.unsloth_num_chunksFnor|rg-C6 ?g{Gz?g?g+?g:0yE>rg@linear皙?passivewarningTstepsriO O1autor adamw_8bitlength every_savelasticolocaterz0.0.0.0i@gn@g333333?gMbP?rrgrouprg333333?g@c >US:a[SUS35 US:a[SUS35 UcU#S:Xa U$S:XaSnS n#WR5S :XaS nOWR5S :XaS nWR5S :XaWS:XaS nOcWS :Xa [S 5 SnOOWR5S :Xa;WS :wa [S5 WS:wa [S5 WS:wa [S5 S nSnSnSnUW-U-U:wa([S[U5-S-[W5-5 UnWS::a [S5eWS:a [S5e[TU]"S0SU_SU_SU_SU_SU_S U_S!U_S"U_S#U _S$U _S%U _S&U _S'U _S(U_S)U_S*U_S+U_S,U_S-U_S.U_S/U_S0U_S1U_S2U_S3U_S4U_S5U_S6U_S7U_S8U_S9U_S:U _S;U!_SU$_S?U%_S@U&_SAU'_SBU(_SCU)_SDU*_SEU+_SFU,_SGU-_SHU._SIU/_SJU0_SKU1_SLU2_SMU3_SNU4_SOU5_SPU6_SQU7_SRU8_SSU9_STU:_SUU;_SVU<_SWU=_SXU>_SYU?_SZW@_S[WA_S\WB_S]WC_S^WD_S_WE_S`WF_SaWG_SbWH_ScWI_SdWJ_SeWK_SfWL_SgWM_ShWN_SiWO_SjWP_SkWQ_SlWR_SmWS_SnWT_SoWU_SpWV_SqWW_SrWX_SsWY_StWZ_SuW[_SvW\_SwW]_SxW^_SyW__SzW`_S{Wa_S|Wb_S}Wc_S~Wd_SWe_SWf_SWg_SWh_SWi_SWj_SWk_SWl_SWm_SWn_SWo_SWp_SWq_SWr_SWs_SWt_SWu_SWv_SWw_SWx_SWy_SWz_SW{_SW|_SW}_SW~_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_WD6 WUlWUlg)NgHz>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!rrunsloth_training_checkpointsrrdapoTzwUnsloth: The Dr GRPO paper recommends setting `scale_rewards` to False! Will override. Set it to `None` to force False.FzFUnsloth: The DAPO paper recommends `mask_truncated_completions = True`gQ?z8Unsloth: The DAPO paper recommends `epsilon_high = 0.28`rzMUnsloth: The DAPO paper recommends setting `beta = 0.0` to remove the KL termrz}Unsloth: We now expect `per_device_train_batch_size` to be a multiple of `num_generations`. We will change the batch size of z to the `num_generations` of rzUUnsloth: Please set a positive non-zero temperature since your results will be wrong. zgUnsloth: Please set a positive non-zero temperature less than 10, since sampling will be quite erratic. output_diroverwrite_output_dirdo_traindo_eval do_predict eval_strategyprediction_loss_onlyper_device_train_batch_sizeper_device_eval_batch_sizeper_gpu_train_batch_sizeper_gpu_eval_batch_sizegradient_accumulation_stepseval_accumulation_steps eval_delaytorch_empty_cache_steps learning_rate weight_decay adam_beta1 adam_beta2 adam_epsilon max_grad_normnum_train_epochs max_stepslr_scheduler_type warmup_ratio warmup_steps log_levellog_level_replicalog_on_each_node logging_dirlogging_strategylogging_first_step logging_stepslogging_nan_inf_filter save_strategy save_stepssave_total_limitsave_safetensorssave_on_each_nodesave_only_model'restore_callback_states_from_checkpointno_cudause_cpuuse_mps_deviceseed data_seed jit_mode_evaluse_ipexbf16r[fp16_opt_levelhalf_precision_backendbf16_full_evalfp16_full_evaltf32 local_rank ddp_backend tpu_num_corestpu_metrics_debugdebugdataloader_drop_last eval_stepsdataloader_num_workersdataloader_prefetch_factor past_indexrun_name disable_tqdmremove_unused_columns label_namesload_best_model_at_endmetric_for_best_modelgreater_is_betterignore_data_skipfsdpfsdp_min_num_params fsdp_config"fsdp_transformer_layer_cls_to_wrapaccelerator_configparallelism_config deepspeedlabel_smoothing_factoroptim optim_args adafactorgroup_by_lengthlength_column_name report_toddp_find_unused_parametersddp_bucket_cap_mbddp_broadcast_buffersdataloader_pin_memorydataloader_persistent_workersskip_memory_metricsuse_legacy_prediction_loop push_to_hubresume_from_checkpoint hub_model_id hub_strategy hub_tokenhub_private_repohub_always_push hub_revisiongradient_checkpointinggradient_checkpointing_kwargsinclude_inputs_for_metricseval_do_concat_batches fp16_backendpush_to_hub_model_idpush_to_hub_organizationpush_to_hub_token mp_parametersauto_find_batch_sizefull_determinism torchdynamo ray_scope ddp_timeout torch_compiletorch_compile_backendtorch_compile_modeinclude_tokens_per_secondinclude_num_input_tokens_seenneftune_noise_alphaoptim_target_modulesbatch_eval_metrics eval_on_startuse_liger_kernelliger_kernel_configeval_use_gather_objectaverage_tokens_across_devicesmodel_init_kwargsdisable_dropoutmax_prompt_lengthnum_generationsrds3_gather_for_generationshuffle_datasetgeneration_batch_sizesteps_per_generationrtop_ptop_kmin_pgeneration_kwargsrepetition_penaltyuse_transformers_pagedcache_implementationuse_vllm vllm_modevllm_model_implvllm_enable_sleep_modevllm_guided_decoding_regexvllm_server_base_urlvllm_server_hostvllm_server_portvllm_server_timeoutvllm_gpu_memory_utilizationvllm_tensor_parallel_sizernum_iterationsepsilonrrrreward_weights scale_rewardsrmask_truncated_completionssync_ref_modelref_model_mixup_alpharef_model_sync_stepstop_entropy_quantileuse_liger_loss#vllm_importance_sampling_correctionvllm_importance_sampling_caplog_completionsnum_completions_to_printwandb_log_unique_promptsrM)printlowerstr MathErrorsuper__init__rr)rhrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr[rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2rr3r4r5r6rr7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrrIrJrrrrKrLrrMrNrOrPrQrRrSrTrUrVrWrrrj __class__s rmr]UnslothGRPOConfig.__init__sk p 4 )I-YB(C"D 1 e&F}oVw%xy  -7":zS?P7J M ??  )!I __ & (I ??  )$ $ $&PQ % __ & ()T1^_t#PQs{ef)- &LDI '? :o MQl l SVYZuVvvyXX[^_n[oo p*9 ' ! st t B FG G q J#q J#7q J q J q J $ q J * q J$8q J+Fq J*Dq J(@q J'>q J+Fq J'>q J$q J'>q J *!q J"(#q J$$%q J&$'q J(()q J**+q J, 0-q J."/q J0!21q J2(3q J4(5q J6"7q J8!29q J: 0;q J<&=q J> 0?q J@"4Aq JB*Cq JD&S3\ \S4\ \S5\\\ \S44S6jjr5S7r6U=r7$)?_UnslothGRPOTraineri;a Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300). Example: ```python from datasets import load_dataset from trl import GRPOTrainer dataset = load_dataset("trl-lib/tldr", split="train") def reward_func(completions, **kwargs): # Dummy reward function that rewards completions with more unique letters. return [float(len(set(completion))) for completion in completions] trainer = GRPOTrainer( model="Qwen/Qwen2-0.5B-Instruct", reward_funcs=reward_func, train_dataset=dataset, ) trainer.train() ``` Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported. reward_funcs (`Union[RewardFunc, list[RewardFunc]]`): Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward functions with the prompts and completions and sum the rewards. Can be either: - A single reward function, such as: - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported. - A custom reward function: The function is provided with the prompts and the generated completions, plus any additional columns in the dataset. It should return a list of rewards. Custom reward functions can also return `None` when the reward is not applicable to those samples. This is useful for multi-task training where different reward functions apply to different types of samples. When a reward function returns `None` for a sample, that reward function is excluded from the reward calculation for that sample. For more details, see [Using a custom reward function](#using-a-custom-reward-function). The trainer's state is also passed to the reward function. The trainer's state is an instance of [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the reward function's signature. - A list of reward functions, where each item can independently be any of the above types. Mixing different types within the list (e.g., a string model ID and a custom reward function) is allowed. args ([`GRPOConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]): Dataset to use for training. It must include a column `"prompt"`. Any additional columns in the dataset is ignored. The format of the samples can be either: - [Standard](dataset_formats#standard): Each sample contains plain text. - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role and content). eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Union[Dataset, IterableDataset]]`): Dataset to use for evaluation. It must meet the same requirements as `train_dataset`. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`] or `None`, *optional*, defaults to `None`): Processing class used to process the data. The padding side must be set to "left". If `None`, the processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default. reward_processing_classes (`Union[PreTrainedTokenizerBase, list[PreTrainedTokenizerBase]]`, *optional*, defaults to `None`): Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either: - A single processing class: Used when `reward_funcs` contains only one reward function. - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`. If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is `None`, the tokenizer for the model is automatically loaded using [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes` are ignored. callbacks (list of [`~transformers.TrainerCallback`], *optional*, defaults to `None`): List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](https://huggingface.co/docs/transformers/main_classes/callback). If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`] method. optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`. peft_config ([`~peft.PeftConfig`], *optional*, defaults to `None`): PEFT configuration used to wrap the model. If `None`, the model is not wrapped. trlrNrd reward_funcsri train_dataset eval_datasetrrreward_processing_classes callbacks optimizers peft_configrc ">^[US5(a0[TS5(a[TSS5S:XaSTlSTlTcO[ U[ 5(aUOUR Rn U RS5Sn [U S35mTR=(d 0n [ U[ 5(aUn U RS 5n[ U[R5(d US :XdUcO9[ U[ 5(a[[U5nXS 'O[S US 35e[R "U 5n[["UR$S 5nUR "U 40U D6nO9UR Rn TRb[&R("S5 [US5(d8[*R,"UR.5R0R35OE[*R,"UR55R.5R0R35UlUc*[:R "UR R5n[ U[<5(a UR>nO#[ U[@5(aUnO [CS5eURDcURFUl"URDUl"URHUl$URJUl%[USS5Ul&[USS5Ul'[UR SS5Ul([UR SS5Ul)[ U[T5(dU/n/Ul+[YU5Hunn[ U[ 5(a[ZR "U4SS0U D6UU'[ UU[\R^5(aFURVRaUUR RRS5S5 MURVRaUURb5 M X l2TRfb[iTRf5[iU5:wa.[S[iTRf5S[iU5S35e[Rj"TRf[RlS9Ul3O1[Rn"[iU5[RlS9Ul3UcS/[iU5-nO[ U[T5(dU/n[iU5[iU5:wa$[SU5S[iU5S35e[Y[qXr55Hununn[ U[r5(dM Uc*[tR "UR R5nURHcURFUl"URHUR l$UUU'M Xpl;TRxUl<TRzUl=TR|Ul>TR~Ul?TRUl@TRUlATRUlBTRUlCTRUlDTRUlTRUlTRUlETRUlFTRUlGTRUlHTRUlITRUlJTRUlKTRUlLTRUlMTRUlNUR(aURS:a [S5eUR(aURS :Xd [S!5eTRUlP[ U[5(dO[ U[5(d:[ U[5(a0[S"UR555(a [S#5eTRUlUTRUlWTRb TRO TRUlXS UlYSUlZSURS$'[TUG]uUT[UUUUU S%S&9 TRUl_URS':XaSUl`Oc[U5(aSUl`OK[R "U 5n[["UR$S 5nUR "U 40U D6Ul`TR(a-[U5 URb[UR5 UR(a[5(d [S(5e[5Ulg[URURURUR~URS':gURURzS)9Uli[[T5[[T5S*.UlkS UllTRUlmTRUlnTRUlo[TRS+9[TRS+9[TRS+9[U4S,j5[TRS+9S-.Ulr[TRSS.9 UR(Ga[5(d [S/5eURS0:XaURR(aTRb TRnOS1TRS2TR3n[UTRS39Ul}URR[RGR5S49 GOURS:XGaURGRUR-S :Xd1[S5URS6URGRS735eURS:a[GRGRG[ URGRUR-5Vs/sH6n[UG[ UUR-US-UR-55PM8 sn5uUln[ URGR 5G[GRS8'[ URGR5G[GRS9'[ URGR5G[GRS:'G[GRRS;S<5G[GRS;'G[GRRS=S>5G[GRS='URxb'URzbURxURz-nOSnUGRUlUGRGR(aUGRGRSS?9 O[S@URSA35eTGRUlSUlURGR%5 OURzSURHUGR&URJUR~URURURURTGR(SB. nTR(aSCUSD'SEUSF'SGUSH'TGR*bUGR-TGR*5 G[/SL0UD6UlSUlUGR4GR7UGR85 URbUGR:(a'G[=URUR5Ul`OcUGR>(a'G[AURUR5Ul`O*URGRCURSSI9Ul`TGRD(a/UGRGG[IURURSJ95 [YURd5H~unn[ U[r5(dMUGR:(a&G[=UUR5URdU'MUURGRCUSSSK9URdU'M gs snf)MN vllm_enginer>FTr/r{z-GRPOrrzInvalid `dtype` passed to `GRPOConfig`. Expected either 'auto' or a string representing a `torch.dtype` (e.g., 'float32'), but got .rzYou passed `model_init_kwargs` to the `GRPOConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.get_base_modelzWThe `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin` image_tokenimage_token_idvision_start_token_idvision_end_token_id num_labelsrzNumber of reward weights (z)) must match number of reward functions ()rz)The number of reward processing classes (z-) must match the number of reward functions (z).rzOLiger Kernels don't currently support masking token positions based on entropy.rzwLiger Kernels currently only support token-level importance sampling. Please set`importance_sampling_level` to 'token'.c3B# UHn[U[5v M g7frE) isinstancer).0dss rm /_UnslothGRPOTrainer.__init__..Ns6wav[]z"o7V7Vavsz^Iterable datasets are not yet supported in GRPOTrainer. Please use a standard dataset instead.estimate_tokensz!non-None value to disable scaling) rdri data_collatorrirjrrrlrmcompute_loss_funcrzWLiger is required to use `liger_loss` as the GRPO loss. Run `pip install liger-kernel`.)rrrr use_ref_modelrr)trainevalmaxlenc*>[TRS9$)Nr)r*r5)risrm._UnslothGRPOTrainer.__init__..s58R8R+Srp)imageprompt completionrewardsr)device_specificzkvLLM is not available and `use_vllm` is set to True. Please install vLLM with `pip install vllm` to use it.serverzhttp://:)base_urlconnection_timeoutrzvllm_tensor_parallel_size (z) must divide world size (z ) evenly.RANK LOCAL_RANK WORLD_SIZE MASTER_ADDR localhost MASTER_PORT12345levelz6vllm_mode must be either 'server' or 'colocate', got 'z'.) max_new_tokens do_sampler bos_token_id eos_token_idrr7r8r9r;r=rmax_batch_tokensi num_blocks block_size)evaluation_mode) ref_modelrt)rdevice_placementrM)rggetattrr>r?r|rZconfig _name_or_pathsplitrr/rrUrrr from_pretrainedrV architecturesr;rr2 signaturerB parameterskeysrsmodel_kwarg_keysrIrr tokenizerr TypeError pad_token eos_tokenrrrtrurvrwlistreward_func_names enumeraterrAModulerrNrhrKrlrronesrrrrkr1rr2rr7r8r9r;r<rGrHrSrTrRrrLrrMrQNotImplementedErrorr4rdictanyvaluesrIrJrr_step_buffered_inputswarnings_issuedr\r]r1rrr7r0r+r6 ImportErrorr$_forward_redirectionLigerFusedLinearGRPOLossliger_grpo_lossr)_metrics_total_train_tokensrUrWrVr*r5_logsrPrr9rtis_main_processrCrDrEr#rF vllm_clientinit_communicatorcudacurrent_device num_processes distributednew_subgroups_by_enumerationritp_group process_indexrCrmlocal_process_indexrpllmrirAsleeprBguided_decoding_regex_last_loaded_stepwait_for_everyonerr=r:updatergeneration_configmodel_accepts_loss_kwargsrdadd_model_tags _tag_namesis_deepspeed_enabledrFis_fsdp_enabledrG prepare_modelrN add_callbackr )rhrdrhririrjrrrkrlrmrn model_namer/model_idrr architecturerr reward_funcreward_processing_classr_ max_model_lenr:r^s ` rmr]_UnslothGRPOTrainer.__init__ss 5- ( (WT:-F-Fj%0E9 $ %DN <",UC"8"8ell>X>XJ#))#.r2J E23D!228b eS ! !H%))'2E%--&EME3''u--2'* BBGK  //9F"<1E1Ea1HIL 00O=NOE||11H%%1?5"233   emm , 7 7 < < >""5#7#7#9#A#ABMMRRT    #,<L!#' 5NA{+s++"D"T"T#,-#1B# Q,q/29955&&--l1o.D.D.R.R.X.XY\.]^`.ab&&--l1o.F.FG6)    *4&&'3|+<< 0T5H5H1I0JK""%l"3!4A7#(,,t/B/B%--"XD "'**S->emm"TD  % ,)-\1B(B %5t<<)B(C % ( )S-> >;C@Yt'A'AB"#STt'A'AB    D1 ===$&&!4 ~~)##3300<#'#<#<%,T-B-B,C1TEZEZD[#\'18X\XpXp'qD$$$66ejj>W>W>Y6Z:-''558V8VVZ[[$5d6T6T5UV ,,::;9F 11A5','8'8'U'U&+4+;+;+I+ITMkMk+k%l%l!q4+I+I'IAPQEUYUsUsKs!tu%l($DM1 &))9)9)G)G%H 6"+.t/?/?/S/S+T <(+.t/?/?/M/M+N <(,.JJNN=+,V =),.JJNN=',R =)))5$:T:T:`$($:$:T=W=W$WM$(M ,,99333HHNNN+ #YZ^ZhZhYiik!lmm)-)H)HD &%'D "    . . 0#'"<"<! ) 6 6 ) 6 6 ) 6 6#//&*&=&=(,(A(A ! **8;!"4526!,/25!,/%%1!(()?)?@%5%J8I%JD " */& !!$//2 >> %(((!24>>4CSCS!T%%%!-dnnd>N>N!O!%!1!1!?!?`d!?!e       2T^^Y]YiYij k'(9(9:NA{+77,,,+<[$JZJZ+[D%%a(,0+;+;+I+I#TD,J,D%%a( ;Cs =AE(c2URc SS/Ulgg)Nrr)_signature_columns)rhs rm _set_signature_columns_if_needed4_UnslothGRPOTrainer._set_signature_columns_if_neededs"  " " *'/&9D # +rpcxURc [S5eURnURn[5(a0[ U[ R 5(aURUSS9nOURUSS9nURURR-UURRURRURRS.n[ U[R R"R$5(dUR'5US'URR(US'[+[,URRURR.S9US'URR0US 'UR2R5[7U40UD65$) Nz+Trainer: training requires a train_dataset.training) description)r collate_fn num_workers pin_memorypersistent_workerssampler drop_last)rrankworker_init_fnprefetch_factor)rirrr4r|r(r_remove_unused_columns"_get_collator_with_removed_columns_train_batch_sizerir6rrr rUutilsdatar_get_train_samplerrrErNrrrtpreparer)rhrirdataloader_paramss rmget_train_dataloader(_UnslothGRPOTrainer.get_train_dataloader+st    %JK K** ** " "z-AQAQ'R'R 77 S]7^M CCM_iCjM004993Q3QQ'99;;))99"&))"I"I  -)9)9)I)IJJ+/+B+B+D i (-1YY-K-K k *29)I)IPTPYPYPgPg3 . /48993W3W / 0'' =(VDU(VWWrpdatasetrc Uc URn[UURURRUR-UR URR -URURRS9$)N) data_sourcemini_repeat_countr repeat_countshuffler) rirr2rir5rIr6r4r)rhrs rmr&_UnslothGRPOTrainer._get_train_samplerIsq2 ?((G"22yy66$:N:NN,,tyy/M/MM((   rpcT[UURURRS9$)N)rrr)rr2rir)rhrjs rm_get_eval_sampler%_UnslothGRPOTrainer._get_eval_samplerms&$"22  rpc D[U5(aURRnX#S.n UbUbXiS'UbXYS'UbXyS'UbXS'SUR;aUS-U S'SU S 'UR"S 0U D6Rn U SS2SS 2SS24n U SS2U*S2SS24n U $) N)rrgrVrUrWrXrrF use_cacher{rM)r7 base_modelrdrlast_hidden_state) rhrrrgrrUrVrWrX model_inputsrs rm_get_last_hidden_state*_UnslothGRPOTrainer._get_last_hidden_stateus  ) )-88>>O&/Q   %,*B-;) *  #+7 (  +3G/ 0  "*5 ' t44 4-;a-?L) *$) [!+11ALASS-a"ai8-a.1A1.DE  rp entropiesr thresholdcjXR5R5nUR5S:Xa#[R"U[RS9$[R "UR5/UR S9nURRU5R5R5n[R"XdR5- UR S9n[R"XG/5n[R"[R"U5U/5n URRU5n URRU 5n XR5n [R"X5n XR5-nX:nXR5-$)ah Returns a binary mask identifying tokens whose entropy exceeds a given quantile threshold. Args: entropies (`torch.Tensor`): Tensor of shape (batch_size, seq_len) with per-token entropy values. mask (`torch.Tensor`): Binary mask of the same shape as `entropies`, where `1` indicates valid tokens and `0` padding. threshold (`float`): Quantile threshold between `0.0` and `1.0` to select high-entropy tokens. Returns: `torch.Tensor`: Boolean mask of shape (batch_size, seq_len), where `True` indicates tokens with entropy >= threshold and `False` otherwise. rrzr)boolfloatnumelrUrrrrtr-rrsr1cat ones_likequantile)rhrrrnon_pad_entropiesnon_pad_entropies_seq_length max_non_pad_entropies_seq_lengthpaddingpadded_entropiespadded_entropies_maskall_padded_entropiesall_padded_entropies_maskall_non_padded_entropiesentropy_thresholdmasked_entropies entropy_masks rmget_high_entropy_mask)_UnslothGRPOTrainer.get_high_entropy_masksf"&iik288:  " " $ )##IUZZ@ @ (-||5F5L5L5N4OXaXhXh'i$+/+;+;+B+BC_+`+d+d+f+k+k+m(++ ,/F/F/H HQbQiQi !99&7%AB % 5??;L+Mw*W X#//667GH$($4$4$;$;>"93?3F_d_l_lH\+1::nd+KVZZXhjnMo.06 ;QRV0WY_YcYcdqrvYw+9;O;O)P!0(1-;+7-;3G*5 "!&"1(1-;+7-;3G*5-;a-?"!&,0! "E 3F ;I9X::=BJJ5 69$ $=,+XWs%7H A&G13H1 G? ;H Hextra_prefixesc^U=(d /nS/U-nUHnURUS5nM U$)Nz_checkpoint_wrapped_module.r)replace)rhnamer4prefixesprefixs rm_fix_param_name_to_vllm+_UnslothGRPOTrainer._fix_param_name_to_vllms8'-212^CF<<+D rprr9cUc [5nUR5H%upEU(aUSU3OUnURXVUS9 M' [U[5(a[R "USSS9 UR 5HupxU(aUSU3OUn URU S/S9n X;aM-URU 5 URS:XaBURR(a'URRXR5 MURS :XdMM SSS5 gg!,(df  g=f) zdMemory-efficient post-order traversal of FSDP modules to extract full parameters and sync with vLLM.Nrr)r9visitedF)recurse writebackz_fsdp_wrapped_module.)r4rr)setnamed_children_sync_fsdp1_params_to_vllmr|rsummon_full_paramsnamed_parametersr:addr?rtrrupdate_named_paramr) rhrr9r= child_name child_module child_prefix param_nameparam full_names rmrB._UnslothGRPOTrainer._sync_fsdp1_params_to_vllms& ?eG(.(=(=(? $J7=fXQzl3:L  + +7 , )@ fd # #((%P)/)@)@)B%JKD||f!56%%'E~~)d.>.>.N.N  33D@:-*rpcgrErM)rhrirjs rm_move_model_to_vllm'_UnslothGRPOTrainer._move_model_to_vllmBs4rpgeneration_batchclURR(aSOSnUS:XaURRUR-nUR U-S:Xd UR cgURU5n[U5n[U5n[XRR5nUVs/sHn[U5PM snUlUR UR URR-nU=R S- slU$URU5nU$! N=fs snf)Nrrrr) rdrrir6rIrr_generate_and_score_completionsrRrQrSrX)rhrWmodegenerate_everygeneration_batchesbatchinputss rm_prepare_inputs#_UnslothGRPOTrainer._prepare_inputsDs"**--w6 7?!YY;;d>Q>QQNzzN*a/43H3H3P#'#G#GHX#Y #=>N#O (=>N(O%&77GIgIg%h"Zl(mZlQV)Ee)LZl(m%**4:: 8V8V+VWF JJ!OJ  99:JKF (ms D*3D1*D.c >URRn[R"[ U5[ UR 5US9nUSVs/sH owS;dM UPM nnUVV s0sHowUV s/sHoUPM sn _M n nn UR U S'[[UR URUR55GHkun upn[X5 [U [R5(a[US5(aE[X#5VVs/sH unnSUU-0PM nnnUVs/sHn[!UU 5SPM nnO#[X#5VVs/sH unnUU-PM nnnU "USSS S S 9n["TU]IU5n[R&"5 U "S0UD6R(SS2S4USS2U 4'SSS5 O[U "SX#US .U D6nUVs/sHnUbUO[R*PM nn[R,"U[R.US 9USS2U 4'SSS5 GMn [R0"U5R3SS9R55(a[R0"U5R3SS9R7SS9SSnU R95VVs0sHunnUS:wdMUUU_M nnnUUUS'UUUS'[:R<"SUS35 [?U5nU$s snfs sn fs sn nfs snnfs snfs snnf!,(df  GN=fs snf!,(df  GM=fs snnf)Nrr)rrcompletion_ids trainer_statemessagestextptTrightFrereturn_tensorsr" padding_sideadd_special_tokens)prompts completionsrbrrrr)as_tuplerrz=All reward functions returned None for the following kwargs: zH Please ensure that at least one reward function returns a valid reward.rM) rtrrUr1rlrhstaterrrkrrKr|rArr3r%r\r_rrnanrrisnanallrnonzerorOr;rr-)rhr^rlrmcompletion_ids_listrrewards_per_funckeyrexample reward_kwargsrrrreward_func_namepcrdrtexts reward_inputsoutput_reward_funcreward nan_row_idxvaluerow_reward_kwargsr^s rm_calculate_rewards&_UnslothGRPOTrainer._calculate_rewardsjso!!(( ;;s7|S9J9J5KTZ[ &ayby7a,aybNRSds6B6s|6BBd S*. o&KT !!4#A#A4CYCY ZL GAG 6F#4:k29955(33DGD]#^D]DAqZQ$7D]#^bj kbj]^!4Q8O!PQW!Xbj k36w3L M3L41aQ3L M$;"4T[pu%M%*G$;M$JM--/1<1M}1M1T1TUVXYUY1Z(A.0/*5* 'Qd*hu*&ew)wdvZ`F4F&EII*Udv&)w-2\\:LTYTaTajp-q$QT*);:L 4 ;;' ( , , , 3 7 7 9 9++&67;;;BJJTXJYZ[\]^_K:G:M:M:O!:OJCSVZiSi'U;'':O !+2+*> h '.9+.F l + NNPQbPcdZZ ""23_cBS$_ k M 0/*x%;:2!s LL+ L5LL%AM &L$ :M L*M +L/ =1M .!L5M 'M-M ? M ML$M 5 M ? M  M r^c8>>URRnURR(aSOSnUVs/sHoDSPM nn[R "U5n0nSUS;nU(a_UV s/sHoR S5PM n n SU V s/sHo/PM sn 0nUH$n [U [5(dM[U SS9 M& [URS S5n U cS n [S 5n/nUHn0n[U5[LaSU0nUR5U- nUH)nUU ;dM UUn[U5[ LdM$UUU'M+ [#UUR40UD6SnUR%U5 M UR"ScUS S SSS.UD6n[&TjU]QU5nUSUSnnUR*GbUR,UR.UR0/nUVs/sH nUcMUPM nn[3UUUR*U5unnUVs/sH=n6"S[4R8"UR:5S3S U5PM? nnUR<bTUVs/sHGn6"S[4R8"UR<5S3UR<U5PMI nnUR>(GaUR@S:XaSURBRD(a8[FRHRK5 URLRO5 URPRRURT:wa+URW5 URPRRUl*UR@S:XGa[YU5nU(a [YW 5nURRZ(aUSSUR\2nU(aWSSUR\2nOSn[_US5 UR`RcUUUR\URdURfURhURjcSO URjURlcSO URlURnURpURBRrS9 n U SU S4n!SSS5 OSn!W!/n"[uU"SS9 U"Sun#n$[wURRx[{U5-URRxS-[{U5-5n%U#U%n#U$U%n$GOUR@S:XGaURp(a[}URpS9n&OSn&SURdURfURhURjcSO URjURlcSO URlURnU&SS . n'URBRrb%U'RURBRr5 [Sc0U'D6n(URS:a[{U5n)[UR5V*s/sHn*SPM n+n*[FRRU+XRS!9 U+V,V-s/sHn,U,Hn-U-PM M nn,n-U(al[UR5V*s/sHn*SPM n.n*[FRRU.W URS!9 U.V,V s/sHn,U,HoPM M nn,n OSnO UnU(aW OSnU(aLU(aE/n/[UU5H2un n0U0bU/R%U SU00S".5 M!U/R%U 5 M4 OUn/[_US5 URLRcU/U(SURRS#S S$9S%9n1SSS5 W1V2V s/sH#n2U2RHn U RPM M% n#n2n U1V2V V3s/sH^n2U2RHJn U RV3s/sH/n3[[U3R555RPM1 sn3PML M` n$n n2n3URS:aF[FRRURS!9n4[wU4W)-U4S-U)-5n5U#U5n#U$U5n$URBRD(aURLRSS&9 W#V6s/sHn6[FR"U6US'9PM n#n6[U#URS(9n#[FR"UU#/SS)9n7W$V8s/sH'n8[FR"U8U[FRS*9PM) n9n8[U9SS(9n9GOUR(GaUR"ScS+U0UD6n:URRRn;[5(aS,URRlYOS-URRlY[_US.5 [URURURBRS/9n<[FR"5 UR(a[R"URSS09O [5 URBR(a U[FR"U>RS54U>RS5[FRUS69n?U>R5RSS)9U>RSS)9U?U>RSS)9'[FR"U>RS5US'9RU>RS5S5n@U@U?RS5:*R5nA[U#UAR55VBVCs/sHunBnCUBUCR5PM nDnBnCWARS5nEURRUE5nFUFR5nGUR(a3U>RSS)9)nHWAUH)RS5R5-nA[FR"UWA/SS)9nIU#RS5nJUS:XaURBROURBRnKU(d [U7URR5n7[FR"5 URBRUR-nLURBGRUL-S:wd#UR>(atUGR(abUGRURU7WIWJWKUR S75UR S85UR S95UR S:5S;9 unMn*OSnMUR>(aMUGR(a;[FGR"WMW9- 5nN[FGR"UNUGR S<9nNUGR S:wGaUGRbcUGRUGRU7WIWJWKUR S75UR S85UR S95UR S:5S=9 unOn*OURGRUR5GR5 UGRURU7WIWJWKUR S75UR S85UR S95UR S:5S=9 unOn*SSS5 OSnOSSS5 URGRU#S S>9nPG[US5(aS/nQ[UWP5H@un nRU SS?S@:XaU GR5SAOS nSWQR%S@USWR-SB./5 MB OWPnQUGRXWQWD5nTUTUGRRU5RS5-GRSS)9nUUUGR!SUR\5GR#SS)9nVUVGR%UR\SS)9nVUUUV- nWUGR&SC;aGWUGR!SUR\5GR)SS)9nXUXGR%UR\SS)9nXONUGR&SD:Xa"WUGR)5GR+UU5nXOG[-SEUGR&SF35e[FGR."WX[FGR0"UX55nYUGR&SG:waWWWXSH-- nW[wURRx[{U5-URRxS-[{U5-5n%WWGR35nZUWU%nWUS:XadURP=GR4URRWIR55R5GR75- slURPGR4/UGR8USI'UGR8USJR%WFGR;5GR#5GR755 UGR8USKR%UFGR;5GR=5GR755 UGR8USLR%UFGR;5GR?5GR755 URRU>RSS)95n[UFU[n\S[{U\5[{UF5- - n]UGR8USMR%U]5 [{U\5S:Xa[FGR@"SUS'9n\UGR8USNR%W\GR;5GR#5GR755 UGR8USOR%U\GR;5GR=5GR755 UGR8USPR%U\GR;5GR?5GR755 G[CUGRD5Hun^n_[FGRF"WTSS2U^45GR75n`UGR8USQU_SR3R%U`5 G[IUTSS2U^45GR75naUGR8USQU_SS3R%Ua5 M UGR8USTR%WVGR#5GR755 UGR8USUR%WXGR#5GR755 UGR8USVR%WYGR;5GR#5GR755 UGRJSGRM[YU55 UGRJSWGRM[YWP55 G[CUGRD5H>un^nbUGRJSXUbGRMWTSS2U^4R55 M@ UGRJSYGRMWZR55 U(a)UGRJSGRM[YW 55 UR>(GaKUGR(Ga8[FGRN"WMW9- 5ncUcWAR5ncUcGRQ5S:a[FGR""Wc5O[FR"SUS'9ndWcGRQ5S:a[FGR>"Wc5O[FR"SUS'9neUGR8USZR%URRWd5GR#5GR755 UGR8US[R%URRUe5GR?5GR755 WNWAR5nfUfGRQ5S:a[FGR<"Wf5O[FR"SUS'9ngWfGRQ5S:a[FGR""Wf5O[FR"SUS'9nhWfGRQ5S:a[FGR>"Wf5O[FR"SUS'9niUGR8US\R%G[SURRWg55GR755 UGR8US]R%URRWh5GRG5GR755 UGR8US^R%G[UURRUi55GR755 UUU#WAWWWGS_.n WMbWMU S`'UR>(aUGR(aWNU Sa'WObWOU Sb'S7U;aUS7U S7'S8U;aUS8U S8'S9U;aUS9U S9'S:U;aUS:U S:'U $s snfs sn fs sn fs snfs snfs snf!,(df  GN1=fs sn*fs sn-n,fs sn*fs sn n,f!,(df  GN]=fs sn n2fs sn3fs sn3n n2fs sn6fs sn8f!,(df  GN=f!,(df  GN=f!,(df  GN=f!,(df  GN=f!,(df  GN"=fs sn fs sn6fs sn6f!,(df  GN=f!,(df  GN=f!,(df  GN=f!,(df  GN=fs snCnBf!,(df  G N=f!,(df  G N#=f)dNrrrrrimagesr) num_images chat_templater)rchosenrejectedrrdlabelrfTleftFrhrrgz^(z)+(rrz vLLM.generater{r) rlrnr;rr7r8r9 max_tokensrr:rblogprobs) from_process)regex) rr;rr7r8r9rguided_decodingr)r)rmulti_modal_datagrpo_trainer_lora_model) load_tensors)ruse_tqdm lora_requestrr) padding_valuer)rrrepaged_attention sdpa_pagedztransformers.generate_batch)gather_deepspeed3_params)r>)r progress_barrg)rrjztransformers.generate)rdisable_compilernrUrVrWrX)rUrVrWrXr)rrUrVrWrX)skip_special_tokensrole assistantcontent)rr)rnoner]z!Invalid value for scale_rewards: z-. Must be one of 'batch', 'group', or 'none'.rg-C6? num_tokenszcompletions/mean_lengthzcompletions/min_lengthzcompletions/max_lengthzcompletions/clipped_ratioz"completions/mean_terminated_lengthz!completions/min_terminated_lengthz!completions/max_terminated_lengthrewards//mean/stdr reward_stdfrac_reward_zero_stdrrrz&sampling/sampling_logp_difference/meanz%sampling/sampling_logp_difference/maxz&sampling/importance_sampling_ratio/minz'sampling/importance_sampling_ratio/meanz&sampling/importance_sampling_ratio/max) prompt_ids prompt_maskrbr|rnum_items_in_batchold_per_token_logpsimportance_sampling_ratioref_per_token_logpsrM)rtrrdrr'deepcopyrr|rrHrrrr@ryrrrZr=rr\r_r1rurvrwrWrMsubescaperrtr>r?rirArUr empty_cacherwake_uprp global_steprrUr.rr2rKrgenerater;rr7r8r9rrr:r&slicerrlGuidedDecodingParamsrrrHrirall_gather_objectrr load_loraoutputs token_idsrnextiterrlogprobget_rankrrrDrrrr< model_wrappedr_attn_implementationr5rYr3rrrrCrBrrror[rnrgenerate_batchrrgenerated_tokensrrfulllongrbargmaxrrexpandrrtolistrr-rMrrrr6rIrrSr2rrrTrrrudisable_adapter batch_decoder3poprrKnansumviewrrepeat_interleaverLstd expand_asriscloserclonenum_input_tokens_seenrsrrrrr1rrnanmeanr@rextendabsrr?r>)krhr^rrZrrloriginal_promptsrj has_imagesrxrimgr_chat_template__supported_keys_ prompts_text _example__tokenizer_kwargs_ _left_keys_kv_x_ prompt_inputsrr protectedrreall_prompts_text all_imagesordered_set_of_promptsordered_set_of_imagesrkpayloadobj_listrb all_logprobs process_slicerr:r orig_sizergathered_promptssublistr{gathered_images vllm_inputsr all_outputsrlplocal_rank_in_grouptp_sliceidsprompt_completion_idsrsampling_per_token_logpspaged_prompt_inputs previous_attnr prompt_lengthis_eoseos_idxsequence_indicesr|rowmask_rowrucompletion_lengthsagg_completion_lengthsrtruncated_completionsrgrrr[rrrcompletions_textrmr bootstraprvrmean_grouped_rewardsr std_rewards is_std_zeroall_process_advantagesagg_terminated_with_eosterm_completion_lengthsclipped_completions_ratiorrz mean_rewardsstd_func_rewardsr7r mean_delta max_delta flat_is_ratiomin_importance_sampling_ratiomean_importance_sampling_ratiomax_importance_sampling_ratior^sk rmrY3_UnslothGRPOTrainer._generate_and_score_completionss!!((**--w6(./1X;/  ==1 q ) :@A&wkk'*&FA& 9&3& 9:F!fd++/1E" "$"7"7$O  "bObc I!# Id*%y1 #..*-==K '!! AAw#~01*1- ! ,It7L7LcPbcdlmC    $ -- $    / > "/ "]K  ! ! -,,d.H.H$JbJbcI,5KI5IIK&DK)?)?' #J _kk^jVZBFFb4>>)B(C2#FDQ^jLk +eq eq]aBFFa $*:*: ; !.v!6J##33.>>UAUAU>U-V*!0:;Rd>R>R;R0S-04-*4A!%!1!1!:!:$:#8"22/3/F/F(,(8(8"&**(, (:" )-);#'+'A'A262L2L.2ii.I.I"; "$**:#;VJ=O"PBA #G$9%hQ?/7{, %$$22S\A%%33a73w<G! "0 !>+M: :---&:A[A[&\O&*O*.*A*A#'#3#3!ZZ#'::#5R4::$(JJ$6SDJJ"&"<"<'6 ! %!99..:%,,TYY-H-HI"0"E3D"E11A5!$L 1I6;Dt?]?]9^*_9^A49^*_));;OV[_[h[h;i9H%\gT[ScT[c %\ %) '3$+54J*"$K),-=z)J  ,'..&W^`eVf/gh'..v6 *K #3K&t_="&(("3"3KQ`kpBFBLBLBVBVWpAEBVBF"3#GK>CN!l+w\c\k\kRX&"2"2\k"2+!l$/ #.")//@FOT$ryy{+,44O"1P#.  11A5+0*;*;*D*D4==*D*Y'$%89%DGZ]^G^bkFklH%3H%=N#/#9L9933HHNNN+KYY.3ell3v>.NY t?P?PQN$)IIz>.JPQ$R ![g([gx XfEMMJ[g %((++CSV'W $  ( ( (#'"7"7"T\"TV"T  ..55JJM(**AR""))>AM""))>!$(EF+&&(8(8SWS\S\SvSv$ NRNbNb''(:(:EJhshuu99>>#&&u~~6YY^^#&&u}}5))+"1"@"@+55I_I_ns#A#K,v G EPDVDVDXYDX&f55DXNYJXY.3ell3v>.NY t?P?P_fgNFYFcFcdFcs%,,s6:FcJdZt7H7HW]^J$)IIz>.JPQ$R !=JD   % % :"$(?@+&&(8(8SWS\S\SvSv$ NRNbNb''(:(:EJhshuuNXZeK k*M:J,K(7(@(@)#)7;7M7M_c)%v A'OOA.M.q.=./@AJ21mn3DEN 4#4#44**fkk!n. AejjY_`%+ZZ\%8%8Q%8%? q @Q%R q !" << AvFMMfkkZ[n^`a+w/@/@/CCHHJLO~_n_s_s_uKvwKv-#xs8}335Kvw-003!%!1!1!8!89K!L3779  * *%+ZZAZ%6$6 !-2G1G0R0RST0U0Y0Y0[[OK#AqI',,Q/>BgoTYY::SWS\S\SwSw $56KTMbMbMoMo$p ! ]]_"YY;;d>Q>QQNyy44~EJ $"J"J"J)-)P)PJJ)""!.!2!2>!B#0#4#45E#F)6):):;Q)R - 1 1- @*Q *&#Q'+#}}!I!I!I,1II6ILd6d,e),1KK-43T3T-) yyC>>--1-T-T-&&#-%2%6%6~%F'4'8'89I'J-:->->?U-V$1$5$5m$D.U .*'))66tzzBRRT151X1X JJ1**'1)6):):>)J+8+<+<=M+N1>1B1BCY1Z(5(9(9-(H2Y 2.+QUT'+#{@ 00==nbf=g VAY ' 'K&)'3C&D" 7=bz&7I[7XFJJL3^` ""[YQ[E[$\#]^'E+K  226[Zmn$d&9&9&<&T>Z>Z>\>a>a>c>h>h>jk d45<<=S=Y=Y=[=_=_=a=f=f=hi d45<<=S=Y=Y=[=_=_=a=f=f=hi#'"2"2"9"9&***:K"L"89P"Q$%,C(DsKaGb(b$b! d78??@YZ & '1 ,&+kk!F&C # d@AHHI`IfIfIhImImIoItItIvw d?@GGH_HeHeHgHkHkHmHrHrHtu d?@GGH_HeHeHgHkHkHmHrHrHtu$-T-C-C#D A ==)9!Q$)?@EEGL MM$ (+;*(E(E(GH  JJw  & &}V'< = ===TEEEII14LLME/..01E.3kkma.?E*U\\RU^dEeJ,1KKMA,= %(5<%BF> " } ,'45E'FF# $ !] 2-:;Q-RF) * M )$1-$@F= ! e0B 9LL l @BAp(]'[+`%\">="mO $Z(8,+vu GF ZYdvu A@2xLUTa_sAw 6Aw Aw.Aw9Aw AAw!7AAw&B0Aw+ Aw= Ax Ax>Ax 5Ax *Ax%=$Ax0!6Ax+ Ax04Ax7.Ax<4Az  Ay7";Ay%B Ay (&Ay Ay Ay%Ay7&Az Az$Az (Az%4A{ A{(;Az<#-Az* Az<A{ A{ ) A{2"GA| FA"A{8G# A| w+ Aw:x Ax"x+Ax0y Ayy Ay y Ay" yAy%y% Ay4 y/Ay7y7 Az zAz z Azz* Az9 z4Az<z< A{ {A{{ A{ {A{ { A{/{8 A| |A| | A|c USUSpCUSUSpe[R"X5/SS9n[R"XF/SS9nURS5n URUUUU UR S5UR S5UR S 5UR S 55n UR U UR RUUUS UR RUR S 5UR S 5S9upURS:waU SOSn U SnURR(aSOSnURS:waVURUSRURRU 5R!5R#55 URUSRURRU5R!5R#55 XR$- $)Nrrrbr|rrrUrVrWrXrrr)_input lin_weightselected_token_idsrgrbiasrrrrr{rrkl clip_ratio)rUrrrrrrrqrrrdrrrrtr-rrs#current_gradient_accumulation_steps)rhrr^rrrbr|rrgrrrmetricsrrrZs rmcompute_liger_loss&_UnslothGRPOTrainer.compute_liger_loss s"("6}8MK*01A*BFK\D]IIz:B K#AqI',,Q/!77     JJ~ & JJ' ( JJ- . JJ} %  ,,$&..55-*l+ ((-- & +@ A & +@ A-  !% S 0'!*dR[ **--w6 99  MM$  % , ,T-=-=-D-DW-M-R-R-T-Y-Y-[ \ dL)001A1A1H1H1T1Y1Y1[1`1`1bc>>>>rpcr^U(a [S5eUSUSpeUSUSpURSS5URSS5pURSS5URS S5p[R"XW/S S 9n U Rup[R"Xh/S S 9nUR S 5nU nUnS.U4S jjnU"XUUS S9nURSS5nUSnURSS5nU SS2U*S24n [ URSS5nUcSn[ URSS5nUcSn[ URSS5nUcSnUbUbUSS2SS2SS24nUbUSS2SS2SS24nUSS2SS2SS24n[UUUU UTRU4U U TRRTRTRTRTRRTRR TRR"UUUS. 6unnnGO,[%TRS5(a['S/0ST_SU_SU _SU _SU_SU_SU_SU_SU_STRR(_STRR_STR_S TR_S!TR_S"TRR_S#TRR _S$TRR"_S%U_S&U_S'U_S(U_6unnnOA['TUUUUUUTRR(TRR"UUUUS)9 unnnS*TR*;aTR,R.(aS+OS*nTR*US,R1UR355 TR*US-R1UR355 U$TR*S,R1UR355 TR*S-R1UR355 U$)0Nz2The GRPOTrainer does not support returning outputsrrrbr|rUrVrWrXrrc z>[TS5(aTRXX#U5$TRXX#XEU5S$)N_get_per_token_logpsr)rgr%r2)rdrrgrrr0r1rhs rmr2_UnslothGRPOTrainer.compute_loss.. s\t344  % %eXi j Y  3 3EnfpDU VWX Y YrpT)r1rrrfinal_logit_softcappingr logit_scalelogits_scalingr{) rUrVrrrrrrrrrrrr{rrrrr?rrrrrrrrrrg) r{rrr|rrrr?rrrrrgrrrrNFFrM)rrrUrrrrrrrrirrrrrrrrgrrrcontrolshould_evaluaterrs) rhrdr^return_outputsrrrrbr|rUrVrWrXrr}r~rgrr;_logits_to_keepget_logps_funcrrrrrrrrrrrZs ` rmr _UnslothGRPOTrainer.compute_loss s QR R"("6}8MK*01A*BFK\D]'-zz.$'GTdfjIkn,2JJ7Md,SU[U_U_`mnrUskIIz:B OO K#AqI',,Q/ ( Y  )>>ost#JJ'URSS:XaUR5$UT-R5T- $r)rrr)rr|completion_token_counts rmr<_UnslothGRPOTrainer._compute_loss..masked_batch_mean s7wwqzQvvxO+0025KKKrprentropyzclip_ratio/low_meanzclip_ratio/low_minzclip_ratio/high_meanzclip_ratio/high_maxzclip_ratio/region_mean)&rUrrr2rrQr+rrrrrrrrrrrirrr>rSrrrrrtrrdrrrr-rrsrr?r>)(rhrdr^rrrbrrgrrrr*r per_token_klrrrrrrper_token_loss1per_token_loss2per_token_lossr normalizerrZrr mean_entropyis_low_clippedis_high_clippedis_region_clippedlow_clip high_cliprgathered_low_clipgathered_high_clipgathered_clip_ratior|r3s( @@rm _compute_loss!_UnslothGRPOTrainer._compute_lossN s"("6}8MK*01A*BFK\D]'IIz:B K#AqI',,Q/&*%L%L     N3!::&67!',B!C =1&M & "  $ $s *55iRSVZVoVoRopLL 99 "()>"?  -?@DWDijmnn  L) %jj)>?:M:Uo446[n#9  ) )W 4%. "  + +z 9&//&A%F%Fr%J_M`M`acMdMjMjorMjMs%s "%;%E%Eb%I "5d6T6T5UV"" 12VQ)9)9%91t?P?P;PQ 99?? &[[TYY__=F :#7#7#:: :#7#7#::))O_EE  #+l:N ==TEE+f5P.QQN 99 +dii,.FFN >>V ##o5::2>ATATUWAXA^A^cfA^AggmmoD$BBBD ^^v %"_499;o>Q>Q>S>Y>Y^a>Y>bbD$BBBD ^^y ("_499;~?R?RST?UX\XrXr?rsD$BBBD ^^v % 458H8H8V8VVJ"_499;jHD24>>2BCD D**--w6!0!4!4!6!.>.E.El.S.[.[.].b.b.de!1t'7'7#77JR>RST>UXY>YZ*_<$^%9%9%;<%o&;&;&=> &'8'>'>'@A  ,,33H= d1299:K:S:S:U:Z:Z:\] d0188@Q9R9W9W9YZ!--44Y? d23::;M;U;U;W;\;\;^_ d1299&AS:T:Y:Y:[\"..55jA d45<<=P=X=X=Z=_=_=ab rp ignore_keysc>URU5n[R"5 UR5 UR X5nSSS5 WR 5R 5nSSS5 WSS4$!,(df  N9=f!,(df  N$=frE)r_rUrcompute_loss_context_managerrrr)rhrdr^rrFrs rmprediction_step#_UnslothGRPOTrainer.prediction_step su%%f- ]]_224((7599;%%'DT454_s"BA= &B= B B Blogs start_timec  >URR(aSOSnURUR5VVs0sHupEU[ U5[ U5- _M nnnUS:Xa(UR5VVs0sH upESU3U_M nnn0UEUEn[ T U]X5 URUR5 URR(GaUR(Ga[5(ab[URSURSURSURSURR UR"5 UR$R&(GatSUR$R&;GaX[(R*GbES SKn[/URR 5/[ URS5-URSURSS .URSES URS0EnURS (a[/US 'URS HCn U b)US R1[(R3U 55 M/US R1S5 ME UR5U5n UR6(aU R9S/S 9n [(RS[(R;U S905 ggggggs snnfs snnf)Nrreval_rrrrwandbr)steprr advantager)subsetrm) dataframe)rdrrrOrrlr\logclearrtrrUr8rJrrprrVrirrOrunpandasrZrImage DataFramerWdrop_duplicatesTable) rhrKrLrZrwvalr pdtablerdfr^s rmrT_UnslothGRPOTrainer.log s**--w6<@MM$:A--/J/hcse}c)/GJ"$"'"  D% d!!#    + + +0D0D0D ""/JJx(JJ|,JJy)JJ|,JJ**11 yy"""w$))2E2E'E%))J_#!!7!789C 8@TB =%+++*CDE/K`'E"1E +Y Ks $K:Lc>URRc*[URR5RnO(URRR S5SnUR US9 [TU]!X5 g)Nrqr{)r) rirrrr7rcreate_model_cardr\_save_checkpoint)rhrdtrialrr^s rmrc$_UnslothGRPOTrainer._save_checkpoint sj 99 ! ! )dii22388J//55c:2>J *5  .rpr dataset_nametagscUR5(dg[URRS5(ac[R R URRR5(d!URRRnOSnUc [5nO$[U[5(aU1nO [U5n[URRS5(aURS5 S[R;aURS5 URUR5 [R "S5n[#UUUR$UU['5(a+[(R*b[(R*R,OS[/5SUS S S 9 nUR1[R R3UR4R6S 55 g) a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. Nrunsloth_versionunslothJOB_IDhf_jobsa @article{shao2024deepseekmath, title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}}, author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo}, year = 2024, eprint = {arXiv:2402.03300}, } GRPOzRDeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelsz 2402.03300) rrrrfrg wandb_url comet_url trainer_nametrainer_citation paper_titlepaper_idz README.md)is_world_process_zerorgrdrrCpathisdirrr@r|rZrErmrrrTdedentr/rr:rOrVurlr0savejoinrir)rhrrfrgrcitation model_cards rmrb%_UnslothGRPOTrainer.create_model_card sn"))++  4::$$o 6 6rww}}TZZM^M^MlMl?m?m**88JJ <5D c " "6Dt9D 4::$$&7 8 8 HHY  rzz ! HHY  DOO$??   )!!**%'9';'; @Ueiimm[_.0%l!    TYY%9%9;GHrp):rYrrrrrrrrrrrrrrrtrurrrrUrrMrr1r9rrrVr2rIrrrr;rrhrkrKrLr4rrQr8r7rrRr<r>rwrvrrGrTrSr?rHrW)NNNNNNr.NrE)NNNNr*)rN)FN)NNN)8rNrOrPrQr`rr rZrrrrrrrrrrr"tuplerUr Optimizer lr_schedulerLambdaLRr]rrrrr rLrrrr+r2r:rArrBrRrUrr_rrYr!rrDrIrTrcrbrSrcrds@rmrfrf;s_BJ &*CGnrUYmq59jv.2xS/)*xJZ(889xz" x  g&> ?@ x uWotCwXgOgIhDh?i%ijk x#5)@.)P#QRx$,E2I4PgKh2h,i#jxD12x(5;;#8#898EKKD\D\DeDe;ffgxl+xxt :$X<" (7*;" w" H  !*!*!X&*u||&*5<<&*\a&*fkfrfr&*PquX]C JHT#Y?x c5s*++ ,xt(?TGR~@ PXY]^aYbPc /FS%Z(/Fhuo/FQU/F/Fd/%)&*,0 CISMCIsmCICcD() CICIrprfc>^\rSrSrSrSU4SjjrSrU=r$)UnslothGRPOTraineriT a Trainer for the Group Relative Policy Optimization (GRPO) method. This algorithm was initially proposed in the paper [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300). Example: ```python from datasets import load_dataset from trl import GRPOTrainer dataset = load_dataset("trl-lib/tldr", split="train") def reward_func(completions, **kwargs): # Dummy reward function that rewards completions with more unique letters. return [float(len(set(completion))) for completion in completions] trainer = GRPOTrainer( model="Qwen/Qwen2-0.5B-Instruct", reward_funcs=reward_func, train_dataset=dataset, ) trainer.train() ``` Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. Only causal language models are supported. reward_funcs (`Union[RewardFunc, list[RewardFunc]]`): Reward functions to be used for computing the rewards. To compute the rewards, we call all the reward functions with the prompts and completions and sum the rewards. Can be either: - A single reward function, such as: - A string: The *model ID* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using [`~transformers.AutoModelForSequenceClassification.from_pretrained`] with `num_labels=1` and the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object: Only sequence classification models are supported. - A custom reward function: The function is provided with the prompts and the generated completions, plus any additional columns in the dataset. It should return a list of rewards. Custom reward functions can also return `None` when the reward is not applicable to those samples. This is useful for multi-task training where different reward functions apply to different types of samples. When a reward function returns `None` for a sample, that reward function is excluded from the reward calculation for that sample. For more details, see [Using a custom reward function](#using-a-custom-reward-function). The trainer's state is also passed to the reward function. The trainer's state is an instance of [`~transformers.TrainerState`] and can be accessed by accessing the `trainer_state` argument to the reward function's signature. - A list of reward functions, where each item can independently be any of the above types. Mixing different types within the list (e.g., a string model ID and a custom reward function) is allowed. args ([`GRPOConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]): Dataset to use for training. It must include a column `"prompt"`. Any additional columns in the dataset is ignored. The format of the samples can be either: - [Standard](dataset_formats#standard): Each sample contains plain text. - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role and content). eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Union[Dataset, IterableDataset]]`): Dataset to use for evaluation. It must meet the same requirements as `train_dataset`. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`] or `None`, *optional*, defaults to `None`): Processing class used to process the data. The padding side must be set to "left". If `None`, the processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default. reward_processing_classes (`Union[PreTrainedTokenizerBase, list[PreTrainedTokenizerBase]]`, *optional*, defaults to `None`): Processing classes corresponding to the reward functions specified in `reward_funcs`. Can be either: - A single processing class: Used when `reward_funcs` contains only one reward function. - A list of processing classes: Must match the order and length of the reward functions in `reward_funcs`. If set to `None`, or if an element of the list corresponding to a [`~transformers.PreTrainedModel`] is `None`, the tokenizer for the model is automatically loaded using [`~transformers.AutoTokenizer.from_pretrained`]. For elements in `reward_funcs` that are custom reward functions (not [`~transformers.PreTrainedModel`]), the corresponding entries in `reward_processing_classes` are ignored. callbacks (list of [`~transformers.TrainerCallback`], *optional*, defaults to `None`): List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](https://huggingface.co/docs/transformers/main_classes/callback). If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`] method. optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of [`AdamW`] on your model and a scheduler given by [`get_linear_schedule_with_warmup`] controlled by `args`. peft_config ([`~peft.PeftConfig`], *optional*, defaults to `None`): PEFT configuration used to wrap the model. If `None`, the model is not wrapped. c >Uc [5n[USS5n [U 5[LaSn [USS5n [U 5[LaSn Sn [R R SS5S:HnU(d1[R R SS5S:Xa [S5 S n [R R S S 5n[URS S5=(d [URS S5nUcUR5RnSSK J n U"U5nU[R:HnU (dU(aU (a [S5eU (dU(dU (a [S5eU (a"SUlSUlS[R S'OCU (d<U (d5US :Xa/UUlU(+UlU(aSOS[R S'[USS5b-[USS5S:XaSUl[USS5cSUl[USS5nUb/US:a)SSKJn [-U5[-S5::a [S5 [USS5S:waL[USS5nUS:Xa!UR.U:aUR.Ul[US S5c UbUUl[US!S5n[U5[LaSn[US"S5n[U5[LaSnUR (aU(aSUlS UlUR"(aU(aS UlSUlU (aSUlSUlOc[R R S S 5S#:XaS UlSUlO0U(d)U(d"UR"UlUR UlSn[95R S$S5bS n[95R S%S5bS nU(aS[R S&'S'[95;a[;US'5(dOD[US'S5n[US'S5nUc'Ub$UR<nS'5(aUUlUb![;US(5(aUR?5 S)[95;a [;[@S*5(a S+[@l!S,[95;aU[;US*5(aS+Ul![;US)5(a,[;UR@S*5(aS+UR@l!/n[EU[F5(dU/nOUnUH<nURHnURKS-US.35 URKS-US/35 M> SS0K&J'n U "S1U5 [US2S5[PRR:Xa(URTS:a[US3S5S:waSUl+S4[95;a![;US(5(aUR?5 [XT#U]"S;UUUUUUUUU S5. U D6 S4[95;a![;US65(aUR]5 [;US75(a-UR^Ra5 [;US75(aU?/[US8S5bURbUR5l1[;US95(aVURdRfn!Un"[;U"S45(a&U!U"l4U"Rjn"[;U"S45(aM&U!U"l4[;US:5(a.[m[oURpRr5U5Ul9g! GM(=f)accelerator_scalerrdrarsr^r)$rhrdrhririrjrrrkrlrnrjuse_bf16use_fp16 force_float32full_finetuningmixed_precision_dtyperrrnga_stepstransformers_versioneval_bszrr_output_logitsmodel_max_seq_lengthargs_max_seq_lengthr other_metrics _reward_funcsrrzrr> current_modelr^s$ rmr]UnslothGRPOTrainer.__init__ s < 1 34/ > %%x4/ > %%x **..)I3OSVVBJJNN3JC$PTW$W _ ` M " /H) T gt4b m]a8b =%"<"<">"D"D%05!5==('hyJA@B:Bg(9NEDF>F DIDI7;BJJ3 481F)1SDI# DIAHvfBJJ3 4 4 . :wt_^b?cgk?k!(D t\408C$/4!>E  HqL H+,0AA@A 4$ /4 7t%A1EH1}!A!AH!LptqQqQdNmt6=E(J^@H`d`| '7?  t +e^ '7?  t +e^ 99u)<\`dFY 99t)<[`TEX "'D "'D  ZZ^^5y AZ O"&D "'D "&))D "&))D  8<<)4 0 J]aN 25BJJ. / 68 +GDBR4S4S #*52BD#I #*42BD#I "*/C/O!&!5!54!122.D4G  !?!?    &( "wy.'I'Idk9Ka  )'88Za:J:W'55'BRB\B\^l:m:mZaopJpJpW ,-- ~}*m(K #.#7#7 !((84D3EU)KL!((84D3ET)JK) @.-8 4$ /<3O3O OTXT^T^abTbtXq)Q. fh 75.#A#A      0')'/(A!% 0)/ 0 fh 75/#B#B    ! 4. / /  $ $ + + -t2339Q 4. 5 A?C?W?WE & & ( < 4 ' '%%,,F!M-1139 0 - 3 3 -1106M , 4 ! !#$=dnn>R>R$SUYZDJ Q Ds 8];;^)r)NNNNNNN)rNrOrPrQr`r]rSrcrds@rmrrT s-aL$(L L rpr addFilterc \rSrSrSrSrSrg)HideLoggingMessageiJ cXlgrEre)rhres rmr]HideLoggingMessage.__init__K sd)rpc<URUR5;$rE)re getMessage)rhrs rmfilterHideLoggingMessage.filterL salln)DErprN)rNrOrPrQr]rrSrMrprmrrJ s2Erprz`use_cache=True`)r{)r`rUrtorch.nnrArFtypingrrrrr r r r trl.trainer.grpo_trainerr rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrVrWrXrY dataclassesr[r\packaging.versionr]numpyrj contextlibr^r_+TransformersDataCollatorForLanguageModelingtransformers.training_argsr`rqtypesrarsr3r2rrbrrrrautogradFunctionr rrrrrfrrgFilterrrrMrprmrs0 $IIIiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii (% "$3  4;PRS"|| \\ &,, %    \\ 6ellCELLO,bJ 5>>22J jy v4;PQO,RO,b  y 5 y 5 y 5vXI'XIr8o ,o b 6;FW^^F  '(:;<  rp