h-BSSKrSSKJr SSKrSSKrSSKJr SSKJ r J r SSKJ r J r SSK Jr SSKJrJrJrJr SSKrSSKrSSKrSSKJs Jr SSKrSSKJrJ r J!r! SSK"J#r# SS K$J%r%J&r& SS K'J(r( SS KJ)r) SS K*J+r+J,r,J-r-J.r.J/r/J0r0J1r1 SS K2J3r3J4r4J5r5J6r6J7r7J8r8 SSK9J:r: \5"5(aSSK;Jr> SSK?J@r@ SSKAJBrB \1"5(aSSKCrC\4"5(aSSKDJErEJFrF \!R"\H5rI\ "SS55rJ\ "SS55rKSS\L\RS\NS\OS\\NS\R4 SjjrP\ "SS 55rQ\ "S!S"55rR\R"5SS#\RS\T\R\R\N44S$jj5rUS%\,S\V\O\W44S&jrXSS'\RS(\NS)\\N\W4S*\NS\R4 S+jjrYS,\R4RSS4S-jr[SS.jr\"S/S05r]S1r^S2\:S\\+4S3jr_S\\V\O\N44S4jr`S2\:SS54S6jraSS7jrbSS8jrcS9\RSS4S:jreS;rfS\/55rh\ "S?S@\055ri\R4SA\RS\R4SBjjrkS,\R4RSC\RSD\NSE\NS\T\R\R\R44 SFjrlS,\R4RSC\RSD\NS\34SGjrmSS,\R4RSH\NSI\nSJ\nS\R4R4 SKjjroSL\NSD\NSM\RS\R4SNjrpSO\R4RSP\RSD\NSQ\-S\T\R\R44 SRjrq\R"5S,\R4RSP\RSS\NSD\NSQ\-4 STj5rrSU\\NSV\NSW\V\O\L\N4SX\NSY\V\O\L\N4SZ\NS[\V\O\L\N44S\jrsS]\NSY\V\O\L\N4S[\V\O\L\N44S^jrtS_\RSL\NSD\NS\T\R\R44S`jruSSajrvSb\RSc\.S\L\O4SdjrwSSe\\OSf\OSg\OSh\\OSi\L\OSj\\OSk\OSl\\OSm\\OSn\\OSo\\OS\%4SpjjrxS\\O4SqjrySr\OSs\RSS4StjrzSu\RS\RS\\R\T\RSv444Swjr{Su\RS\RS\\R\T\RSv444Sxjr|S\R4Syjr}SSz\RS{\NS\R4S|jjr~SS}\L\OS~\L\OS\V\O\L\W4S\L\WS\NS\NSS4Sjjr"SS\)5rS'\RS\R4SjrS\V\O\\R4S\NS\L\V\O\\R44SjrS\V\O\\ 4S\V\O\\ 44SjrS'\RS\R4SjrS'\RS\R4SjrSrS\V\O\R4S\V\O\\R\L\R444SjrS\V\O\\R\L\R44S\V\O\R44SjrS\RSu\RS\NS\L\NS\T\R\R44 Sjrg)N)deque)SequenceSized) dataclassfield)version)AnyLiteralOptionalUnion) Accelerator PartialStatelogging)AcceleratorState) ModelCard ModelCardData) pad_sequence)Sampler)BitsAndBytesConfigEvalPredictionGenerationConfigPreTrainedTokenizerBase TrainerStateTrainingArgumentsis_comet_available) ModelOutputis_peft_availableis_rich_availableis_torch_mlu_availableis_torch_npu_availableis_torch_xpu_available) ModelConfig)Console)Panel)Table)Text) LoraConfig PeftConfigc\rSrSr%Sr\\S'Sr\\S'Sr \\S'Sr \ \S 'S r \ \S 'S r S \\\ \4S\\ \R$44SjrSrg)DataCollatorForChatMLHz+ Data collator for ChatML format datasets. tokenizer ignore_indexN max_lengthprompt prompt_keymessages messages_keycURRc [S5eURc&[ URR S5Ulgg)NzTThe tokenizer does not have a pad token. Please set `pad_token_id` in the tokenizer.i)r- pad_token_id ValueErrorr0minmodel_max_lengthselfs K/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/utils.py __post_init__#DataCollatorForChatML.__post_init__TsE >> & & .st t ?? "!$.."A"A4HDO #examplesreturnc /n/n/n/n/nUGHnURURS5nUc,XpRSSn URR U SSS9nSU;aXpRn URR U SSS9n URU SUR SSSS9n UR U S5 SU;aUR U S5 OrUR S/[U S5-5 OPUR US5 SU;aUR US5 O!UR S/[US5-5 URUS[US5SSSS9n UR U S5 UR U S5 UR/[US5-n[U S5nUSUSXS&UR U5 GM UVs/sH&n[R"U[RS 9PM( nnUVs/sH&n[R"U[RS 9PM( nnUVs/sH&n[R"U[RS 9PM( nn[US URRS 9n[US S S 9n[US URS 9nUVs/sH&n[R"U[RS 9PM( nnUVs/sH&n[R"U[RS 9PM( nn[US URRS 9n[US S S 9nUUUUUS .$s snfs snfs snfs snfs snf)NFT)tokenizeadd_generation_prompt input_ids) truncationr0paddingreturn_tensorsadd_special_tokensattention_maskdtypeleft) padding_side padding_valuer)rFrKlabelspromptsprompt_attention_mask)getr2r4r-apply_chat_templater0appendlenr/torchtensorlongpadr6)r;r@rFrKprompts_input_idsrTrRexampleformatted_promptr1messageformatted_messagetokenized_messagetokenized_promptlabelcompletion_start_idxidsmasks r<__call__DataCollatorForChatML.__call__[s  "G&{{4??DA ' !2!23CR8#'>>#E#EU$$F$ ')!"3"34$(NN$F$Fe5%G%!%)NN%##!#'', %3%!  !2;!?@#w."))*;4%,,t5::>>ZEKLVE%,,uEJJ7VL dnnFaFab ^&PQRV&@Q@QRL]^L]SU\\#UZZ@L]^Rg hRg$d%**!ERg h 1VZVdVdVqVqr #$9^_ `#,(%:   OZL _ hs -M -M-M8-M+-M )r0)__name__ __module__ __qualname____firstlineno____doc__r__annotations__r/intr0r2strr4r=listdictr rYTensorrh__static_attributes__r?r<r+r+Hsn'&L#JJ"L#"IM d38n!5M $sELL?P:QM r?r+c\rSrSr%Sr\\S'Sr\\ \ 4\S'Sr \ \ \S'Sr\ \S 'S \\\ \4S \\ \44S jrS rg)RewardDataCollatorWithPaddinga0 Reward DataCollator class that pads the inputs to the maximum length of the batch. Args: tokenizer (`PreTrainedTokenizerBase`): The tokenizer used for encoding the data. padding (`Union[bool, str, `PaddingStrategy`]`, `optional`, defaults to `True`): padding_strategy to pass to the tokenizer. pad_to_multiple_of (`int` or `None`, `optional`, defaults to `None`): If set will pad the sequence to a multiple of the provided value. return_tensors (`str`, `optional`, defaults to `"pt"`): The tensor type to use. r-TrHNpad_to_multiple_ofptrIfeaturesrAc/n/n/nSUS;nUHwnSU;dSU;d SU;dSU;a [S5eURUSUSS.5 URUSUSS.5 U(dMcURUS5 My URRUURUR UR S 9nURRUURUR UR S 9nUS US US US S S .n U(a'[R"U[RS9nXIS'U $)Nmarginrinput_ids_choseninput_ids_rejectedattention_mask_chosenattention_mask_rejectedz{The features should include `input_ids_chosen`, `attention_mask_chosen`, `input_ids_rejected` and `attention_mask_rejected`)rFrK)rHrzrIrFrKT)rrrr return_lossrM) r7rWr-r\rHrzrIrYrZfloat) r;r|features_chosenfeatures_rejectedr~ has_marginfeature batch_chosenbatch_rejectedbatchs r<rh&RewardDataCollatorWithPadding.__call__s!, G#'1'w6*'9,G; R  " "!();!<&-.E&F   $ $!()=!>&-.G&H  z gh/03 4~~)) LL#66.. * ++ LL#66.. , !-[ 9%12B%C"0"='56F'G   \\& >> import torch >>> pad([torch.tensor([1, 2, 3]), torch.tensor([4, 5])]) tensor([[1, 2, 3], [4, 5, 0]]) >>> pad([torch.tensor([[1, 2], [3, 4]]), torch.tensor([[5, 6]])]) tensor([[[1, 2], [3, 4]], [[5, 6], [0, 0]]]) ``` rNrNdevicerOrightz&padding_side must be 'left' or 'right'c3:# UHn[SU5v M g7f)rN)slice).0ss r< pad..5s%G;aeAqkk;srL) npmaxshapetolistrYfullrXrNr enumerater7rtuple) rrQrPrzt output_shape remainderoutputi seq_start seq_sliceslicess r<r\r\s1N66G4Gq77G4a8??AL% O&88 > O1= =OZZW5 5}TUJL\L\elmneoevev wF'" 6 !$Q!''!*4I W $IEF F)%;< %G17712;%G GGq &# M15sD(cv\rSrSr%SrSr\\S'Sr\\S'Sr \ \ \S'S \ \ \\4S \ \\44S jrS rg )DPODataCollatorWithPaddingi;a DPO DataCollator class that pads the tokenized inputs to the maximum length of the batch. Args: pad_token_id (`int` defaults to 0): The tokenizer's pad_token_id. label_pad_token_id (`int`, defaults to -100): The label used for masking. is_encoder_decoder (`bool` or `None`, `optional`, defaults to `None`): Whether you model has an encoder_decoder architecture. rr6r.label_pad_token_idFis_encoder_decoderr|rAc R0nUSR5GHxnURS5(GaUR(aUVs/sHn[R"XC5PM nnUR S5(a;URS5(a%UR c [S5eUR nOQURS5(aSnO8UR S5(dSU;a URnO[S US 35e[US US 9X#'GMURS 5(a%UR c [S5eUR nOdURS5(a URnOAURS5(aSnO(URS5(aSnO[S US 35eUS;aSnOSnURS5(a[RnO[RnUVs/sHn[R"XCUS9PM nn[XVUS9X#'GMURS5(a.[R"UVs/sHoDUPM sn5X#'GMaUVs/sHoDUPM snX#'GM{ U$s snfs snfs snfs snf)Nr) _input_ids_attention_mask_labels _pixel_valuesr1rFzPadding is enabled, but the tokenizer is not configured with a padding token. Explicitly set `tokenizer.pad_token` (e.g. `tokenizer.pad_token = tokenizer.eos_token`) before calling the trainer.r)chosenrejected completiondecoderzUnexpected key in batch ''T) batch_firstrQrrr)prompt_input_idsrTrOrrMrQrP_logps)keysendswithrrY LongTensor startswithr6r7rrfloat32int64rZr\) r;r| padded_batchkexto_padrQrPrNs r<rh#DPODataCollatorWithPadding.__call__Msh !!!#AzzWXX**@HI"e..ru5FI X..QZZ 5L5L,,4",!?# )-(9(9 $566() &JKKPY]^P^(,(?(? (+DQCq)IJJ&26t[h&iLOzz,//,,4",!?# )-(9(9 I..(,(?(? $566() O44() (+DQCq)IJJII'- '. zz/22 %  % JRR2ell25>FR&)&\h&iLOH%%"',,/I"1/I"J 3;"<8Ra58"< u$xsJbS0J"N>NPR(S %GXxxzH#nnR%@OF!--/6<<>))#JJ) #((TZZ'%(TZZ*?(*JY*VV# e&288:: %'9q=9@@BGGINNP<<> ||~ 1X\ BIIKPPRWWYYYr? json_pathcURR(aU[R"USS9n[R "USSS9S-n[ USSS 9nURU5 S S S 5 g g !,(df  g =f) zDSave the content of this instance in JSON format inside `json_path`.cLUVVs0sHupUS:wdM X_M snn$s snnf)Nrrv)xrvs r<-RunningMoments.save_to_json..s*\]Ht\]RXSTabfsas\]HtHts  ) dict_factoryr"T)indent sort_keys wutf-8encodingN)ris_main_process dataclassesasdictjsondumpsopenwrite)r;r save_dict json_stringfs r< save_to_jsonRunningMoments.save_to_jsonsm    + +#**4>tuI**YqDIDPKiw71 $87 ,87s A22 Bc[USS9nUR5nSSS5 U"SSU0[R"W5D6$!,(df  N,=f)z3Create an instance from the content of `json_path`.rrNrrv)rreadrloads)clsrrrtexts r<load_from_jsonRunningMoments.load_from_jsonsE)g .!668D/?{?djj.>??/ .s A A)rrrN)rjrkrlrmrnr rorrrrrrYno_gradrtrrrqr classmethodrrurvr?r<rrs D%OCNCNE5 ]]_ZZ%u *=ZZ6%c%@@@@r?rrcURUR5n[R"UR 5UcUR 5OUR 5/URS9nUR U5nUupVXV- n[R"X- S-RUcSOU55nUR U5nX- n URU5U RU5UR54$)z Computes element-wise mean and variance of the tensor across processes. Reference: https://github.com/OpenLMLab/MOSS-RLHF/blob/40b91eb2f2b71b16919addede0341d2bef70825d/utils.py#L57C1-L73C75 rr"rL) torrYrZsumrreducemulr) rrrgr sum_and_count global_sumr global_meansum_var global_vars r<rrs {!! "BLL"&&(4>& !:==#8%**, FFr? eval_predc UupURS:Xa[R"USS9n[R"[ X5VVVVs/sH$up4[ X45HupVUS:wdM UPM M& snnnn5n[R"UVVs/sHoDH ofS:wdM UPM M snn5nOUSS2S4USS2S4:Hn[ UR 55nUS:a6[5 [RSUS[USS2S45S 35 X)nX')n[R"USS9n[R"X:H[S 9R5R5n S U 0$s snnnnfs snnf) Nr")axisr.rrLz There are z out of zu instances where the predictions for both options are equal. These instances are ignored in the accuracy computation.rMaccuracy)ndimrargmaxarrayziprprrloggerwarningrXrrr) r predictionsrR predictionrdplbl equal_maskequal_predictions_countrs r<compute_accuracyr st#K1ii !4 hh(+K(@ w(@$:QTU_QgXaknrvkvQQgQ(@ w FSF5esd{3e3FST !A&+ad*;; "%jnn&6"7 "Q & N NN45Xc+aQRdBS>T=UVgg  "+.  $ii !4 xx -U;@@BGGIH  !!9 xSs E;+ E;F & F rZlength pad_valuedimc URU5U:aU$[UR5nXRU5- XC'[R"UU[R "X@R URS.6-/US9$)Nrr#)sizerrrrYcatonesrNr)rZr!r"r#pad_sizes r< pad_to_lengthr*sr {{36!  %S!11 yyEJJ V]][[    r?modelcUR5H5n[U[RR5(dM.SUlM7 g)Nr)modules isinstancerYnnDropoutr)r+modules r<disable_dropout_in_modelr2s0--/ fehh.. / /FH"r?c NX-nXU-:wa[USUSUSX- 35eU$)Nz, inexact division: z / z = )r7)abcustom_error_messageqs r< exact_divr8sA AEz011EaSA3cRSRWQXYZZ Hr?c*\rSrSrSrSrSrSrSrg)PerPromptStatTrackeri$a- Class for tracking statistics per prompt. Mainly used to calculate advantage for the DPPO algorithm Args: buffer_size (`int`): Size of the buffer to keep for each prompt. min_count (`int`): Minimum number of samples to keep in the buffer before calculating the mean and std. c*XlX l0UlgN buffer_size min_countstats)r;r>r?s r<__init__PerPromptStatTracker.__init__/s&" r?c[R"U5n[R"U5n[R"U5n[R"U5nUGHnX!U:HnXPR;a [ UR S9URU'URURU5 [URU5UR:a0[R"U5n[R"U5S-nOI[R"URU5n[R"URU5S-nXg- U- XAU:H'GM U$)N)maxlengư>) rrunique empty_liker@rr>extendrXr?rr) r;rSrewardsrE advantagesr1prompt_rewardsrrs r<rPerPromptStatTracker.update4s((7#((7#7#]]7+ F$%67NZZ'%*$2B2B%C 6" JJv  % %n 54::f%&7www'ffWo,wwtzz&12ffTZZ/047-;-Bc,IJ&( )r?c URR5VVs0sH=upU[R"U5[R"U5[ U5S._M? snn$s snnf)N)rrr)r@itemsrrrrX)r;rrs r< get_statsPerPromptStatTracker.get_statsIsMW[WaWaWgWgWijWitqBGGAJrvvay3q6JJWijjjsAA&r=N) rjrkrlrmrnrArrNrurvr?r<r:r:$s *kr?r:c^UR5Humn[U[RR5(dST;a!UR [R 5nMV[U4SjS55(dMr[US5(dMURR[R :XdMUR [R5nM g)Nnormc3,># UH oT;v M g7fr<rv)rrnames r<r.peft_module_casting_to_bf16..QsN$Mqd$Ms)lm_head embed_tokenswtewpeweight) named_modulesr.rYr/ LayerNormrranyhasattrrYrNbfloat16)r+r1rSs @r<peft_module_casting_to_bf16r_Ms++- f fehh00 1 1Vt^YYu}}-F N$MN N Nvx((==&&%--7#YYu~~6F .r? model_argscUR(a7[SURURURURS9nU$UR (a [SS9nU$SnU$)NT) load_in_4bitbnb_4bit_compute_dtypebnb_4bit_quant_typebnb_4bit_use_double_quantbnb_4bit_quant_storage) load_in_8bit)rbrrNrduse_bnb_nested_quantrg)r`quantization_configs r<get_quantization_configrjWst0#-#3#3 * > >&0&E&E#-#3#3     0  # r?c[RR5(d[5(aS[ 5R 0$g)N)rYcuda is_availabler!rlocal_process_indexrvr?r<get_kbit_device_maprpjs4 zz  $:$<$<LN6677r?zOptional[PeftConfig]c URSLag[5(d [S5e[URUR UR URURSURURURS9 nU$)NFzYou need to have PEFT library installed in your environment, make sure to install `peft`. Make sure to run `pip install -U peft`.none) task_typertarget_modules lora_alpha lora_dropoutbias use_rslorause_doramodules_to_save) use_peftrr7r(lora_task_typelora_rlora_target_modulesrvrwryrzlora_modules_to_save)r` peft_configs r<get_peft_configrqse#    6  ++   !55((,, (($$"77 K r?cj[R"S/5RUR5[R"UR5R -n[R "U5RUR5nUS:a"[R"USU--5SU-- $U$)a Get the exponent cap of a value. This is used to cap the exponent of a value to avoid overflow. The formula is : log(value.dtype.max) E.g. for float32 data type, the maximum exponent value is 88.7228 to 4 decimal points. Args: value (`torch.Tensor`): The input tensor to obtain the data type decimal (`int`): The number of decimal points of the output exponent cap. eg: direct calling exp(log(torch.float32.max)) will result in inf so we cap the exponent to 88.7228 to avoid overflow. rLr ) rYzerosrrNfinforlogrfloor)valuedecimal vdtype_maxvdtype_log_maxs r< get_exp_caprsaS!$$U[[1EKK 4L4P4PPJYYz*--ell;NFMPQk5;;~G 3 4r7{ BeWeer?cxUS:a [U5OUn[R"[R"XS95$)Nr)r)rrYexpclamp)rcaps r<cap_exprs, #a+e SC 99U[[0 11r?dfcZ[5(d [S5e[5n[SS9nURHnUR U5 M UR 5H5upEUR"UR[5R56 M7 URU5 g)NzgThe function `print_rich_table` requires the `rich` library. Please install it with `pip install rich`.T) show_lines) r ImportErrorr$r&columns add_columniterrowsadd_rowastyperqrprint)rconsoletablecolumn_rows r<print_rich_tablers    u  iG T "E**  ++- szz#--/0  MM%r?zT{% for message in messages %}{{' ' + message['content']}}{% endfor %}{{ eos_token }}z{% for message in messages %}{{message['role'].capitalize() + ': ' + message['content'] + ' '}}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}c$\rSrSr%Sr\\S'Srg)OnlineTrainerStateirepisodervN)rjrkrlrmrrprorurvr?r<rrs GSr?rc ^\rSrSr%Sr\"SSS0S9r\\S'\"SSS 0S9r \ \S '\"S SS 0S9r \ \ \S '\"S SS0S9r \ \\S'\"S SS0S9r\ \\S'\"SSS0S9r\\S'\"S SS0S9r\ \\S'\"SSS0S9r\\S'\"SSS0S9r\\S'\"SSS0S9r\\S'\"S SS0S9r\ \S \S!'\"S SS"0S9r\ \\S#'\"S$SS%0S9r\\S&'\"S SS'0S9r\ \\S('\"S)SS*0S9r\\S+'\"S SS,0S9r\ \\S-'\"S SS.0S9r\ \\S/'\"S SS00S9r\ \\S1'\"S SS20S9r\ \\S3'\"S SS40S9r \ \\S5'\"S SS60S9r!\ \\S7'\"S SS80S9r"\ \\S9'\"S:SS;0S9r#\ \S<'U4S=jr$S>r%U=r&$)?OnPolicyConfigia9 Base configuration class for on-policy trainers. This class includes only the parameters that are specific to some on-policy training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: run_name (`str` or `None`, *optional*, defaults to `None`): Name of the run. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of processes to use for processing the dataset. num_mini_batches (`int`, *optional*, defaults to `1`): Number of minibatches to split a batch into. total_episodes (`int` or `None`, *optional*, defaults to `None`): Total number of episodes in the dataset. local_rollout_forward_batch_size (`int`, *optional*, defaults to `64`): Per rank no grad forward pass in the rollout phase. num_sample_generations (`int`, *optional*, defaults to `10`): Number of debugging samples generations (i.e., `generate_completions` calls) throughout training. response_length (`int`, *optional*, defaults to `53`): Length of the response. stop_token (`str` or `None`, *optional*, defaults to `None`): Specifies the stop token to use for text generation. This parameter is mutually exclusive with `stop_token_id`. - `None`: No stop token is applied, unless `stop_token_id` is specified. - `'eos'`: Uses the tokenizer's `eos_token`. stop_token_id (`int` or `None`, *optional*, defaults to `None`): Specifies the ID of the stop token to use for text generation. If `None`, no stop token ID is applied, unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`. temperature (`float`, *optional*, defaults to `0.7`): Sampling temperature. missing_eos_penalty (`float` or `None`, *optional*, defaults to `None`): Penalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value. sft_model_path (`str`, *optional*, defaults to `"EleutherAI/pythia-160m"`): Path to the SFT model. world_size (`int` or `None`, *optional*, defaults to `None`): Number of processes (GPUs) to use for the training. num_total_batches (`int` or `None`, *optional*, defaults to `None`): Number of total batches to train. micro_batch_size (`int` or `None`, *optional*, defaults to `None`): Micro batch size across devices (HF's `per_device_train_batch_size` * `world_size`). local_batch_size (`int` or `None`, *optional*, defaults to `None`): Batch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`). batch_size (`int` or `None`, *optional*, defaults to `None`): Batch size across devices (HF's `per_device_train_batch_size` * `world_size` * `gradient_accumulation_steps`). local_mini_batch_size (`int` or `None`, *optional*, defaults to `None`): Mini batch size per GPU. mini_batch_size (`int` or `None`, *optional*, defaults to `None`): Mini batch size across GPUs. push_to_hub (`bool`, *optional*, defaults to `False`): Whether to push the model to the Hub after training. rhelpzLog every X updates steps. Should be an integer or a float in range `[0,1)`. If smaller than 1, will be interpreted as ratio of total training steps.)defaultmetadata logging_stepsTzZIf True, use gradient checkpointing to save memory at the expense of slower backward pass.gradient_checkpointingNzWhether to use bf16 (mixed) precision instead of 32-bit. Requires Ampere or higher NVIDIA architecture or Intel XPU or using CPU (use_cpu) or Ascend NPU. If not set, it defaults to `True` if `fp16` is not set.bf16zName of the run.run_namez6Number of processes to use for processing the dataset.dataset_num_procrLz,Number of minibatches to split a batch into.num_mini_batchesz(Total number of episodes in the dataset.total_episodes@z3Per rank no grad forward pass in the rollout phase. local_rollout_forward_batch_sizezaNumber of debugging samples generations (i.e., `generate_completions` calls) throughout training.num_sample_generations5zLength of the response.response_lengthzoSpecifies the stop token to use for text generation. This parameter is mutually exclusive with `stop_token_id`.eos stop_tokenzSpecifies the ID of the stop token to use for text generation. If `None`, no stop token ID is applied, unless `stop_token` is specified. This parameter is mutually exclusive with `stop_token`. stop_token_idgffffff?zSampling temperature. temperaturezPenalty applied to the score when the model fails to generate an EOS token. This is useful to encourage to generate completions shorter than the maximum length (`max_new_tokens`). The penalty must be a positive value.missing_eos_penaltyzEleutherAI/pythia-160mzPath to the SFT model.sft_model_pathz3Number of processes (GPUs) to use for the training. world_sizez!Number of total batches to train.num_total_batcheszTMicro batch size across devices (HF's `per_device_train_batch_size` * `world_size`).micro_batch_sizezXBatch size per GPU (HF's `per_device_train_batch_size` * `gradient_accumulation_steps`).local_batch_sizeznBatch size across devices (HF's `per_device_train_batch_size` * `world_size` * `gradient_accumulation_steps`). batch_sizezMini batch size per GPU.local_mini_batch_sizezMini batch size across GPUs.mini_batch_sizeFz4Whether to push the model to the Hub after training. push_to_hubc>URcUR(+O URUl[TU] 5 gr<)rfp16superr=)r; __class__s r<r=OnPolicyConfig.__post_init__rs*'+yy'8Odii  r?)r)'rjrkrlrmrnrrrrorrrr rrqrrprrrrrrr rrrrrrrrrrrrr=ru __classcell__)rs@r<rrsA=@! D M5$) p $D ! ! D(4.$,-Hhsm',RS'hsm"HIc%*DE%NHSM-2OP-$c#( w #C !34OS,1  ,J($) q $M8C=12K,1   ,% (23NC!&OP!J (-=>(x}',pq'hsm',tu'hsm!& . !J ,145,8C=&+89&OXc]PQK   r?rboolscURS5nX )RU5-[R"X!URS9-n[R "USS9R $)as Takes an N-dimensional bool tensor and returns an (N-1)-dimensional tensor of integers giving the position of the first True in each "row". Returns the length of the rows (bools.size(-1)) if no element is True in a given row. Args: bools (`torch.Tensor`): An N-dimensional boolean tensor. dtype (`torch.dtype`, optional): The desired data type of the output tensor. Defaults to `torch.long`. Returns: `torch.Tensor`: An (N-1)-dimensional tensor of integers indicating the position of the first True in each row. If no True value is found in a row, returns the length of the row. rCrr%)r&typerYarangerr8values)rrNrow_len zero_or_indexs r<first_true_indicesrxsR$jjnGvmmE22U\\'_d_k_k5llM 99] + 2 22r?query_responsesr6context_lengthc X:gnURS5UR5- n[XR5n[R "X)S5nU"UUUSSSS9nUR URS5n [USS2US24U:H5S- U-n U U [R"U RS5U RS9U 4RS5U 4$) a Computes the reward logits and the rewards for a given model and query responses. Args: model (`torch.nn.Module`): The model used to compute the reward logits. query_responses (`torch.Tensor`): The tensor containing the query responses. pad_token_id (`int`): The token ID representing the pad token. context_length (`int`): The length of the context in the query responses. Returns: tuple: - `reward_logits` (`torch.Tensor`): The logits for the reward model. - `final_rewards` (`torch.Tensor`): The final rewards for each query response. - `sequence_lengths` (`torch.Tensor`): The lengths of the sequences in the query responses. rLrTF)rFrK position_ids return_dictoutput_hidden_states use_cacherCNr) cumsumr[getattrbase_model_prefixrY masked_fillscore hidden_statesrrr&rsqueeze) r+rr6rrKr lm_backbonerFr reward_logitssequence_lengthss r< get_rewardrs2%4N!((+n.A.A.CCL%!8!89K!!/?AFI %!! FKK 4 4R 89M)/!^_:L*MQ]*]^abbess  LL++A.}7K7K L    '"+  r?cX:gnURS5UR5- n[R"X)S5nU"UUUSSS9$)a Performs a forward pass through the model with the given query responses and pad token ID. Args: model (`torch.nn.Module`): The model to perform the forward pass. query_responses (`torch.Tensor`): The tensor containing the query responses. pad_token_id (`int`): The token ID representing the pad token. Returns: `ModelOutput`: The output of the model, including hidden states. rLrT)rFrKrrr)rr[rYr)r+rr6rKrrFs r<forwardrsZ(%4N!((+n.A.A.CCL!!/?AFI %!!  r?per_device_train_batch_sizerrcSSKn[5RnURnUSSS:wa+XS'USSSS.nU(aS S 0US 'OU(aS S 0US 'O[ US 5(a{[ UR SS5(a[UR R5O[ UR SS5nUb&USSS:XaURXw-SU-SS.5 URXS9tpUR5 U$)aO Prepares the model for training with DeepSpeed (both for stage 2 and 3), configuring the appropriate settings based on the model and batch size. Args: model (`torch.nn.Module`): The model to be prepared for DeepSpeed training. per_device_train_batch_size (`int`): The training batch size per device. fp16 (`bool`, defaults to `False`): Whether to use FP16 precision. bf16 (`bool`, defaults to `False`): Whether to use BF16 precision. Returns: `torch.nn.Module`: The model initialized and configured with DeepSpeed for training. rNzero_optimizationstagertrain_micro_batch_size_per_gpuF)rprescale_gradientswall_clock_breakdownenabledTrrconfig hidden_sizes hidden_sizer)z$zero_optimization.reduce_bucket_sizez4zero_optimization.stage3_param_persistence_thresholdz-zero_optimization.stage3_prefetch_bucket_size)r+r) deepspeedrdeepspeed_plugindeepspeed_configr]rrrrr initializeeval) r+rrrrr config_kwargsrrs r<prepare_deepspeedrs,*')::$55M()'2a7:U67.;<\.]"'$) %.$5M& ! %.$5M& ! 5( # #5<<>>ELL--.U\\=$?  &=9L+Mg+VZ[+[$$@K@YPRU`P`IJ$$5$GIE JJL Lr?r responsescF[X :H5RS5nS/[UR55S- -URS/-n[ R "URSURS9R"U6n[ R"X%U:U5nU$)a Truncates the responses at the first occurrence of the stop token, filling the rest with pad tokens. Args: stop_token_id (`int`): The token ID representing the stop token where truncation occurs. pad_token_id (`int`): The token ID representing the pad token used to fill the truncated responses. responses (`torch.Tensor`): The tensor containing the responses to be truncated. Returns: `torch.Tensor`: The truncated responses tensor with pad tokens filled after the stop token. rCrLr) r unsqueezerXr&rrYrrviewr)rr6r trunc_idxsnew_sizeidxspostprocessed_responsess r<truncate_responsers $I$>?II"MJsc)..*+a/0IOOA4F3GGH << *93C3C D I I8 TD#// *;Ll[ ""r?rqueriesgeneration_configcURSnX:gn[R"X)S5nURUUUSSS9n[R"UR S5n[R "XRSS2US244SS9U4$)a Generates sequences from the language model backbone in a way that does not affect padding tokens. Args: lm_backbone (`torch.nn.Module`): The language model backbone used for generation. queries (`torch.Tensor`): The tensor containing the input queries. pad_token_id (`int`): The token ID representing the pad token. generation_config (`GenerationConfig`): The configuration for the generation process. Returns: tuple: - `generated_sequences` (`torch.Tensor`): The concatenated tensor of input queries and generated sequences. - `logits` (`torch.Tensor`): The logits output from the generation process. rLrT)rFrKrreturn_dict_in_generate output_scoresNr%)rrYrgeneratestackscoresr' sequences) rrr6rrrKrFrlogitss r<rr2s.]]1%N,N!!'?A>I  ! !%, $"F[[ *F 99g//>?0BCD! Lf TTr?rcx/n/nURSn[SXr5H;nXX-n [UU UU5upURU 5 URU 5 M= [ XSSS9n [ USSS9n U R SU RS5SUn U R "S/U RSSQ76SUn X4$)NrrrrCr")rrangerrWr\r)r+rrr6rrlogitssrrqueryquery_responser padded_query_responsespadded_logitsss r<batch_generationrYsOGq!J 1j CA@A!)     "  ~.vD![bcHN488=S=Y=YZ\=]^_j`jk#((Gn.B.B12.FG TN ! 11r? bos_token_idprompt_len_input_ids prompt_tokenschosen_prompt_len_input_ids chosen_tokensrejected_prompt_len_input_idsrejected_tokenscUb{US:Xd XSS:waU/US-US'S/US-US'US:Xd XSS:waU/US-US'S/US-US'US:Xd XSS:waU/US-US'S/US-US'X$U4$)NrrrLrTrv)rrrrrrrs r<add_bos_token_if_neededrzs 1 $ FX8YZ[8\(\1=OaAb0bM, -67S=I`;a5aM1 2 &! +|M_?`ab?c/c1=OaAb0bM, -67S=I`;a5aM1 2 (A -QcAdefAg1g3?.?SeCf2fO. /89s_Md=e7eO3 4  88r? eos_token_idc[US5S:Xd XSS:wa(USRU5 USRS5 [US5S:Xd XSS:wa(USRU5 USRS5 X4$)NrFrrCrKrL)rXrW)rrrs r<add_eos_token_if_neededrs = %&!+|[?YZ\?]/]k")),7&'..q1 ?; '(A-Q\A]^`Aa1a $++L9()003  ))r?rFc[X:H5RS5nS/[UR55S- -URS/-n[ R "URSURS9R"U6n[ R"XU:U5n[ R"[ R"U5XS:S5nXg4$)a Truncates the input tensor from the right side after the first occurrence of the stop token. Args: input_ids (`torch.Tensor`): The tensor containing the responses to be truncated stop_token_id (`int`): The token ID representing the stop token where truncation occurs pad_token_id (`int`): The token ID representing the pad token used to fill the truncated responses Returns: tuple: - `output_ids` (`torch.Tensor`): The truncated responses tensor with pad tokens filled after the stop token - `mask` (`torch.Tensor`): The mask tensor to indicate the padding tokens rCrLrr) rrrXr&rrYrrrr ones_like)rFrr6rrr output_idsrgs r<truncate_rightr#s*$I$>?II"MJsc)..*+a/0IOOA4F3GGH << *93C3C D I I8 TD""9Z.?NJ   U__Y79JA ND  r?cT[5(a[RR5 g[ 5(a[R R5 g[ 5(a[RR5 g[RR5 g)a#Empties the cache of the available torch device. This function checks for the availability of different torch devices (XPU, MLU, NPU, CUDA) and empties the cache of the first available device it finds. If none of the specific devices are available, it defaults to emptying the CUDA cache. N) r!rYxpu empty_cachermlur npurmrvr?r<r&r&s_   ! !   ! !   r?inputsr-cURUSS9nUVs/sHo3RURS5PM sn$s snf)aQ Decodes the input tensor and strips the padding tokens. Args: inputs (`torch.Tensor`): The input tensor to be decoded. tokenizer (`transformers.PreTrainedTokenizerBase`): The tokenizer used to decode the input tensor. Returns: `list[str]`: The list of decoded strings with padding tokens stripped. F)skip_special_tokensrl) batch_decodereplace pad_token)r)r-decodedds r<decode_and_strip_paddingr1s@$$V$GG8? @1IIi))2 . @@ @s%= base_model model_name hub_model_id dataset_nametags wandb_url trainer_nametrainer_citation paper_titlepaper_id comet_urlc d[UUSSUS/UQS9n [R"U 40S[[R "S5R S55_SU_S U_S U_S U_S U_S U _SU_SU_SU_SU _S[S5_S[S5_S[S5_S[S5_S[S5_6n U $)a7 Generate a `ModelCard` from a template. Args: base_model (`str` or `None`): Base model name. model_name (`str`): Model name. hub_model_id (`str`): Hub model ID as `username/model_id`. dataset_name (`str` or `None`): Dataset name. tags (`list[str]`): Tags. wandb_url (`str` or `None`): Weights & Biases run URL. comet_url (`str` or `None`): Comet experiment URL. trainer_name (`str`): Trainer name. trainer_citation (`str` or `None`, defaults to `None`): Trainer citation as a BibTeX entry. paper_title (`str` or `None`, defaults to `None`): Paper title. paper_id (`str` or `None`, defaults to `None`): ArXiv paper ID as `YYMM.NNNNN`. Returns: `ModelCard`: A ModelCard object. transformerslicensegenerated_from_trainer)r2datasets library_namelicencer3r6 template_pathtrlztemplates/lm_model_card.mdr2r3r4r5r7r<r8r9r:r; trl_versiontransformers_versionpytorch_versionrYdatasets_versionrAtokenizers_version tokenizers)rr from_templaterq pkg_resourcesfilesjoinpathr) r2r3r4r5r6r7r8r9r:r;r< card_datacards r<generate_model_cardrRsX#& . . I  " " ---e4==>Z[\     "   "    " *    EN %^4  (  !,! "#<0# D& Kr?c[5(dg[R"5b[R"5R$g)zl If Comet integration is enabled, return the URL of the current Comet experiment; otherwise, return `None`. N)rcomet_mlget_running_experimenturlrvr?r<get_comet_experiment_urlrW's7   &&(4..0444 r?rSrc[5(d [S5e[R"5nUbUR XS9 gg)z If Comet integration is enabled logs a table to the Comet experiment if it is currently running. Args: name (`str`): Table name. table (`pd.DataFrame`): The Pandas DataFrame containing the table to log. zLThe comet-ml is not installed. Please install it first: pip install comet-mlN) tabular_datafilename)rModuleNotFoundErrorrTrU log_table)rSr experiments r<log_table_to_comet_experimentr^4sC   !"pqq002J%?r?rg.cURup#UR5nUVs/sHoUR5PM nnURSS9n[R"X4R S9R S5nXvR S5-U-nURSU5n UVs/sHoURSU5PM n nU RSS9n U S:Hn U R5(a6[U R[R5R55OUn U SS2SU 24nU Vs/sH oUSS2SU 24PM nnU(dU$U/UQ7$s snfs snfs snf)aQ Shift non-zero elements in the mask and corresponding tensors to the left. This function operates on a binary mask and any number of additional tensors with the same dimensions as the mask. For each row, non-zero values are shifted to the leftmost positions. Then, columns that contain only zeros across all rows are truncated from the mask and tensors. Visually, this operation can be represented as follows: ``` [[0, 0, x, x, x, x], -> [[x, x, x, x], [0, x, x, x, 0, 0]] [x, x, x, 0]] ``` Args: mask (`torch.Tensor`): 2D tensor (binary mask) with shape `(N, M)`. *tensors (`torch.Tensor`): One or more 2D tensors with the same shape as `mask`. These tensors will be processed alongside `mask`, with non-zero values shifted and excess zero columns truncated in the same manner. Returns: `torch.Tensor`: Updated binary mask with non-zero values flushed to the left and trailing zero columns removed. `*torch.Tensor` Updated tensors, processed in the same way as the mask. Example: ```python >>> mask = torch.tensor([[0, 0, 1, 1, 1], [0, 1, 1, 0, 0]]) >>> tensor = torch.tensor([[9, 9, 2, 3, 4], [9, 5, 6, 9, 9]]) >>> new_mask, new_tensor = flush_left(mask, tensor) >>> print(new_mask) tensor([[1, 1, 1], [1, 1, 0]]) >>> print(new_tensor) tensor([[2, 3, 4], [5, 6, 0]]) ``` rLr%rrN) rclonerrYrrrgatherrr\rprint8)rgrrM mask_copyrfirst_non_zeroposidx_roll mask_rollrolled_tensorscol_sums empty_colsfirst_empty_col flushed_maskflushed_tensorss r< flush_leftroFsXP ::DA I")*'Qwwy'G*%%!%,N ,,q!1!1 2 < GsE&E3E"cURup#UR5nUVs/sHoUR5PM nn[R"U5nUR SS9n[R "X4R S9RS5nXRS5- U-n URSU 5n UVs/sHoURSU 5PM n nU RSS9n U S:gn U R5(a6[U R[R5R 55OUnU SS2US24nU Vs/sH oUSS2US24PM nnU(dU$U/UQ7$s snfs snfs snf)zk Shift non-zero elements in the mask and corresponding tensors to the right. See `flush_left` for details. rLr%rrN)rr`rYfliplrrrrrrarr\rprrb)rgrrrcrdr flipped_maskrerfrgrhrirjnon_empty_colsfirst_non_empty_colrmrns r< flush_rightrusf ::DA I")*'Qwwy'G*<< *L!((Q(/N ,,q!1!1 2 < a/001>OJ   )/ ))'+>KsE.<E3 E8c TUR[R[R4;at[R"USUR S5S9R S5n[R"UVs/sHn[R"USS9PM sn5nX$- nU$/n[X5HYupg[R"USS9nUR SUR S5S9R S5n URU 5 M[ [R"U5nU$s snf)a? A memory-efficient implementation of the common `log_softmax -> gather` operation. This function is equivalent to the following naive implementation: ```python logps = torch.gather(logits.log_softmax(-1), dim=-1, index=index.unsqueeze(-1)).squeeze(-1) ``` Args: logits (`torch.Tensor`): Logits tensor of shape `(..., num_classes)`. index (`torch.Tensor`): Index tensor of shape `(...)`, specifying the positions to gather from the log-softmax output. Returns: `torch.Tensor`: Gathered log probabilities with the same shape as `index`. rC)r#indexr%) rNrYrfloat64rarrr logsumexprF log_softmaxrW) r rwselected_logitslglogsumexp_valuesper_token_logps row_logits row_labels row_logpsrow_per_token_logpss r<selective_log_softmaxrs&|| u}}55,,v2U__R=PQYYZ\] ;;f'Uf(Cf'UV)<  &)&&8 "J jb9I"+"2"2rAUAUVXAY"2"Z"b"bce"f   " "#6 7'9 ++o6 (Vs7D%r  chunk_sizec~URSSnURSnURSU5n/nURUSS9HRn[R"USS9n[ R "U5U-RS5*nURU5 MT [ R"USS9nURU5$)a Compute the Shannon entropy (in nats) for each row of *logits* in a memory-efficient way. Instead of materializing the full softmax for all rows at once, the logits are flattened to shape (N, num_classes), where N is the product of all leading dimensions. Computation is then performed in chunks of size `chunk_size` along this flattened dimension, reducing peak memory usage. The result is reshaped back to match the input's leading dimensions. Args: logits (`torch.Tensor`): Logits tensor of shape `(..., num_classes)`. Entropy is taken along the last axis; all leading dimensions are preserved in the output. chunk_size (`int`, *optional*, defaults to `128`): Number of rows from the flattened logits to process per iteration. Smaller values reduce memory usage at the cost of more iterations. Returns: `torch.Tensor`: Entropy values with shape `logits.shape[:-1]`. NrCrr%) rreshapesplitrzr{rYrrrWr') r roriginal_shape num_classes flat_logits entropieschunklogps chunk_entropys r<entropy_from_logitsrs*\\#2&N,,r"K..[1KI"":1"5 e,))E*U277;; '6  )+I   ^ ,,r?rS completionsrHrIstep num_samplesc [5(d [S5e[5n[SSSS9nUR SSS9 UR SS S9 UR 5HnUR US S S 9 M UR S SS S 9 UbU[ U5:aSnOUS::agUb[R"[[ U55U5n U V s/sHoU PM nn U V s/sHoU PM nn UR5V V V s0sHupXV s/sHoU PM sn _M nn n n U V s/sHoU PM nn [[ U55Hjn UR 5V s/sH oU U SPM n n UR"[X 5[X5/U QX:SP76 UR5 Ml [USSU3SS9nURU5 gs sn fs sn fs sn fs sn n n fs sn fs sn f)u Print out a sample of model completions to the console with multiple reward metrics. This function creates a nicely formatted table showing prompt-completion pairs, useful for monitoring model outputs during training. It requires the `rich` library to be installed. Args: prompts (`list[str]`): List of prompts. completions (`list[str]`): List of completions corresponding to the prompts. rewards (`dict[str, list[float]]`): Dictionary where keys are reward names and values are lists of rewards. advantages (`list[float]`): List of advantages corresponding to the prompts and completions. step (`int`): Current training step number, used in the output title. num_samples (`int` or `None`, *optional*, defaults to `None`): Number of random samples to display. If `None` (default), all items will be displayed. Example: ```python >>> from trl.trainer.utils import print_prompt_completions_sample >>> prompts = ["The sky is", "The sun is"] >>> completions = [" blue.", " in the sky."] >>> rewards = {"Correctness": [0.123, 0.456], "Format": [0.789, 0.101]} >>> advantages = [0.987, 0.654] >>> print_prompt_completions_sample(prompts, completions, rewards, advantages, 42) ╭──────────────────────────── Step 42 ─────────────────────────────╮ │ ┏━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┓ │ │ ┃ Prompt ┃ Completion ┃ Correctness ┃ Format ┃ Advantage ┃ │ │ ┡━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━┩ │ │ │ The sky is │ blue. │ 0.12 │ 0.79 │ 0.99 │ │ │ ├────────────┼──────────────┼─────────────┼────────┼───────────┤ │ │ │ The sun is │ in the sky. │ 0.46 │ 0.10 │ 0.65 │ │ │ └────────────┴──────────────┴─────────────┴────────┴───────────┘ │ ╰──────────────────────────────────────────────────────────────────╯ ``` zvThe function `print_prompt_completions_sample` requires the `rich` library. Please install it with `pip install rich`.Tz bold white) show_header header_styleexpandPrompt bright_yellow)style Completion bright_greenz bold cyanr)rjustify Advantagez bold magentaNrz.2fFzStep )rtitle border_style)rrr$r&rrrXrandomsampler rMrr' add_sectionr%r)rSrrHrIrrrr reward_nameindicesrkeyval reward_valuespanels r<print_prompt_completions_samplers`    "  iG dd KE X_5 \8||~  KI& [H #g, &K A  --c'l 3[A'./w!1:w//67w!1~w 7BI--/R/hc31AQ11/R-45WmW 5 3w< =D\\^L^cCLOC01^ L d7:&[^(<e}eQ[Q^_bPce ! %uTF^, WE MM%071R5Ms0G))G. G8G3'G86G?/H3G8c`\rSrSrSrSS\S\S\S\S\S \\4 S jjr S r S \4S jr Sr g) RepeatSampleriDau Sampler that repeats the indices of a dataset in a structured manner. Args: data_source (`Sized`): Dataset to sample from. mini_repeat_count (`int`): Number of times to repeat each index per batch. batch_size (`int`, *optional*, defaults to `1`): Number of unique indices per batch. repeat_count (`int`, *optional*, defaults to `1`): Number of times to repeat the full sampling process. shuffle (`bool`, *optional*, defaults to `True`): Whether to shuffle the dataset. seed (`int` or `None`, *optional*, defaults to `None`): Random seed for reproducibility (only affects this sampler). Example: ```python >>> sampler = RepeatSampler(["a", "b", "c", "d", "e", "f", "g"], mini_repeat_count=2, batch_size=3, repeat_count=4) >>> list(sampler) [4, 4, 3, 3, 0, 0, 4, 4, 3, 3, 0, 0, 4, 4, 3, 3, 0, 0, 4, 4, 3, 3, 0, 0, 1, 1, 2, 2, 6, 6, 1, 1, 2, 2, 6, 6, 1, 1, 2, 2, 6, 6, 1, 1, 2, 2, 6, 6] ``` ```txt mini_repeat_count = 3 - - - [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, | 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, | 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, | repeat_count = 2 0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, | 4, 4, 4, 5, 5, 5, 6, 6, 6, 7, 7, 7, | 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 11, 11, ...] | --------- --------- --------- --------- --------- --------- --------- --------- --------- --------- --------- --------- batch_size = 12 ``` N data_sourcemini_repeat_countr repeat_countshuffleseedcXlX lX0lX@l[ U5UlXPlX`lU(a:[R"5Ul UbURRU5 gggr<) rrrrrXrrrrY Generator generator manual_seed)r;rrrrrrs r<rARepeatSampler.__init__use'!2$({+  "__.DN**40  r?c#`# UR(a8[R"URURS9R 5nO[ [UR55n[S[U5UR5Vs/sHo!X"UR-PM nnUVs/sH n[U5UR:XdMUPM" nnUHHn[UR5H,nUH#n[UR5HnUv M M% M. MJ gs snfs snf7f)N)rr) rrYrandpermrrrrrr rXrrr)r;indexesrrrrws r<__iter__RepeatSampler.__iter__s <<nnT%5%5PWWYG5!1!123G>C1c'lTXTcTc=de=d14??23=de'.OgUUt1N5gOE4,,-"E"4#9#9:# ;#. fPs%B D. D$&D.,D) D)AD.rAcURUR-UR-UR-UR-$r<)rrrrr:s r<__len__RepeatSampler.__len__s7  DOO3tFI_I__bfbsbsssr?)rrrrrrrr)rLrLTN) rjrkrlrmrnrrprr rArrrurvr?r<rrDsl.h"111 1  1  1sm1,$*ttr?rc[R"U[R"USS9- S-5n[R"[R"U5)5nXUS- - -n[R"U5$)a Compute the standard deviation of a tensor, ignoring NaNs. This function only supports 1D tensors. Args: tensor (`torch.Tensor`): Input tensor of shape `(N,)`. Returns: `torch.Tensor`: Standard deviation of the tensor, ignoring NaNs. T)keepdimr"rL)rYnanmeanrisnanr)rZvariancers r<nanstdrsa}}fu}}VT'JJqPQH IIu{{6** +E ##H ::h r? tensor_dict num_chunksc[SUR555nURSU-n/n[U5Hn0nUR 5H[upxUb6[ U[ 5(dURS:aXU-US-U-Xg'M>UbURS:XaXU'MWSXg'M] URU5 M U$)a Splits a dictionary of tensors along the first dimension into `num_chunks` equal parts. Example: ```python >>> x = torch.arange(12).reshape(6, 2) >>> y = torch.arange(6).reshape(6, 1) >>> tensor_dict = {"x": x, "y": y} >>> split_tensor_dict(tensor_dict, 3) [ {"x": tensor([[0, 1], [2, 3]]), "y": tensor([[0], [1]])}, {"x": tensor([[4, 5], [6, 7]]), "y": tensor([[2], [3]])}, {"x": tensor([[ 8, 9], [10, 11]]), "y": tensor([[4], [5]])} ] ``` c3.# UH ocMUv M g7fr<rv)rrZs r<r$split_tensor_dict..sX-A6-A rNrL) nextrrr rMr.rrrrW) rr first_tensorrchunksr chunk_dictrrZs r<split_tensor_dictrs&X[-?-?-AXXL##A&*4J F :  &,,.KC!z&$'?'?6;;QR?"(Z1q5J:N"O # q(8"(3"&  /  j! Mr?seq_dictc(^[[SUR5555n[R"U5mS[ [ S[ [ 4U4SjjnUR5VVs0sH up4X2"U5_M snn$s snnf)aM Shuffles all sequence-like values in a dictionary along the first dimension in unison. Example: ```python >>> x = torch.arange(6).reshape(3, 2) >>> y = ["a", "b", "c"] >>> seq_dict = {"x": x, "y": y} >>> shuffle_sequence_dict(seq_dict) {'x': tensor([[2, 3], [0, 1], [4, 5]]), 'y': ['b', 'a', 'c']} ``` c3.# UH ocMUv M g7fr<rv)rrs r<r(shuffle_sequence_dict..sH%6!!%6rrrAc >Ucg[U[R5(aURS:XaU$[U[R5(aURS:aUT$TVs/sHoUPM sn$s snf)NrrL)r.rYrtr)rr permutations r<permute&shuffle_sequence_dict..permutesj 9 a & &166Q;H a & &166Q;[> !)*k!k***s/B)rXrrrYrr rrM)rrrrrrs @r<shuffle_sequence_dictrsz"THX__%6HHIJ..,K+8H%+(8*<+/7nn.> ?.>(#C .> ?? ?s6Bc[R"U5R5(a3[R"[ S5UR UR S9$[R"U[R"U5)5$)a Compute the minimum value of a tensor, ignoring NaNs. This function only supports 1D tensors. Args: tensor (`torch.Tensor`): Input tensor of shape `(N,)`. Returns: `torch.Tensor`: Minimum value of the tensor, ignoring NaNs. Returns NaN if all values are NaN. nanr)rYrallrZrrNrr8rZs r<nanminr[ {{6  ||E%L V]]SS 99VU[[001 22r?c[R"U5R5(a3[R"[ S5UR UR S9$[R"U[R"U5)5$)a Compute the maximum value of a tensor, ignoring NaNs. This function only supports 1D tensors. Args: tensor (`torch.Tensor`): Input tensor of shape `(N,)`. Returns: `torch.Tensor`: Maximum value of the tensor, ignoring NaNs. Returns NaN if all values are NaN. rr)rYrrrZrrNrrrs r<nanmaxr rr?cU$)z Do we really need docs for this?rv)rs r<identityrs Hr?rcJSU;dSU;aU$USRSS9R5nUSn[U5URS5:wa)[ S[U5SURS535e[ [ R"USUSS95n0UESU0E$)z Splits `batch["pixel_values"]` into a list of tensors based on the product of each row in `batch["image_grid_thw"]`, while keeping other entries unchanged. image_grid_thw pixel_valuesrLr%rzMismatch: sum(lengths) = z != pixel_values.size(0) = )prodrrr&r7rrrYr)rlengthsr split_valuess r<split_pixel_values_by_gridrs u$e(C $%**q*188:G(L 7||((++4S\NB]^j^o^opq^r]stuu E.$97JKL 2e 2^\ 22r?cURS5n[U[5(a[R"USS9n0UESU0E$U$)z Opposite of `split_pixel_values_by_grid`. Merges a list of tensors in `batch["pixel_values"]` back into a single tensor along the first dimension. rrr%)rUr.rrrYr')rrmergeds r<unsplit_pixel_values_by_gridr/sF 99^,L,%%[R"UT 5nU)nUR5R5nT U- nUS:a[ ST SUSUS35e[R "U5Sn[R "U5nUS:a Xe*SnSXx'X'-n X X4$)Nrztarget_length (z)) is too small for the protected tokens (z4 tokens). Please increase target length to at least z or disable truncation.T)rYisinrrr7where zeros_like) rfrg is_protectedis_non_protected num_protectednum_non_protected_needednon_protected_indiceskeep_non_protected keep_indices keep_maskprotected_tokens_tensorrs r<process_sequence8truncate_with_protected_tokens..process_sequenceQszz#'>? (=%((*//1 #0=#@ #a '!-0YZgYhi==JOKbd  !& ,< =a @"--.>? #a '01J1KLL/3  ,!5 ~t..r?r) setrYrZrrrr rrWr) rfrgrr protected_setr truncated_seqtruncated_maskrnew_idsnew_maskrs ` @r<truncate_with_protected_tokensr=s ()M#ll4 +>szzR/8MN 399Q< ,SVTW=W%h'! ;;} %u{{>'B BBr?)rrN)Ncpu)rC)rl))FF)rAN)NNNN)r<)rimportlib.resources resourcesrMrr collectionsrcollections.abcrrrrimportlib.metadatartypingr r r r numpyrpandaspdrYtorch.nn.functionalr/ functionalrztorch.utils.data accelerater rraccelerate.staterhuggingface_hubrrtorch.nn.utils.rnnrrr>rrrrrrrtransformers.utilsrrrrr r!trainer.model_configr# rich.consoler$ rich.panelr% rich.tabler& rich.textr'rTpeftr(r) get_loggerrjrr+rxrrrtrprqr\rrrrrrsrr r*Moduler2r8r:r_rjrprrr DataFramerSIMPLE_SFT_CHAT_TEMPLATESIMPLE_CHAT_TEMPLATErrr[rrrrrrrrrrr#r&r1rRrWr^rorurrrrrrrrrrrrrrvr?r<r%s + +(&00 99-4+$/$  +   H % _ _  _ D JJ J^(, ? %,, ???! ?  \\ ?D PP Pf 7@7@ 7@t5:G\\G 5<<s *+GG(%"%"4U 3C%"P  %,,    c5j@Q  X[  ejeqeq  EHHOO  &k&kR7 AS8T&Xd38n5 0F2f"2  $ rC   z &z  z z38**3ell33./ 88??/-2\\/IL/^a/ 5<<u|| 34/d 88??\\ B`e7 88??79<7DH7X\7 XX__7t#S## #Y^YeYe#.$U$U+0<<$UGJ$U_o$U 5<< %&$UN2 88??2 \\2'*2 2 ( 22@93-99T#Y'9"% 9 T#Y' 9 $' 9#tCy.)9, * *&*3S >&: *MQRUW[\_W`R`Ma *||,/?B 5<< %&:!$AU\\A>UAZ^_bZcA4'+!%"#G GGG3- G s) G } GGsmG#GsmG}GGT (3- @@BLL@T@$>*U\\>*U\\>*eELLRWX]XdXdfiXiRjDj>k>*B*ell*ell*uU\\SXY^YeYegjYjSkEk?l*< ELL F"- "-#"- "-V T #YTcT#tE{" #TU T  T  T Tn]tG]tB 5<< ELL $ c8ELL112 @C  $sHU\\** +, F@Dhx.@)@$A@d3PXYaPbKbFc@@ 35<< 3ELL 3 35<< 3ELL 3 3d3 +<&=3$sERWR^R^`dejeqeq`rRrLsGsBt3$ S% d5<