h{RSSKrSSKrSSKJr SSKJr SSKJr SSKJ r SSK J r J r J r JrJr SSKrSSKJr SSKrSSKJrJr SSKJrJr SS KJrJrJrJrJrJrJ r J!r!J"r"J#r#J$r$ SS K%J&r& SS K'J(r( SS K)J*r* SS K+J,r, SSK-J.r.J/r/J0r0J1r1J2r2J3r3J4r4 SSK5J6r6J7r7J8r8 SSK9J:r: SSK;Jr>J?r?J@r@JArA \,"5(aSSKBJCrCJDrD \$"5(aSSKErE\R"\G5rH\"S\I\5rJS\JS\J4SjrK\"SS\&55rL\"SS\&55rMSrN"SS\"5rOg) N) defaultdict)Mapping) dataclass)Path)AnyCallableOptionalTypeVarUnion) PartialStatelogging)DatasetIterableDataset) AutoConfig AutoProcessorBaseImageProcessor DataCollatorFeatureExtractionMixinPreTrainedModelPreTrainedTokenizerBaseProcessorMixinTrainerTrainingArgumentsis_wandb_available)DataCollatorMixin)TrainerCallback)EvalPrediction)is_peft_available)apply_chat_templateis_conversationalis_conversational_from_valuemaybe_convert_to_chatml pack_datasetprepare_multimodal_messagestruncate_dataset)clone_chat_templateget_act_offloading_ctx_managerprepare_peft_model) SFTConfig)entropy_from_logits flush_leftgenerate_model_cardget_comet_experiment_urlpadselective_log_softmax) PeftConfig PeftModelTListOrMappingexamplereturnc [U[5(a9UVs/sH+n[U[[45(a [U5OUPM- sn$[U[5(aQUR 5VVs0sH3up!UcM U[U[[45(a [U5OU_M5 snn$[ S5es snfs snnf)a Recursively removes entries with `None` values from a nested structure (list or dictionary). Args: example (`list` or `Mapping`): Input nested structure (list or dictionary) from which to remove `None`. Example: ```python >>> [ ... { ... "a": {"aa": None, "ab": 1}, ... "b": "my_string", ... } ... ] >>> remove_none_values(example) [{'a': {'ab': 1}, 'b': 'my_string'}] ``` z%Input must be a list or a dictionary.) isinstancelistdictremove_none_valuesritems TypeError)r5valuekeys Q/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/sft_trainer.pyr;r;Ns('4  elmel\aZd|-L-L"5)RWWelmm GW % %&mmo -  YCjt .M.M#E*SX X-  ?@@n s2C7 C-Cc\rSrSr%Sr\\S'Sr\\S'Sr \\S'Sr \\S'S r \ \\S 'S r \\S 'S \\\\\\\\44S\\\44Sjr\S\\\S\\R,4Sj5rSrg )DataCollatorForLanguageModelingna Data collator used for language modeling data. Inputs are dynamically padded to the maximum length of a batch. This collator expects each example in the input list to be a dictionary containing at least the `"input_ids"` key. If the input contains a `"completion_mask"`, it is used to set the labels to `-100` for tokens that are not in the completion. If `"assistant_masks"` are present, they are used to set the labels to `-100` for tokens that are not in the assistant part of the sequence. The collator returns a dictionary containing the following keys: - `"input_ids"`: Tensor of input IDs, padded to the maximum length of the batch. - `"attention_mask"`: Tensor of attention mask, padded to the maximum length of the batch. - `"position_ids"`: Tensor of position IDs, padded to the maximum length of the batch. - `"labels"`: Tensor of labels, padded to the maximum length of the batch. If `completion_only_loss` is set to `True`, tokens that are not in the completion are set to -100. If `assistant_masks` are present, tokens that are not in the assistant part of the sequence are set to -100. Args: pad_token_id (`int`): Token ID to use for padding. completion_only_loss (`bool`, *optional*, defaults to `True`): When the input contains a completion mask (`completion_mask`), the labels are set to -100 for the tokens that are no in the completion. padding_free (`bool`, *optional*, defaults to `False`): If set to `True`, the sequences will be flattened into a single sequence, and the position IDs will be generated accordingly. pad_to_multiple_of (`int` or `None`, *optional*, defaults to `None`): If set, the sequences will be padded to a multiple of this value. return_tensors (`str`, *optional*, defaults to `"pt"`): Type of Tensor to return. Only `"pt"` is currently supported. Examples: ```python >>> from trl.trainer.sft_trainer import DataCollatorForLanguageModeling >>> collator = DataCollatorForLanguageModeling(pad_token_id=0) >>> examples = [{"input_ids": [1, 2, 3]}, {"input_ids": [4, 5]}] >>> collator(examples) {'input_ids': tensor([[ 1, 2, 3], [ 4, 5, 0]]), 'attention_mask': tensor([[ 1, 1, 1], [ 1, 1, 0]]), 'position_ids': tensor([[0, 1, 2], [0, 1, 0]]), 'labels': tensor([[ 1, 2, 3], [ 4, 5, -100]])} >>> # With completion mask >>> examples = [ ... {"input_ids": [1, 2, 3], "completion_mask": [0, 1, 1]}, ... {"input_ids": [4, 5], "completion_mask": [0, 1]}, ... ] >>> collator(examples) {'input_ids': tensor([[ 1, 2, 3], [ 4, 5, 0]]), 'attention_mask': tensor([[ 1, 1, 1], [ 1, 1, 0]]), 'position_ids': tensor([[0, 1, 2], [0, 1, 0]]), 'labels': tensor([[-100, 2, 3], [-100, 5, -100]])} >>> # With padding_free >>> collator = DataCollatorForLanguageModeling(pad_token_id=0, padding_free=True) >>> collator(examples) {'input_ids': tensor([[ 1, 2, 3, 4, 5]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]]), 'position_ids': tensor([[0, 1, 2, 0, 1]]), 'labels': tensor([[1, 2, 3, 4, 5]])} ``` pad_token_idTcompletion_only_lossF padding_freereturn_position_idsNpad_to_multiple_ofptreturn_tensorsexamplesr6cVUVs/sHn[R"US5PM nnUR=(a SUS;=(a URnU(d&UVs/sHn[R"U5PM nnUR(a]SUS;a%UR UVs/sHo"SPM sn5nO/WVs/sH"n[R "[U55PM$ nnSUS;a*UVs/sHn[R"US5PM nnO)UVs/sHn[R"US5PM nnUR(a2SUS;a)UVs/sHn[R"US5PM n nSUS;a)UVs/sHn[R"US5PM n n0n UR(a[R"WSS9/nU(d[R"WSS9/nUR(a[R"WSS9/n[R"USS9/nUR(aSUS;a[R"W SS9/n SUS;a[R"W SS9/n [WURSURS 9U S'U(d[WSSURS 9U S 'UR(a[WSSURS 9U S '[US SURS 9U S'UR(a*SUS;a![W SSURS 9n S U SU S:H'SUS;a![W SSURS 9n S U SU S:H'U $s snfs snfs snfs snfs snfs snfs snfs snf) N input_ids seq_lengthsrlabelscompletion_maskassistant_masksdimright) padding_value padding_siderHattention_mask position_ids) torchtensorrGrF ones_like(get_position_ids_from_packed_seq_lengthsarangelenrEcatr0rDrH) selfrKr5rMhas_packed_position_idsrWrXidsrOrPrQoutputs r@ torch_call*DataCollatorForLanguageModeling.torch_callsGOPxGU\\'+"67x P#'":":"q}PXYZP[?["q`d`q`q'JST)Yeooi8)NT  # # +#LL;CD8]+8D  CLL)3 SX 6) L x{ "EMNX'ell78#45XFNFHPQWell7;#78FQ  $ $):hqk)IW_`W_Gu||G4E,FGW_O`  +W_`W_Gu||G4E,FGW_O`   9!45I*"'))N"B!C'' % ,A >? iiA./F((->(1+-M#(99_!#D"E HQK/#(99_!#D"E" ++ #66  { ''*agZ^ZqZq(F# $  # #%(AGX\XoXo&F> " $WQUQhQh x  $ $):hqk)I!qw[_[r[rO6:F8 _1 2  +!qw[_[r[rO6:F8 _1 2 IQU E MNQ``s/#N# N.N )N#N*#N-#N!#N&batch_seq_lengthscUVs/sHn[U5PM nn[R"UVVs/sH oHo3PM M snn5n[R"[U5URS9nSUS'USSS- *X@SSR S5'UR S5n[ URU55$s snfs snnf)a- Get position IDs for packed sequences. Args: batch_seq_lengths (`list[list[int]]`): A list of lists containing the lengths of each individual document in the packed batch. Return: `list[torch.Tensor]`: A list of tensors containing the position IDs for each packed sequence. )dtyperNr*)sumrZr[onesricumsumr9split)rgrNexample_lengths seq_lengthrXs r@r]HDataCollatorForLanguageModeling.get_position_ids_from_packed_seq_lengthss@QQ?P 3{+?PQ!LL+< [+O>U>UV  Q;LSb;QTU;U9V s+22156#**1- L&&788R \s CC )__name__ __module__ __qualname____firstlineno____doc__int__annotations__rEboolrFrGrHr rJstrr9r rr:re staticmethodrZTensorr]__static_attributes__rrr@rBrBnsCJ!%$%L$ $$(, ,NCF4d3id38n.L(M#NFSWX[]`X`SaFP9DcO9X\]b]i]iXj99rrBc0\rSrSr%Sr\\S'Sr\\ \S'Sr \ \S'Sr \\ \S'S r \\S 'S r\\S 'S \\\\ \\\\44S\\\44SjrS \\\\ \\\\44S\\\44SjrS \\\\ \\\\44S\\\44SjrSrg)%DataCollatorForVisionLanguageModelingi u Data collator for vision-language modeling tasks. Unlike text-only datasets—where the collator typically receives pre-tokenized inputs ready for batching, vision-language data processing involves converting images into pixel values. This conversion is disk-intensive, making upfront preprocessing of the entire dataset impractical. Therefore, this collator performs tokenization and image processing on-the-fly to efficiently prepare batches. Each input example should be a dictionary containing at least: - An `"images"` key holding the image data. - [language modeling](#language-modeling) type: either a `"messages"` key for conversational inputs or a `"text"` key for standard text inputs. - [prompt-completion](#prompt-completion) type: keys `"prompt"` and `"completion"` for the prompt and completion. The collator outputs a dictionary including: - `"input_ids"`: Tensor of token IDs. - `"attention_mask"`: Tensor indicating attention mask. - `"pixel_values"`: Tensor representing image pixel values. - `"labels"`: Tensor for training labels. Additional keys may be present depending on the processor, such as `"image_grid_thw"`. Args: processor (`ProcessorMixin`): The processor used to tokenize text and process images. It must be a subclass of `ProcessorMixin` and include a `tokenizer` with a defined `pad_token_id`. max_length (`int` or `None`, optional, defaults to `None`): Maximum sequence length for input tokens. If `None`, no truncation is applied. completion_only_loss (`bool`, *optional*, defaults to `False`): Whether to compute loss only on the completion part of the sequence. When `True`, the labels for the prompt part are set to -100. It requires the dataset type to be prompt-completion. pad_to_multiple_of (`int` or `None`, optional, defaults to `None`): If set, the sequences will be padded to a multiple of this value. dataset_text_field (`str`, optional, defaults to `"text"`): Name of the column that contains text data in the dataset. This parameter is only relevant for [standard datasets format](dataset_formats#standard). return_tensors (`str`, optional, defaults to `"pt"`): The tensor type to return. Currently, only `"pt"` (PyTorch tensors) is supported. Example: ```python >>> from trl.trainer.sft_trainer import DataCollatorForVisionLanguageModeling >>> from transformers import AutoProcessor >>> processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct") >>> collator = DataCollatorForVisionLanguageModeling(processor) >>> examples = [ ... {"images": [Image.open("image_0.png")], "messages": [{"role": "user", "content": "What is this?"}]}, ... {"images": [Image.open("image_1.png")], "messages": [{"role": "user", "content": "Describe this image."}]}, ... ] >>> collator(examples) {'input_ids': tensor([[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374, 419, 30, 151645, 198], [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419, 2168, 13, 151645, 198]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]), 'pixel_values': tensor([[-0.9893, 0.1785, 1.5362, ..., -0.0582, 0.8661, -0.2431], [-0.2302, 0.9522, -1.1061, ..., 0.0555, 1.3354, -0.6412], [ 1.2150, 0.9084, 0.7041, ..., 0.2404, -0.8403, -0.5133], ..., [ 0.6895, 0.2807, 0.2515, ..., -0.2004, -1.2100, 0.0555], [ 0.8209, -0.9748, 1.5654, ..., 1.6055, -0.4706, 0.5817], [-1.0915, 0.4559, 0.9230, ..., 0.5106, 0.0982, -0.1720]]), 'image_grid_thw': tensor([[1, 4, 4], [1, 4, 4]]), 'labels': tensor([[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 3838, 374, 419, 30, 151645, 198], [151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 151652, 151655, 151655, 151655, 151655, 151653, 74785, 419, 2168, 13, 151645, 198]])} ``` processorN max_lengthFrErHtextdataset_text_fieldrIrJrKr6c,SUS;dURUS;a-UR(a [S5eURU5$SUS;aSUS;aUR U5$[ S[ USR55S35e)NmessagesrzTThe `completion_only_loss` argument is not supported for language modeling datasets.prompt completionz#Unexpected input keys in examples: .)rrE ValueError_collate_language_modeling_collate_prompt_completionKeyErrorr9keys)rarKs r@re0DataCollatorForVisionLanguageModeling.torch_callvs ! $(?(?8A;(N(( j228< < ! $!)D228< <@hqkFVFVFXAY@ZZ[\] ]rc DUVs/sHo"SPM nnSUS;aTUHn[US[US55 M UVs/sHo"SPM nnURRU5nO=URUS;aUVs/sHo"URPM nnO [ S5eURUUSSUR URSLURURSS9 nUS R5nS XvS S:H'XvS 'U$s snfs snfs snf) NimagesrrzfThe input examples must contain either 'messages' for conversational data or 'text' for standard data.TrTF) rrpaddingrVrH truncationrrJadd_special_tokensrMrYrWrO) r%r_rr rrrHrrJclone)rarKr5rrtextsrdrOs r@r@DataCollatorForVisionLanguageModeling._collate_language_modelingsH3;<8(#8< ! $#+GJ,?WXEVAWX$;CD8 +8HDNN66x@E  $ $ 3EMNX'T445XENE   #66d2..$   $**,04&'1,-"x A= EOsDD Dc URb [S5eUVs/sHo"SPM nn[US5(aOUH$n[USUS-[ US55 M& UVs/sHn[ X R 5PM nnUVs/sHo"SPM nnUVs/sHo"SPM nnUR UUSSURSS 9nUR USS URSS 9nUS US pUS US p[R"X4SS9n [R"X4SS9n [R"[R"U 5U 4SS9n [XU 5upn URb?U SS2SUR24n U SS2SUR24n U SS2SUR24n U R5nSXS:H'UR(aSXS:H'UnXS 'XS 'XS'U$s snfs snfs snfs snf)NztPadding to a multiple of a value is not yet implemented for vision-language modeling and prompt-completion data yet.rrrrTleftF)rrrrVrJrrT)rrrVrJrrMrWr*rRrYrO)rHNotImplementedErrorr!r%r_r rrJrZr` zeros_liker-rrrE)rarKr5rprompts completionsprocessed_promptsprocessed_completions prompt_idscompletion_ids prompt_maskrPrMrWrOrds r@r@DataCollatorForVisionLanguageModeling._collate_prompt_completionsa  " " .%. 4<<8(#8< Xa[ ) )#+GH,= @U,UWZ[bck[lWmn$T\]T\+G^^DT\H]4<=H8$H=\8]3^!cghkmphpcq!F:4d3idSVX[S[n>\8]3^:cghkmphpcq:rrc [RRUSSS9nUSSS24R5nUS:gnSX4)'[ UR U5nUR 5R5*U-nXd-R5U- nU$)z DFT loss function, as presented in [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://huggingface.co/papers/2508.05629) )rr*rY)r>.r*Nr) nn functionalr0 contiguousr1logitsexpdetachrk)outputsrOnum_items_in_batch shift_labels loss_masklogprobsper_token_losslosss r@dft_lossrs ]]  vvT  :F#qr'?--/L$I L$W^^\BHlln++--8N  & + + -0B BD Krc B^\rSrSrSrSS/rS&S\\\R\ 4S\ \\ \ 4S\ \S \ \\\4S \ \\\\\44S \ \\\4S \ \S \ \\/\4S\ \\S\\ \R4R6\ \R4R8R:4S\ \\\R4R6\\\44S\ \\R@\R@/\R@4S\ SS\ \\/\44U4Sjjjr!S\\\4S \\\"\#\4S\ S\$S\ \\/\4S\S\\\44Sjr%Sr&S'U4Sjjr'U4Sjr(S(S\\\)4S\ \)SS4U4S jjjr*U4S!jr+S)S"\ \S\ \S#\\\\S44S$jjr,S%r-U=r.$)* SFTTraineriaZ Trainer for Supervised Fine-Tuning (SFT) method. This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods. Example: ```python from datasets import load_dataset from trl import SFTTrainer dataset = load_dataset("roneneldan/TinyStories", split="train[:1%]") trainer = SFTTrainer(model="Qwen/Qwen2-0.5B-Instruct", train_dataset=dataset) trainer.train() ``` Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using `.from_pretrained` (where `` is derived from the model config) with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. If you're training a model with an MoE architecture and want to include the load balancing/auxilliary loss as a part of the final loss, remember to set the `output_router_logits` config of the model to `True`. args ([`SFTConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. data_collator ([`~transformers.DataCollator`] or `None`, *optional*): Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`. Will default to [`~trainer.sft_trainer.DataCollatorForLanguageModeling`] if the model is a language model and [`~trainer.sft_trainer.DataCollatorForVisionLanguageModeling`] if the model is a vision-language model. train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]): Dataset to use for training. SFT supports both [language modeling](#language-modeling) type and [prompt-completion](#prompt-completion) type. The format of the samples can be either: - [Standard](dataset_formats#standard): Each sample contains plain text. - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role and content). The trainer also supports processed datasets (tokenized) as long as they contain an `input_ids` field. eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Union[Dataset, IterableDataset]]`): Dataset to use for evaluation. It must meet the same requirements as `train_dataset`. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`] or `None`, *optional*, defaults to `None`): Processing class used to process the data. If `None`, the processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default. compute_loss_func (`Callable` or `None`, *optional*, defaults to `None`): A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618) used by [`Trainer`]. compute_metrics (`Callable[[EvalPrediction], dict]` or `None`, *optional*, defaults to `None`): The function that will be used to compute metrics at evaluation. Must take a [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing [`SFTConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the function needs to calculate and return the global summary statistics rather than accumulating the batch-level statistics. callbacks (list of [`~transformers.TrainerCallback`] or `None`, *optional*, defaults to `None`): List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](https://huggingface.co/docs/transformers/main_classes/callback). If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`] method. optimizers (`tuple[Optional[torch.optim.Optimizer], Optional[torch.optim.lr_scheduler.LambdaLR]]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`. optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*, defaults to `None`): A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in `args`. Incompatible with the `optimizers` argument. Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before initializing the Trainer. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*, defaults to `None`): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. peft_config ([`~peft.PeftConfig`] or `None`, *optional*, defaults to `None`): PEFT configuration used to wrap the model. If `None`, the model is not wrapped. formatting_func (`Callable` or `None`, *optional*, defaults to `None`): Formatting function applied to the dataset before tokenization. Applying the formatting function explicitly converts the dataset into a [language modeling](#language-modeling) type. trlsftNmodelargs data_collator train_dataset eval_datasetprocessing_classcompute_loss_funccompute_metrics callbacks optimizersoptimizer_cls_and_kwargspreprocess_logits_for_metrics peft_configr2formatting_funcc>UcP[U[5(aUOURRnUR S5Sn[ US35nOe[U[ 5(aP[U[ 5(d;UR5nURUS'URS5 [ S>0UD6nUR=(d 0n[U[5(aUnURS5n[U[R5(d US:XdUcO@[U[5(aUS;a[[U5nUUS'O[S US 35e[ R""U5n[[$UR&S 5nUR""U40UD6nO8URRnURb[(R+S 5 Uc[,R""U5n[U[.5(aUR0nS UlO*[U[45(a UnSUlO [7S5eUR8bMUR8nUR;U5nUc&[SUSUR<R>S35eUUl URBb[DRFRIURB5(aUURBRKS5(a5[MURBSS9nURO5Ul(SSS5 /nO[SXURB5upnO/nUR2(aURT(a [S5eUR2(aURV(a [S5eUR2(aURX(a [S5eU bU(aU RZc SU0U l-O>SU RZ;aUU RZS'OU RZSR]U5 U R^bSU R^;aF[(R+S5 U R^c S/U l/OU R^RaS5 S Ul1U c$[e5(af[U[f5(aQ[iXU5nURjURl;a+URlURjn[USS 5Ul1URV=(d" URT=(a URnS:HUl+URRpS;nURV(aUb [S5eURT(a%URnS :Xa[(R+S!5 U(d[(R+S"5 URrS#:Xa&URT(d[(R+S$5 [u[wU55nURxcS%U;=(a S&U;Ul<OURxUl<UcUR2(dURz=(d URz=(d UR8nUR;U5nUc&[S'USUR<R>S(35e[}UURxURVUUR~S)9nOIUcFUR2(a5[UURURxUR~URS*9nURT(a,URnS:XaU(d[(R+S+5 URX(a5(d [S,5eURSL=(a URRS-S5=(d UR2n U (dURx(aU(a [S.5eURXFX"RTUS/5nUbURc URTO URn![U[5(a:UR5V"V#s0sHun"n#U"URU#XbU!UU"5_M nn"n#OURXVUU!US05nURS1:XaO>URS2:XaUb [S35e[nO[S4URS535e[[5[[5S6.UlMS UlN[T$UG]AUUUUUUUUU U U U S79 URR(a[URS89UlUO[R"5UlU[URS95(a%URRUR5 [URS:S5Ul[[URS;S<5Ul\UR(a'URS<:Xa[(R+S=5 ggg!,(df  GNA=fs sn#n"f)?N/rjz-SFT hub_tokenpush_to_hub_tokenriauto)bfloat16float16float32zInvalid `dtype` passed to `SFTConfig`. Expected either 'auto' or a string representing a valid `torch.dtype` (e.g., 'float32'), but got rrzYou passed `model_init_kwargs` to the `SFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.TFzWThe `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`zThe specified `eos_token` ('zC') is not found in the vocabulary of the given `processing_class` (zX). Ensure that the `eos_token` exists in the vocabulary before using it as an EOS token.)z.jinjaz.j2zutf-8)encodingzaPacking is not supported for vision-language models. Please set `packing=False` in the SFTConfig.zzPadding-free training is yet not supported for vision-language models. Please set `padding_free=False` in the `SFTConfig`.zAssistant-only loss is not yet supported for vision-language models. Please set `assistant_only_loss=False` in the `SFTConfig`. embed_tokenslm_heada-Cloning chat template added new tokens to the tokenizer, but 'lm_head' is not in PEFT's `modules_to_save`. As a result, the model may not learn to generate outputs with these new tokens, leading to degraded generation quality. To fix this, add `modules_to_save=['lm_head']` to your PEFT configuration.num_virtual_tokensbfd)flash_attention_2flash_attention_3z"kernels-community/vllm-flash-attn3zHPassing a custom data collator is not supported when using padding-free.wrappedzYou are passing `padding_free=True` with the 'wrapped' packing strategy, which is not recommended. Please refer to the documentation to understand why this is not recommended.aPadding-free training is enabled, but the attention implementation is not set to 'flash_attention_2'. Padding-free training flattens batches into a single sequence, and 'flash_attention_2' is the only known attention mechanism that reliably supports this. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation='flash_attention_2'` in the model configuration, or verify that your attention mechanism can handle flattened sequences.r*zYou are using a per_device_train_batch_size of 1 with padding-free training. Using a batch size of 1 anihilate the benefits of padding-free training. Please consider increasing the batch size to at least 2.rrzThe specified `pad_token` ('z[). Ensure that the `pad_token` exists in the vocabulary before using it as a padding token.)rDrErFrGrH)rrrErHra$You are using packing, but the attention implementation is not set to 'flash_attention_2' or 'kernels-community/vllm-flash-attn3'. Packing flattens batches into a single sequence, and Flash Attention is the only known attention mechanisms that reliably support this. Using other implementations may lead to cross-contamination between batches. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation='flash_attention_2'` or `attn_implementation='kernels-community/vllm-flash-attn3'` in the model configuration.zYou set `assistant_only_loss=True`, but the dataset is not conversational. This option is only supported for conversational datasets.skip_prepare_datasetaEA formatting function was provided while `completion_only_loss=True`, which is incompatible. Using a formatter converts the dataset to a language modeling type, conflicting with completion-only loss. To resolve this, apply your formatting function before passing the dataset, or disable `completion_only_loss` in `SFTConfig`.trainevalnlldftzYou passed a `compute_loss_func` together with `loss_type='dft'` to the `SFTTrainer`. When using `loss_type='dft'`, the loss function is internally set to the DFT loss, so passing a `compute_loss_func` is not allowed.zInvalid `loss_type` z. passed. Supported values are 'nll' and 'dft'.)rr) rrrrrrrrrrrr)radd_model_tagsoutput_router_logitsrouter_aux_loss_coefa-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.rr)]r8r{config _name_or_pathrnr+rto_dictrpopmodel_init_kwargsgetrZrigetattrrrfrom_pretrained transformers architecturesloggerwarningrr tokenizer_is_vlmrr= eos_tokenconvert_tokens_to_ids __class__rs eos_token_idchat_template_pathospathisfileendswithopenread chat_templater'packingrFassistant_only_losstrainable_token_indicesextendmodules_to_saveappendrrr3r)active_adapterrpacking_strategy_attn_implementationper_device_train_batch_sizenextiterrE pad_tokenrBrHrrrr!dataset_kwargs_prepare_dataset eval_packingr:r< loss_typerrr9_metrics_total_train_tokenssuper__init__ractivation_offloadingr(r maybe_activation_offload_context contextlib nullcontexthasattrr _tag_namesaux_loss_enabled aux_loss_coef)%rarrrrrrrrrrrrrr model_name dict_argsrmodel_idrir architecturerrrchat_template_file added_tokenspeft_model_configuse_flash_attentiondataset_samplerrDrrr?datasetrs% r@r SFTTrainer.__init__Ns] $ <",UC"8"8ell>X>XJ#))#.r2J |401D / 0 0D)9T9T I%)^^Ik " MM- .)y)D!228b eS ! !H%))'2E%--&EME3''E5W,Wu--2!'* HHMwaQ //9F"<1E1Ea1HIL 00O=NOE||11H%%1?  #,<> %I$::9EL# 29+>++;+E+E+N+N*OPII &2I "  " " .ww~~d55664;R;R;[;[\m;n;n$11GDHZ5G5L5L5N$2E! 8KT-D-D95L <;I<:XK7#;+N+NNJVK77G77GNN|\..6);KfKf:fNNT#22:7@k 3#33::9E#$  "'8':':z%QZ?[?[&u4@E##u'8'88$)$5$5e6J6J$K!*12CEY[\*]' !--b$,,2a4CXCX\aCa#ll??D     ( !kll|| 5 5 Bp'J//14T\\%d=12  $ $ ,(0N(B(e|WeGeD %(,(A(AD %  T)*=*=TATATI$::9EL# 29+>++;+E+E+N+N*OPLL <)%)%>%>!..$7#'#:#: M "t||A*??%)%>%>#'#:#:#'#:#: M <,>,@$,@LCT227>U "  ^^u $ , : !) 3DNN3CCqrs s#.d"3[=NO #$  ''%-/+!%=*G  99 * *4RY]YcYc4dD 14>4J4J4LD 1 4::/ 0 0 JJ % %doo 6 ' 6Le T$U\\3I3O  T%7%73%> NN &? CEDJ$si$#i6$ i3rr dataset_namer6c 0^[U[5(aUR[5n[ [ [ U55R55nSU;n0n [U[5(aURU S'[5R5 TbU(a[RS5 Tb@U(d9[U[5(a SUS3U S'U4Sjn UR"U 4SS 0U D6nU(GdB[ [ U55n [U 5(aa[U[5(a S US 3U S'[ [ U55R5nUR"[4S S U;aS OS0U D6n[ [ U55n [!U 5(dK[U[5(a SUS3U S'Sn UR"U 4SUR"0SU;aSOSS.U D6n[U[5(a SUS3U S'Sn UR"U 4SUUR$UR&S.0U D6nU(aUR(c [+S5e[U[5(a SUS3U S'S/nSUR,;aUR/S5 SUR,;aUR/S5 UR1U5n[3XR(UR4U 5nOAUR(b4[U[5(a SUS3U S'[7XR(U 5nUR8(a.1SknUR1UR;UR,55nSSS5 U$!,(df  U$=f)NrMnum_proczYou passed a dataset that is already processed (contains an `input_ids` field) together with a formatting function. Therefore `formatting_func` will be ignored. Either remove the `formatting_func` or pass a dataset that is not already processed.z Applying formatting function to z datasetdescc>ST"U50$)Nrrr)r5rs r@_func*SFTTrainer._prepare_dataset.._funcs"OG$<==rbatchedFz Converting z dataset to ChatMLremove_columns conversationszAdding EOS to cSU;a&USRU5(d USU-US'U$SU;a$USRU5(d USU-US'U$)Nrr)r)r5rs r@add_eos,SFTTrainer._prepare_dataset..add_eossp!W,WV_5M5Mi5X5X.5fo .IGFO '*W4W\=R=[=[\e=f=f4;L4II4UGL1&rrr) fn_kwargsr)z Tokenizing cfSU;Ga0n[U5(aUR"US4SURS50URS05D6nUR"USUS-4SUURS5S.URS05D6nUSnSU;aUSUS'OU"USS 9SnU"USUS-S 9SnUS[U5U:Xd[R S 5 S /[U5-S /[U5[U5- --nXtS'XS 'U$[U5(aqUR"US4SUURS5S.URS05D6n SU ;aS U S;a [ S5eSV s0sHoU ;dM XU _M nn U$SU"XS 9S0nU$s sn f)Nrtoolschat_template_kwargsrT) return_dictreturn_assistant_tokens_maskr0rMrQ)rzMismatch between tokenized prompt and the start of tokenized prompt+completion. This may be due to unexpected tokenizer behavior, whitespace issues, or special token handling. Verify that the tokenizer is processing text consistently.rr*rPru?You're using `assistant_only_loss=True`, but at least one example has no assistant tokens. This usually means the tokenizer's chat template doesn't generate assistant masks — it may be missing the `{% generation %}` keyword. Please check the template and ensure it's correctly configured to support assistant masking.)rMrQ)r!r rr_rr RuntimeError) r5rrrrdrprompt_completion_processedprompt_completion_idsrP processedks r@tokenize-SFTTrainer._prepare_dataset..tokenizesP7*!#,W55)9)M)M ' 1*&-kk'&:*#*++.Db"I*J ;K:^:^ ' 1GL4I I;,0=P&-kk'&: ; #*++.Db"I ;75PP[4\104OOrMrNrQrP)r8rwith_transformr;r9rrrdataset_num_procr main_process_firstrrmapr"r#r!rrrrr column_namesrselect_columnsr$rr&use_liger_kernel intersection)rarrrrrr!r? is_processed map_kwargsr& first_exampler,r9columnscollator_expected_keyss ` r@rSFTTrainer._prepare_datasetsu gw ' ',,-?@GDg/4467 "l2  gw ' '%)%:%:Jz " ^ . . 0*|Y *<gw//+KL>Ya)bJv&>"++eIUIjI $T'] 3 / >>!'733/:<.HZ-[ 6*#'W #6#;#;#=L%kk/:I\:Y_c%G!%T'] 3 (77!'733/=l^8-T 6*'&kk#.0@0J0J"K5?<5OzUY% Ggw//+6|nH)MJv&:"x"++-=.2.E.E/3/G/G!??*$%]^^gw//+3L>)JJv&&-$(<(<<NN#45$(<(<<NN#45!009'wAVAVXbc,gw//+6|nH)MJv&*7OOZP$$)k&!001G1T1TU\UiUi1jkk1no1 0ns K"N NchURc%UR(a /SQUlg/SQUlgg)N)rrrr)rMrOrNrPrQ)_signature_columnsr)ras r@ _set_signature_columns_if_needed+SFTTrainer._set_signature_columns_if_needed8s,  " " *||*X'*v' +rc >URR(aSOSnUSnSUS'[TU] XSUS9upxUR(a.UR (aUR UR -n Xy- nURR(Gd7[R"5 [UR5n SU;a~USn [R"U RS 5URU R S 9n [R""X4S S 9n [R$"X-5U R%5- n O(S U;a[R&"U 5n O [)S5eUR*R-U 5R'5R/5n SSS5 UR0USR3W 5 US:XaSU;aIUR*R-USR%55R%5R/5nO}S U;al[R4"US RS 5US R S 9nUR*R-U5R%5R/5nO [)S5eU=R6U- slUR6/UR0US'URR(Gd[R"5 SU;a URR95nUSnO;URSSS2SS24R95nUSS S24R95nUSS2URS2SS24nUR;SS 9nUS:gnUU:HU-nUR%5nUR%5nUR*R-U5nUR*R-U5nUR%5nUS :a!UR%5U- R/5OSnUR0USR3U5 UR(aXUR*R-W 5R'5R/5n UR0USR3U 5 SSS5 U(aXx4$U$!,(df  GN(=f!,(df  N,=f)zA Compute training loss and additionally compute token accuracies rrrOF use_cacheT)return_outputsrrWr)devicer*rRrXz6Expected 'attention_mask' or 'position_ids' in inputs.Nentropy num_tokensr.rjrYrmean_token_accuracyaux_loss)rtrainingr  compute_lossrrrTrrArZno_gradr,rrlsizerrPr`rkmeanr acceleratorgather_for_metricsitemr rr[r rargmax)rarinputsrOrmoderOrrrTper_token_entropyrWvirtual_attention_maskrQnum_tokens_in_batchlocal_num_tokens shift_logitsr predictionsmaskcorrect_predictions total_tokenscorrect_tokens total_sumaccuracyrs r@rVSFTTrainer.compute_lossCsL**--w6!${'. $CU/   T%7%7))G,<,<>#<#<#>L#).#9L#*>>#ssA+#>#I#I#KL#)#qr'?#=#=#?L ,At/F/F/H!,KL +11b19 $t+(3l'Bd&J##xxz !4!8!8!:"&!1!1!D!D^!T#//BB<P ),,. HQTU N..09<BBD[^ d#$9:AA(K((#//BB8LQQSXXZHMM$' 3::8DI!L#1:d:M!@!s,C9SF/S S S&cr>UR [TU]"U0UD6sSSS5 $!,(df  g=fN)rr  training_step)rarkwargsrs r@roSFTTrainer.training_steps*  2 27($9&93 2 2s( 6logs start_timec >URR(aSOSnURUR5VVs0sHupEU[ U5[ U5- _M nnnUS:Xa(UR5VVs0sH upESU3U_M nnnUR U5 [TU]!X5 URUR5 gs snnfs snnf)Nrreval_) rrUr r<rkr_updater logclear)rarrrsr_r?valmetricsrs r@rwSFTTrainer.logs**--w6<@MM$:A--/J/hcse}c)/GJ G  D% d!!#Y Ks $CCc>URRc*[URR5RnO(URRR S5SnUR US9 [TU]!X5 g)Nrrj)r) r hub_model_idr output_dirnamerncreate_model_cardr _save_checkpoint)rartrialrrs r@rSFTTrainer._save_checkpointsj 99 ! ! )dii22388J//55c:2>J *5  .rrtagsc UR5(dg[URRS5(ac[R R URRR5(d!URRRnOSnUc [5nO$[U[5(aU1nO [U5n[URRS5(aURS5 S[R;aURS5 URUR5 [UUUR U[#U5[%5(a+[&R(b[&R(R*OS[-5SS9nUR/[R R1UR2R4S 55 g) a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. Nrunsloth_versionunslothJOB_IDhf_jobsSFT) base_modelrr}r!r wandb_url comet_url trainer_namez README.md)is_world_process_zerorrrrrisdirrsetr8r{addenvironrvrr.r}r9rwandbrunurlr/savejoinrr~)rarr!rr model_cards r@rSFTTrainer.create_model_cardsR"))++  4::$$o 6 6rww}}TZZM^M^MlMl?m?m**88JJ <5D c " "6Dt9D 4::$$&7 8 8 HHY  rzz ! HHY  DOO$(!!**%d'9';'; @Ueiimm[_.0    TYY%9%9;GHr) rr rJr rrrErrrF) NNNNNNNN)NNNNNN)FNrn)NNN)/rsrtrurvrwrr r{rModulerr r+rrrrr:rrrrr9rtuplerZoptim Optimizer lr_schedulerLambdaLRtyperr}r rrrzrrKrVrofloatrwrrr~ __classcell__)rs@r@rrs XtJ ?C04CGEIUY04FJ59jvaehl.2;?xS"))_45xuY(99:;x - x  g&> ?@ x uWd3<.@%@AB x#5)@.)P#QRx$H-x"(N+;T+A"BCxD12x(5;;#8#898EKKD\D\DeDe;ffgx#+5ekk6K6K1LdSVX[S[n1\+]"^x(0%,, 9UW\WcWc9c0d'exl+x"(D63;"78xxt nw/0n 79KMces stn n  n "(D63;"78 nn w' (n` w];@: $S%Z( $huo $QU $ $/%)&*,0 4ISM4Ism4ICcD() 4I4Irr)Prr collectionsrcollections.abcr dataclassesrpathlibrtypingrrr r r rZtorch.nnrr accelerater r datasetsrrrrrrrrrrrrrtransformers.data.data_collatorrtransformers.trainer_callbackrtransformers.trainer_utilsrtransformers.utilsr data_utilsr r!r"r#r$r%r&modelsr'r(r) sft_configr+utilsr,r-r.r/r0r1peftr2r3r get_loggerrsrr9r4r;rBrrrrrrr@rs ##!:: ,-    >950]\!*   H %)49AA>A@ n9&7n9 n9b ~,=~ ~B @ I@ Ir