hS Z S SK r S SKrS SKJr S SKJrJrJr S SKrS SK J
r
S SK J r S SK
Jr S SKJrJrJrJrJrJrJrJrJrJrJrJrJ r S SK J r S S K J r S
S K!J"r" S S
K#J$r$ S SK%J&r&J'r' \ " 5 ( a S SK(J)r) \ " 5 ( a S SK*r*\
RV " \,5 r- " S S\5 r.g) N)Path)CallableOptionalUnion)logging)Dataset)
DataLoader)
AutoModelForCausalLM
AutoTokenizerBaseImageProcessor DataCollator DataCollatorForLanguageModelingDataCollatorForSeq2SeqFeatureExtractionMixinPreTrainedModelPreTrainedTokenizerBaseProcessorMixinTrainerTrainingArgumentsis_wandb_available)EvalLoopOutput)is_peft_available )
PPODecorators )IterativeSFTConfig)generate_model_cardget_comet_experiment_url) PeftModelc ^ \ rS rSrSrSS/r S!S\\\4 S\ \\
\ 4 S\ \ S \ \\
\\\
4 4 S
\ \\\\\4 S \\R* R, \R* R. R0 4 S \ \\R4 \R4 /\R4 4 S
\ \\/\4 4U 4S jjjr S\S\
S\4S jr S\R4 S\R4 S\R4 4S jr \ S\ \RB S\ \RB S\ \RB S\ \ S\ \ 4
S j5 r"\#RH " 5 S"S\ \ \RB S\ \ \RB S\ \ \RB S\ \ \ S\ \ \ 4
S jj5 r%S r&U 4S jr' S#S \ \ S \ \ S \\\ \ S4 4S jjr(S r)U =r*$ )$IterativeSFTTrainer9 a
The IterativeSFTTrainer can be used to finetune models with methods that requires some steps between optimization.
The [`IterativeSFTTrainer`] is deprecated and will be removed in version 0.24.0. Please use the [`SFTTrainer`].
Args:
model (`Union[str, PreTrainedModel]`):
Model to be trained. Can be either:
- A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a
path to a *directory* containing model weights saved using
[`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded
using [`~transformers.AutoModelForCausalLM.from_pretrained`] with the keyword arguments in
`args.model_init_kwargs`.
- A [`~transformers.PreTrainedModel`] object. Only causal language models are supported.
args ([`IterativeSFTConfig`], *optional*, defaults to `None`):
Configuration for this trainer. If `None`, a default configuration is used.
data_collator (`DataCollator`, *optional*):
Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`.
Will default to [`~transformers.default_data_collator`] if no `processing_class` is provided, an instance
of [`~transformers.DataCollatorWithPadding`] otherwise if the processing_class is a feature extractor or
tokenizer.
eval_dataset (`datasets.Dataset`):
The dataset to use for evaluation.
processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.BaseImageProcessor`], [`~transformers.FeatureExtractionMixin`] or [`~transformers.ProcessorMixin`], *optional*, defaults to `None`):
Processing class used to process the data. If `None`, the processing class is loaded from the model's name
with [`~transformers.AutoTokenizer.from_pretrained`].
optimizers (`tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]`):
The optimizer and scheduler to use for training.
preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`):
The function to use to preprocess the logits before computing the metrics.
compute_metrics (`Callable[[EvalPrediction], dict]`, *optional*):
The function to use to compute the metrics. Must take a `EvalPrediction` and return a dictionary string to
metric values.
trlz
iterative-sftNmodelargs
data_collator eval_datasetprocessing_class
optimizers preprocess_logits_for_metricscompute_metricsc | > [ R " S[ 5 [ U[ 5 ( a UOUR
R n Uc# U R S5 S n
[ U
S35 nOe[ U[ 5 ( aP [ U[ 5 ( d; UR 5 n UR U S' U R S5 [ S0 U D6nUc [ R " U 5 nUR b* [ U[ 5 ( d [ R# S5 [ U[ 5 ( a U R% X5 n[' 5 ( a [ U[( 5 ( a SU l OS U l XPl [/ UR
S
S 5 U l Uc< U R0 ( a [3 US S S
9U l O [7 U R, S S9U l OX0l UR8 U l UR: U l UR< U l [> T U ] UUU R4 UUUUUS9 [C U RD S5 ( a% U RD RG U RH 5 U RK U RL RN 5 U RP RS U RD U RT U RV 5 u U l" U l* U l+ U R: S:X a SOSU R, l, [C U S5 ( d [[ S5 eU R< [\ l g )NzkThe `IterativeSFTTrainer` is deprecated and will be removed in version 0.24.0. Please use the `SFTTrainer`./z
-IterativeSFT hub_tokenpush_to_hub_tokenzYou passed model_init_kwargs to the `IterativeSFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.TFis_encoder_decoder )label_pad_token_idpad_to_multiple_of)mlm)r$ r% r& r' r( r+ r) r* add_model_tagskeep_endleftright acceleratorzXYour `Trainer` does not have an `accelerator` object. Consider upgrading `transformers`. )/warningswarn
FutureWarning
isinstancestrconfig
_name_or_pathsplitr r to_dictr/ popr from_pretrainedmodel_init_kwargsloggerwarning_create_model_from_pathr r
is_peft_modelr( getattrr1 r r& r
max_lengthtruncation_modeoptimize_device_cachesuper__init__hasattrr$ r7
_tag_names create_optimizer_and_schedulerr% max_stepsr; prepare optimizer lr_schedulertruncation_sideAttributeErrorr )
selfr$ r% r& r' r( r) r* r+ model_id
model_name dict_args __class__s
[/home/james-whalen/.local/lib/python3.13/site-packages/trl/trainer/iterative_sft_trainer.pyrR IterativeSFTTrainer.__init__d s
'uc 2 25 8R8R <! ,R0J% M&BCD
/
0
0DJ\9]9] I%)^^Ik " MM- .%2 2D # , < G Hiseooc 2iN H " " ++ *-Y)O)O
# #&L)O
b""#
NN. 5_cJx H!59N9N9[9[![ \ ++KNyKijKixss:Kijb""#
?? &##|3BLBRBRBT UBT$!a#4T__!5 5BT
U %%3CMCSCSCU VCU41aDOO#3#5!6 6CU
V !# S: a URA 5 nURC 5 nT R8 RE U5 T R8 RF ( a_ T R RH bH T R8 RK T R RM 5 T R RH 5 T RN RQ 5 T RN RS 5 T RT b T RT RQ 5 T R =R S-
sl T =R U-
sl T RW 5 SSS5 GM gs snf ! , ( d f GM = f)aP
Run an optimisation step given a list of input_ids, attention_mask, and labels or a list of text and
text_labels.
Args:
input_ids (list[`torch.LongTensor`]):
List of tensors containing the input_ids (if not provided, text will be used)
attention_mask (list[`torch.LongTensor`], , *optional*):
List of tensors containing the attention_mask
labels (list[`torch.FloatTensor`], *optional*):
List of tensors containing the labels (if set to None, will default to input_ids)
texts (list[`str`], *optional*):
List of strings containing the text input (if not provided, input_ids will directly be used)
texts_labels (list[`str`], *optional*):
List of strings containing the text labels (if set to None, will default to text)
Returns:
`dict[str, Any]`: A summary of the training statistics
r g Nz@Step should include `input_ids` or `texts` as keyword arguments.ztBoth `input_ids` and `texts` argument are provided. `input_ids` will be ignored. Please provide only one of the two.zNo 'labels' or 'text_labels' are provided. When using an encoder-decoder architecture, 'labels' or 'text_labels' must be passed.Tpt)rN
truncationpaddingreturn_tensorsrh ri c > [ 5 nU S HY nUS; d M [ R " U Vs/ s H o3U PM sn5 R TR R
5 X' M[ U$ s snf )Nr rl )dictro stackrr r$ rs )data return_dictkeydr\ s ra collator*IterativeSFTTrainer.step..collators sf &KAwCC',{{D3IDqcFD3I'J'M'MdjjN_N_'`K$ 4Js A0
ro )
batch_sizeshuffle
collate_fnr ),r$ trainstate global_stepro tensorrr r% rs tr_loss_globalstep_last_loggedrv rI rJ r1 r r( rN r} r keysupdater from_dict
set_formatr per_device_train_batch_size enumerater;
accumulate compute_lossn_gpumeandetachbackwardsync_gradients
max_grad_normclip_grad_norm_
parametersrX step zero_gradrY _maybe_log_save_evaluate)r\ rh ri rj r r model_inputsmodel_inputs_names
batch_dictr
batch_datastep_dataloader_batchr{ loss tr_loss_steps ` ra r IterativeSFTTrainer.step# s 8
:: ! !Q & <<,// 0@0@ADL+/::+A+AD ( _` `
"u'8 NN6
>l 2t7N7N S
%.$9IaLt .<.H *d $ 0d ! -a4*6*B|A AEAZAZ vlB
> 6, 00$//dDae 1 L )5[(A F00FS !,"3"3"5 6
,' &&z2
g&$ yy<<
"/2HA!!,,TZZ85G H5G58 5G H((\B99??Q& 99;D #{{} ))$/##22tyy7N7N7Z$$44
--/ //
##%((*$$0%%**,
&&!+& , --/7 98 3 H 98s 8O*=O% F O*%O**
O: c \ U R R bf U R R U R R - S:X a5 U R R S:w a U R U R
5 U R R Gb U R R U R R - S:X a U R R S:w a 0 nU R U R 5 R 5 R 5 nU =R U R - sl [ X R R U R -
- S5 US' U R 5 US' U R R U l U R U5 g g g g )Nr r
learning_rate)r%
eval_stepsr r evaluater'
logging_steps_nested_gatherr r itemroundr _get_learning_ratelog)r\ logstr_loss_scalars ra r ,IterativeSFTTrainer._maybe_log_save_evaluate sD 99 +zz%% (<(<<AdjjF\F\`aFa
d / /0 99 " " .zz%% (?(??1DI_I_cdId)+!%!4!4T\\!B!G!G!I!N!N!P , $^zz7M7MPTPlPl7l%mop qV (,(?(?(A_%/3zz/E/E, JeD /rc c > U R R c* [ U R R 5 R nO(U R R R S5 S nU R
US9 [ TU ]! X5 g )Nr- r. )r^ ) r% hub_model_idr
output_dirr rD create_model_cardrQ _save_checkpoint)r\ r$ trialr^ r` s ra r $IterativeSFTTrainer._save_checkpoint sj 99 ! ! ) dii 2 2388J //55c:2>J * 5
.rc r^ dataset_nametagsc U R 5 ( d g[ U R R S5 ( ac [ R
R
U R R R 5 ( d! U R R R nOSnUc [ 5 nO$[ U[ 5 ( a U1nO [ U5 n[ U R R S5 ( a UR S5 S[ R ; a UR S5 UR U R 5 [ UUU R UU[# 5 ( a+ [$ R& b [$ R& R( OS[+ 5 SS9nUR- [ R
R/ U R0 R2 S 5 5 g)
a
Creates a draft of a model card using the information available to the `Trainer`.
Args:
model_name (`str` or `None`, *optional*, defaults to `None`):
Name of the model.
dataset_name (`str` or `None`, *optional*, defaults to `None`):
Name of the dataset used for training.
tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`):
Tags to be associated with the model card.
NrC unsloth_versionunslothJOB_IDhf_jobsz
Iterative SFT)
base_modelr^ r r r wandb_url comet_url trainer_namez README.md)is_world_process_zerorS r$ rB ospathisdirrC setr@ rA addenvironr rT r r r wandbrunurlr savejoinr% r )r\ r^ r r r
model_cards ra r %IterativeSFTTrainer.create_model_card sN " ))++ 4::$$o 6 6rww}}TZZM^M^MlMl?m?m **88J J <5D
c
"
"6Dt9D 4::$$&7 8 8 HHY rzz ! HHY DOO$(!! **%'9';'; @Ueiimm[_.0(
TYY%9%9;GHrc ) r r& r1 rL rY rN r$ rP rX r( r rO )NNNN)NNNN)NNNNN)NNN)+__name__
__module__ __qualname____firstlineno____doc__rT r rA r r r r r
r r r r r r tuplero optim OptimizerrY LambdaLRr r r rR rK r} staticmethodr
LongTensorr r empty_device_cacher r r r __static_attributes__
__classcell__)r` s @ra r! r! 9 s &P )J
HL04EI
W
imFJ bIS/)*bI u /1B BCDbI - bI
uWd3<.@%@ AB bI # )+=?UWee f
bI %++//1I1I1R1RRSbI (0%,, 9UW\WcWc9c0d'ebI "(N+;T+A"BC bI bIHU# U=O UTc U
ell ELL bgbnbn @ 3F(()3F U - -.3F U%%&3F Cy 3F
3i 3F 3Fj %%' 7;;?37%),0
|0D!1!1 23|0 !e&6&6!78|0 e . ./0 |0
S " |0 tCy)
|0 (|0| 0/ %)&*,0 4I SM4I sm4I CcD() 4I 4Irc r! )/r r= pathlibr typingr r r ro
accelerater datasetsr torch.utils.datar transformersr
r r r
r r r r r r r r r transformers.trainer_utilsr transformers.utilsr corer iterative_sft_configr utilsr r peftr r
get_loggerr rI r! r< rc ra r s
, , ' 6 0 4 @ H %}I' }Irc