Y:i?3 SrSSKJr SSKrSSKJr SSKJr SSKJrJ r J r J r J r J r JrJr SSKJrJrJrJrJrJrJrJrJrJrJ r JrJrJrJrJrJrJrJ r J!r!J"r"J r J#r#J$r$J%r%J&r&J'r'J(r(J)r)J*r*J+r+J,r,J-r-J.r.JrJ/r/J0r0J1r1J2r2J3r3JrJ4r4JrJrJrJrJrJ r J r J/r/J0r0J1r1J4r4J r JrJ r J-r-J/r/JrJ/r/ SSK/r/SSK7 SSK5J%r%J6r6 SS K7J8r8 SSKrSSK9r:SS K$J;r; SSKJr SS K4JJ?r? SSK@r@SS KAJBrB SrCSSSSSS.rD\R"SS\DS9S5rFS\RS\GS\GS\R4SjrHS\RS\RS\GS\GS\R4 SjrIS\RS\GS\R4SjrJ\%"S S!\55rK"S"S#\ 5rL"S$S%\L5rM\N"\-S&5(a3SSK.r."S'S(\.R5rP\-R"\P"S)55 gg)*z; 2025.10.10 2025.10.9 4.56.2 0.23.0 __UNSLOTH_VERSIONING__ )TensorN) functional)AnyListOptionalTupleUnionDictSetCallable)[US5(a5[URS5(aURR5 T"U/UQ70UD6n[US5(a5[URS5(aURR5 U$)Nmodel for_training for_inference)hasattrr=r>r?)selfargskwargsoutputfs H/home/james-whalen/llama.cpp/unsloth_compiled_cache/UnslothSFTTrainer.pywrapper*prepare_for_training_mode..wrapper0sx 4 ! !gdjj.&I&I JJ # # %4)$)&) 4 ! !gdjj/&J&J JJ $ $ & ) functoolswraps)rErGs` rFprepare_for_training_moderL/s%__Q NrITF)epilogue_fusion max_autotune shape_paddingz trace.enabledztriton.cudagraphs)dynamic fullgraphoptionscd[R"URSURS5SSS9n[R"URS5SSS9n/n[ X#5HupVUR [R 5n[R"USURS5S9RS5n[R"USS9nXx- n URU 5 M [R"U5nURURSURS45nU$)Nr)chunksdim)rWindexrW) r1chunkreshapeshapeziptofloat32gather unsqueezesqueeze logsumexpappendconcat) logitsrXchunked_logits chunked_indexall_per_token_logps chunk_logits chunk_indexselected_logitslogsumexp_valuesper_token_logpss rFchunked_selective_log_softmaxrpEs[[FLL4D!EPQYZ[N[[r!2QaHM%(%G! #u}}5 ,,|2{G\G\]_G`aiijlm ??9CCAFG88;;J,<.J rItensorpad_idclX:gn[R"USSSS9n[R"USU5nU$)zD Moves all padding tokens in each sequence of a batch to the right. rZT)rW descendingstable)r1argsortra)rrmasksorted_indices packed_tensors rFleft_pack_paddingrs8  D]]4Q4MNLLN;M rIc^\rSrSr%Sr\"SSS0S9r\\\ S'\"SSS 0S9r \\ \ S '\"SSS 0S9r \\ \ S 'SU4S jjr SrU=r$)UnslothSFTConfiga Configuration class for the [`SFTTrainer`]. This class includes only the parameters that are specific to SFT training. For a full list of training arguments, please refer to the [`~transformers.TrainingArguments`] documentation. Note that default values in this class may differ from those in [`~transformers.TrainingArguments`]. Using [`~transformers.HfArgumentParser`] we can turn this class into [argparse](https://docs.python.org/3/library/argparse#module-argparse) arguments that can be specified on the command line. Parameters: > Parameters that control the model model_init_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Keyword arguments for [`~transformers.AutoModelForCausalLM.from_pretrained`], used when the `model` argument of the [`SFTTrainer`] is provided as a string. If you're training a MoE architecture and want to include the load balancing/auxilliary loss as a part of the final loss, remember to set `output_router_logits=True` in this dictionary. chat_template_path (`str` or `None`, *optional*, defaults to `None`): If specified, sets the model's chat template. This can either be the path to a tokenizer (local directory or Hugging Face Hub model) or a direct path to a Jinja template file. When using a Jinja file, you must ensure that any special tokens referenced in the template are added to the tokenizer and that the model's embedding layer is resized accordingly. > Parameters that control the data preprocessing dataset_text_field (`str`, *optional*, defaults to `"text"`): Name of the column that contains text data in the dataset. dataset_kwargs (`dict[str, Any]` or `None`, *optional*, defaults to `None`): Dictionary of optional keyword arguments for the dataset preparation. The only supported key is `skip_prepare_dataset`. When the model is a VLM, `skip_prepare_dataset` is automatically treated as `True` regardless of the provided value, since preprocessing is done on the fly. dataset_num_proc (`int` or `None`, *optional*, defaults to `None`): Number of processes to use for processing the dataset. eos_token (`str` or `None`, *optional*, defaults to `None`): Token used to indicate the end of a turn or sequence. If `None`, it defaults to `processing_class.eos_token`. pad_token (`int` or `None`, *optional*, defaults to `None`): Token used for padding. If `None`, it defaults to `processing_class.pad_token`, or if that is also `None`, it falls back to `processing_class.eos_token`. max_length (`int` or `None`, *optional*, defaults to `1024`): Maximum length of the tokenized sequence. Sequences longer than `max_length` are truncated from the right. If `None`, no truncation is applied. When packing is enabled, this value sets the sequence length. packing (`bool`, *optional*, defaults to `False`): Whether to group multiple sequences into fixed-length blocks to improve computational efficiency and reduce padding. Uses `max_length` to define sequence length. packing_strategy (`str`, *optional*, defaults to `"bfd"`): Strategy for packing sequences. Can be either `"bfd"` (best-fit decreasing, default), or `"wrapped"`. padding_free (`bool`, *optional*, defaults to `False`): Whether to perform forward passes without padding by flattening all sequences in the batch into a single continuous sequence. This reduces memory usage by eliminating padding overhead. Currently, this is only supported with the FlashAttention 2 or 3, which can efficiently handle the flattened batch structure. When packing is enabled with strategy `"bfd"`, padding-free is enabled, regardless of the value of this parameter. pad_to_multiple_of (`int` or `None`, *optional*, defaults to `None`): If set, the sequences will be padded to a multiple of this value. eval_packing (`bool` or `None`, *optional*, defaults to `None`): Whether to pack the eval dataset. If `None`, uses the same value as `packing`. > Parameters that control the training completion_only_loss (`bool` or `None`, *optional*, defaults to `None`): Whether to compute loss only on the completion part of the sequence. If set to `True`, loss is computed only on the completion, which is supported only for [prompt-completion](#prompt-completion) datasets. If `False`, loss is computed on the entire sequence. If `None` (default), the behavior depends on the dataset: loss is computed on the completion for [prompt-completion](#prompt-completion) datasets, and on the full sequence for [language modeling](#language-modeling) datasets. assistant_only_loss (`bool`, *optional*, defaults to `False`): Whether to compute loss only on the assistant part of the sequence. If set to `True`, loss is computed only on the assistant responses, which is supported only for [conversational](#conversational) datasets. If `False`, loss is computed on the entire sequence. loss_type (`str`, *optional*, defaults to `"nll"`): Type of loss to use. Possible values are `"nll"` (negative log-likelihood, default) and `"dft"` (Dynamic Fine-Tuning, as described in [this paper](https://huggingface.co/papers/2508.05629)). activation_offloading (`bool`, *optional*, defaults to `False`): Whether to offload the activations to the CPU. NhelpzvLLM SamplingParams)defaultmetadatavllm_sampling_paramsrTz8Chunk size to reduce memory usage. -1 is most efficient.unsloth_num_chunksz'Maximum sequence length to truncate to.max_seq_lengthc >US:a[SUS35 US:a[SUS35 UcU#S:Xa U$S:XaSnS n#Wc$S S KJn [[ U"5S -S 5S5n[ R RSS5S:XaS SKJ n U(a WcS SKJ n Un[TU]0"S0SU_SU_SU_SU_SU_SU_SU_SU_SU _SU _SU _SU _S U _S!U_S"U_S#U_S$U_S%U_S&U_S'U_S(U_S)U_S*U_S+U_S,U_S-U_S.U_S/U_S0U_S1U_S2U_S3U _S4U!_S5U"_S6U#_S7U$_S8U%_S9U&_S:U'_S;U(_SU+_S?U,_S@U-_SAU._SBU/_SCU0_SDU1_SEU2_SFU3_SGU4_SHU5_SIU6_SJU7_SKU8_SLU9_SMU:_SNU;_SOU<_SPU=_SQU>_SRU?_SSW@_STWA_SUWB_SVWC_SWWD_SXWE_SYWF_SZWG_S[WH_S\WI_S]WJ_S^WK_S_WL_S`WM_SaWN_SbWO_ScWP_SdWQ_SeWR_SfWS_SgWT_ShWU_SiWV_SjWW_SkWX_SlWY_SmWZ_SnW[_SoW\_SpW]_SqW^_SrW__SsW`_StWa_SuWb_SvWc_SwWd_SxWe_SyWf_SzWg_S{Wh_S|Wi_S}Wj_S~Wk_SWl_SWm_SWn_SWo_SWp_SWq_SWr_SWs_SWt_SWu_SWv_SWw_SWx_SWy_SWz_SW{_SW|_SW}_SW~_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_SW_WD6 WUl WUlWUlg)NgHz>z Unsloth: Your learning rate of `zi` is too small and less than 1e-7! Consider increasing it, otherwise gradient updates will be close to 0!rZza` is way too larger > 1! Consider decreasing it to 1e-1, otherwise gradient updates will explode!stepsunsloth_training_checkpointsnor cpu_countrU@UNSLOTH_ENABLE_FLEX_ATTENTION01)HAS_FLEX_ATTENTION)FLEX_ATTENTION_BLOCK_SIZE output_diroverwrite_output_dirdo_traindo_eval do_predict eval_strategyprediction_loss_onlyper_device_train_batch_sizeper_device_eval_batch_sizeper_gpu_train_batch_sizeper_gpu_eval_batch_sizegradient_accumulation_stepseval_accumulation_steps eval_delaytorch_empty_cache_steps learning_rate weight_decay adam_beta1 adam_beta2 adam_epsilon max_grad_normnum_train_epochs max_stepslr_scheduler_type warmup_ratio warmup_steps log_levellog_level_replicalog_on_each_node logging_dirlogging_strategylogging_first_step logging_stepslogging_nan_inf_filter save_strategy save_stepssave_total_limitsave_safetensorssave_on_each_nodesave_only_model'restore_callback_states_from_checkpointno_cudause_cpuuse_mps_deviceseed data_seed jit_mode_evaluse_ipexbf16fp16fp16_opt_levelhalf_precision_backendbf16_full_evalfp16_full_evaltf32 local_rank ddp_backend tpu_num_corestpu_metrics_debugdebugdataloader_drop_last eval_stepsdataloader_num_workersdataloader_prefetch_factor past_indexrun_name disable_tqdmremove_unused_columns label_namesload_best_model_at_endmetric_for_best_modelgreater_is_betterignore_data_skipfsdpfsdp_min_num_params fsdp_config"fsdp_transformer_layer_cls_to_wrapaccelerator_configparallelism_config deepspeedlabel_smoothing_factoroptim optim_args adafactorgroup_by_lengthlength_column_name report_toddp_find_unused_parametersddp_bucket_cap_mbddp_broadcast_buffersdataloader_pin_memorydataloader_persistent_workersskip_memory_metricsuse_legacy_prediction_loop push_to_hubresume_from_checkpoint hub_model_id hub_strategy hub_tokenhub_private_repohub_always_push hub_revisiongradient_checkpointinggradient_checkpointing_kwargsinclude_inputs_for_metricseval_do_concat_batches fp16_backendpush_to_hub_model_idpush_to_hub_organizationpush_to_hub_token mp_parametersauto_find_batch_sizefull_determinism torchdynamo ray_scope ddp_timeout torch_compiletorch_compile_backendtorch_compile_modeinclude_tokens_per_secondinclude_num_input_tokens_seenneftune_noise_alphaoptim_target_modulesbatch_eval_metrics eval_on_startuse_liger_kernelliger_kernel_configeval_use_gather_objectaverage_tokens_across_devicesmodel_init_kwargschat_template_pathdataset_text_fielddataset_kwargsdataset_num_proc eos_token pad_token max_lengthpackingpacking_strategy padding_freepad_to_multiple_of eval_packingcompletion_only_lossassistant_only_loss loss_typeactivation_offloading)printmultiprocessingrminmaxr,environgetunsloth_zoo.flex_attentionrrsuper__init__rrr)rArrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr r r r r rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<rrrrCrrr __class__s rFrFUnslothSFTConfig.__init__sr 4 )I-YB(C"D 1 e&F}oVw%xy  -7":zS?P7J M  # 1"3y{1}a#8"=  ::>>93 ?3 F E!&8&@P%>" R D#R D#7R D R D R D $ R D * R D$8R D+FR D*DR D(@R D'>R D+FR D'>R D$R D'>R D *!R D"(#R D$$%R D&$'R D(()R D**+R D, 0-R D."/R D0!21R D2(3R D4(5R D6"7R D8!29R D: 0;R D<&=R D> 0?R D@"4AR DB*CR DD&g?g@rTlinear皙?rpassivewarningTNrFrZFrrNTFFFFFFO rNFFFFO1autoFFNrTNNFFNrNrTNNTNFNNFrQrNNNNN adamw_8bitNFFlengthNNNNTFTFFNN every_saveNNFNTNFTrPNNNrQFFNlastiFNNFFNNFFFNFTNNtextNNNNFbfdFNNNFnllFNrTN)__name__ __module__ __qualname____firstlineno____doc__r4rrr__annotations__rintrrF__static_attributes__ __classcell__rGs@rFrrsCO`+012+(3-*/VW*#&+EF&NXc] #$&'%&#'"&&'"#"%$%""!&!27!'!$!"%) $!& $  -1!!!$%%) $ $(-"%*!%#!%(,%*!%##' $  $!$)(-"#" "!&(, !# !## %#m}-}-rIrc ,^\rSrSrSrSS/rS&S\\\R\ 4S\ \\ \ 4S\ \S \ \\\4S \ \\\\\44S \ \\\4S \ \S \ \\/\4S\ \\S\\ \R4R6\ \R4R8R:4S\ \\\R4R6\\\44S\ \\R@\R@/\R@4S\ SS\ \\/\44U4Sjjjr!S\\\4S\"S\ \\/\4S\S\\\44 Sjr#Sr$S'U4Sjjr%U4Sjr&S(S\\\'4S\ \'SS4U4S jjjr(U4S!jr)S)S"\ \S\ \S#\\\\S44S$jjr*S%r+U=r,$)*_UnslothSFTTraineri,aZ Trainer for Supervised Fine-Tuning (SFT) method. This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods. Example: ```python from datasets import load_dataset from trl import SFTTrainer dataset = load_dataset("roneneldan/TinyStories", split="train[:1%]") trainer = SFTTrainer(model="Qwen/Qwen2-0.5B-Instruct", train_dataset=dataset) trainer.train() ``` Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using `.from_pretrained` (where `` is derived from the model config) with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. If you're training a model with an MoE architecture and want to include the load balancing/auxilliary loss as a part of the final loss, remember to set the `output_router_logits` config of the model to `True`. args ([`SFTConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. data_collator ([`~transformers.DataCollator`] or `None`, *optional*): Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`. Will default to [`~trainer.sft_trainer.DataCollatorForLanguageModeling`] if the model is a language model and [`~trainer.sft_trainer.DataCollatorForVisionLanguageModeling`] if the model is a vision-language model. train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]): Dataset to use for training. SFT supports both [language modeling](#language-modeling) type and [prompt-completion](#prompt-completion) type. The format of the samples can be either: - [Standard](dataset_formats#standard): Each sample contains plain text. - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role and content). The trainer also supports processed datasets (tokenized) as long as they contain an `input_ids` field. eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Union[Dataset, IterableDataset]]`): Dataset to use for evaluation. It must meet the same requirements as `train_dataset`. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`] or `None`, *optional*, defaults to `None`): Processing class used to process the data. If `None`, the processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default. compute_loss_func (`Callable` or `None`, *optional*, defaults to `None`): A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618) used by [`Trainer`]. compute_metrics (`Callable[[EvalPrediction], dict]` or `None`, *optional*, defaults to `None`): The function that will be used to compute metrics at evaluation. Must take a [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing [`SFTConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the function needs to calculate and return the global summary statistics rather than accumulating the batch-level statistics. callbacks (list of [`~transformers.TrainerCallback`] or `None`, *optional*, defaults to `None`): List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](https://huggingface.co/docs/transformers/main_classes/callback). If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`] method. optimizers (`tuple[Optional[torch.optim.Optimizer], Optional[torch.optim.lr_scheduler.LambdaLR]]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`. optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*, defaults to `None`): A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in `args`. Incompatible with the `optimizers` argument. Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before initializing the Trainer. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*, defaults to `None`): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. peft_config ([`~peft.PeftConfig`] or `None`, *optional*, defaults to `None`): PEFT configuration used to wrap the model. If `None`, the model is not wrapped. formatting_func (`Callable` or `None`, *optional*, defaults to `None`): Formatting function applied to the dataset before tokenization. Applying the formatting function explicitly converts the dataset into a [language modeling](#language-modeling) type. trlsftNr=rB data_collator train_dataset eval_datasetprocessing_classcompute_loss_funccompute_metrics callbacks optimizersoptimizer_cls_and_kwargspreprocess_logits_for_metrics peft_configrformatting_funcc>UcP[U[5(aUOURRnUR S5Sn[ US35nOe[U[ 5(aP[U[ 5(d;UR5nURUS'URS5 [ S90UD6nUR=(d 0n[U[5(aUnURS5n[U[R5(d US:XdUcO@[U[5(aUS;a[[U5nUUS'O[S US 35e[ R""U5n[[$UR&S 5nUR""U40UD6nO9URRnURb[(R*"S 5 Uc[,R""U5n[U[.5(aUR0nS UlO*[U[45(a UnS UlO [7S5eUR8bMUR8nUR;U5nUc&[SUSUR<R>S35eUUl URBb[DRFRIURB5(aUURBRKS5(a5[MURBSS9nURO5Ul(SSS5 /nO[SXURB5upnO/nUR2(aURT(a [S5eUR2(aURV(a [S5eUR2(aURX(a [S5eS Ul1URV=(d" URT=(a URjS:HUl+URRlS;nURV(aUb [S5eURT(a&URjS:Xa[(R*"S5 U(d[(R*"S5 URnS:Xa'URT(d[(R*"S5 [q[sU55nURtcS U;=(a S!U;Ul:OURtUl:UcUR2(dURv=(d URv=(d UR8nUR;U5nUc&[S"USUR<R>S#35e[yUURtURVUURzS$9nOIUcFUR2(a5[}UUR~URtURzURS%9nURT(a-URjS:XaU(d[(R*"S&5 URX(a5(d [S'5eURSL=(a URRS(S 5=(d UR2n U (dURt(aU(a [S)5eURXFX"RTUS*5nUbURc URTO URn![U[5(a:UR5V"V#s0sHun"n#U"URU#XbU!UU"5_M nn"n#OURXVUU!US+5nURS,:XaO>URS-:XaUb [S.5e[nO[S/URS035e[[5[[5S1.UlKS UlL[T$UG]9UUUUUUUUU U U U S29 URR(a[URS39UlSO[R"5UlS[URS45(a%URRUR5 [URS5S 5UlY[URS6S75UlZUR(a(URS7:Xa[(R*"S85 ggg!,(df  GN=fs sn#n"f):N/rTz-SFTr rdtyperP)bfloat16float16r`zInvalid `dtype` passed to `SFTConfig`. Expected either 'auto' or a string representing a valid `torch.dtype` (e.g., 'float32'), but got .rzYou passed `model_init_kwargs` to the `SFTConfig`, but your model is already instantiated. The `model_init_kwargs` will be ignored.FzWThe `processing_class` must be either a `PreTrainedTokenizerBase` or a `ProcessorMixin`zThe specified `eos_token` ('zC') is not found in the vocabulary of the given `processing_class` (zX). Ensure that the `eos_token` exists in the vocabulary before using it as an EOS token.)z.jinjaz.j2zutf-8)encodingzaPacking is not supported for vision-language models. Please set `packing=False` in the SFTConfig.zzPadding-free training is yet not supported for vision-language models. Please set `padding_free=False` in the `SFTConfig`.zAssistant-only loss is not yet supported for vision-language models. Please set `assistant_only_loss=False` in the `SFTConfig`.rY)flash_attention_2flash_attention_3z"kernels-community/vllm-flash-attn3zHPassing a custom data collator is not supported when using padding-free.wrappedzYou are passing `padding_free=True` with the 'wrapped' packing strategy, which is not recommended. Please refer to the documentation to understand why this is not recommended.aPadding-free training is enabled, but the attention implementation is not set to 'flash_attention_2'. Padding-free training flattens batches into a single sequence, and 'flash_attention_2' is the only known attention mechanism that reliably supports this. Using other implementations may lead to unexpected behavior. To ensure compatibility, set `attn_implementation='flash_attention_2'` in the model configuration, or verify that your attention mechanism can handle flattened sequences.rZzYou are using a per_device_train_batch_size of 1 with padding-free training. Using a batch size of 1 anihilate the benefits of padding-free training. Please consider increasing the batch size to at least 2.prompt completionzThe specified `pad_token` ('z[). Ensure that the `pad_token` exists in the vocabulary before using it as a padding token.)rsr9r6return_position_idsr7) processorr3r9r7r.a$You are using packing, but the attention implementation is not set to 'flash_attention_2' or 'kernels-community/vllm-flash-attn3'. Packing flattens batches into a single sequence, and Flash Attention is the only known attention mechanisms that reliably support this. Using other implementations may lead to cross-contamination between batches. To avoid this, either disable packing by setting `packing=False`, or set `attn_implementation='flash_attention_2'` or `attn_implementation='kernels-community/vllm-flash-attn3'` in the model configuration.zYou set `assistant_only_loss=True`, but the dataset is not conversational. This option is only supported for conversational datasets.skip_prepare_datasetaEA formatting function was provided while `completion_only_loss=True`, which is incompatible. Using a formatter converts the dataset to a language modeling type, conflicting with completion-only loss. To resolve this, apply your formatting function before passing the dataset, or disable `completion_only_loss` in `SFTConfig`.trainevalrZdftzYou passed a `compute_loss_func` together with `loss_type='dft'` to the `SFTTrainer`. When using `loss_type='dft'`, the loss function is internally set to the DFT loss, so passing a `compute_loss_func` is not allowed.zInvalid `loss_type` z. passed. Supported values are 'nll' and 'dft'.)rr) r=rBrirjrkrlrmrnrorprqrr)r=add_model_tagsoutput_router_logitsrouter_aux_loss_coefrRa-You set `output_router_logits` to `True` in the model config, but `router_aux_loss_coef` is set to `0.0`, meaning the auxiliary loss will not be used. Either set `router_aux_loss_coef` to a value greater than `0.0`, or set `output_router_logits` to `False` if you don't want to use the auxiliary loss.r=)[ isinstancestrconfig _name_or_pathsplitrrto_dictr popr,rCr1rwgetattrrvr from_pretrainedr2 architecturesr)rMrr tokenizer_is_vlmr TypeErrorr1convert_tokens_to_idsrGr[ eos_token_idr-r,pathisfileendswithopenread chat_templaterr4r6r:trainable_token_indicesextendmodules_to_saverenum_virtual_tokensr/active_adapterrsr5_attn_implementationrnextiterr9r2rr7rr3r.r'r/_prepare_datasetr8dictitemsr;r#r"list_metrics_total_train_tokensrErFrBr<r%r= maybe_activation_offload_contextr r6r@r _tag_namesaux_loss_enabled aux_loss_coef)%rAr=rBrirjrkrlrmrnrorprqrrrsrt model_name dict_argsr,model_idrwr architecturerr1rchat_template_file added_tokenspeft_model_configuse_flash_attentiondataset_sampler2rsrr4keydatasetrGs% rFrF_UnslothSFTTrainer.__init__sJ$ <",UC"8"8ell>X>XJ#))#.r2J |401D / 0 0D)9T9T I%)^^Ik " MM- .)y)D!228b eS ! !H%))'2E%--&EME3''E5W,Wu--2!'* HHMwaQ //9F"<1E1Ea1HIL 00O=NOE||11H%%1?  #,<> %I$::9EL# 29+>++;+E+E+N+N*OPII &2I "  " " .ww~~d55664;R;R;[;[\m;n;n$11GDHZ5G5L5L5N$2E! 8KT-D-D95L <++;+E+E+N+N*OPLL <)%)%>%>!..$7#'#:#: M "t||A*??%)%>%>#'#:#:#'#:#: M <,>,@$,@LCT227>U "  ^^u $ , : !) 3DNN3CCqrs s#.d"3[=NO #$  ''%-/+!%=*G  99 * *4RY]YcYc4dD 14>4J4J4LD 1 4::/ 0 0 JJ % %doo 6 ' 6Le T$U\\3I3O  T%7%73%> NN &? CEDJ$sd2##e2 err4 dataset_namertc ^^^^^^^[U[5(aU$0n[U[5n[US5n UmU (a URm[ USS5mTS:Xa [ USS5mTS:Xa [ USS5mTS:Xa [ USS5mTS:Xa [ S5e[ USS5mTS:gmS mS n [[[U55R55n S /n S U ;aU RS 5 SS K J n Jn SU ;aRU (a*[TS5(d[ SURS35eU "T5UlU RS5 S n O\S U ;a@U (a*[TS5(d[ SURS35eU"TS S9UlS n OTU ;aS mTc [ S5eU (GaT(a@T"[[U555n[U["5(d [%S5eUSnO[[U55TSn[ USS5nUS:XaU (a [ TSS5nUcSnS m[ USS5n[ TSS5nU=(d UnUb)UR'U5(dUU;a S m[)S5 UUUUUUU4Sjn[U[*5(d0[ USS5nUcSSKJn [1U"5S-S5nUUS'OUR2R4US'U(a S TS!3US"'UR6"U4S#S 0UD6nU (a[US5(dU"TS S9nUUlU(aP[8 TS:Xa [%S%5eU(a S&US'3US"'[9UR;U 5T[ US(S)5U5nU$! GN=f! [)S$5 Us$=f)*Nrr3rrmax_seqz1Unsloth: max_seq_length is 0! Please specify one!r.rWFTrqattention_maskr7labelsr.z Unsloth: z does not have .pad!)mlmz-Unsloth: You must specify a `formatting_func`zIUnsloth: The `formatting_func` should return a list of processed strings.rrQ bos_tokenzHUnsloth: We found double BOS tokens - we shall remove one automatically.c<>T"T(dUTOT"U5TTSTS9$)NF) truncationr3return_token_type_idsadd_special_tokensr=)examplerr.do_formatting_func do_truncationrtrrs rF _tokenize6_UnslothSFTTrainer._prepare_dataset.._tokenize s/ 7IG./_fOg!.!/,1); rIr0rrUrnum_procrzUnsloth: Tokenizing ["z"]descbatchedzPUnsloth: Hugging Face's packing is currently buggy - we're disabling it for now!z:When packing is enabled, `max_seq_length` can't be `None`.zUnsloth: Packing z datasetr5rY)rConstantLengthDatasetrr@rr RuntimeErrorsetrrkeysrer2r8rrGrirrv startswithr>rr?rrA _ex_iterablermapr-select_columns)rArrlrBr4rtr map_kwargsuse_descis_vlm do_tokenize column_namesused_column_namesr8r test_textr bos_token_1 bos_token_2rrr0rrirr.rrrrs ` @@@@@@rFr#_UnslothSFTTrainer._prepare_datasets '#899'>9 gw/);7$ /999!|Q7 Q ?OQR1S Q ?OQR1S Q y!1L Q l3f&g g$T+?H&!+ " 4W .3356 (M | +  $ $%5 6 Y | #gi77"Y/?/I/I.JJ^#_``!7 !BD   $ $X .K L (gi77"Y/?/I/I.JJ^#_``!@RW!XD K | 3!% &"#RSS !+Dg,?@ !)T22$c&aL  g/0BCAF $$4orJM"v ' ?B G $ " "& !"2KFK!)[$?K#2{I$'' 22i=6P).&de    g77#*41CT#J #+9'*9;q=!'<$)9 :&+2+?+?+J+J <(0FGYFZZ\.]F+kk)JtJzJGg&6>> ? QV W %2"    " !]^^0A,x.XF+"&&'890%8 G A  ` hisN>,O>OOchURc%UR(a /SQUlg/SQUlgg)N)messagesrrimages)rqr seq_lengthscompletion_maskassistant_masks)_signature_columnsr)rAs rF _set_signature_columns_if_needed3_UnslothSFTTrainer._set_signature_columns_if_neededQs,  " " *||*X'*v' +rIc(>[TU]UUUUS9nU$)N)return_outputsnum_items_in_batch)rE compute_loss)rAr=inputsrroutputsrGs rFr_UnslothSFTTrainer.compute_loss\s*'&  +!3 '  rIcr>UR [TU]"U0UD6sSSS5 $!,(df  g=fN)rrE training_step)rArBrCrGs rFr _UnslothSFTTrainer.training_stepfs*  2 27($9&93 2 2s( 6logs start_timec >URR(aSOSnURUR5VVs0sHupEU[ U5[ U5- _M nnnUS:Xa(UR5VVs0sH upESU3U_M nnnUR U5 [TU]!X5 URUR5 gs snnfs snnf)Nrreval_) r=trainingrrrwlenupdaterElogclear)rArrmodervalmetricsrGs rFr_UnslothSFTTrainer.logjs**--w6<@MM$:A--/J/hcse}c)/GJ G  D% d!!#Y Ks $CCc>URRc*[URR5RnO(URRR S5SnUR US9 [TU]!X5 g)NrvrT)r) rBr rrnamercreate_model_cardrE_save_checkpoint)rAr=trialrrGs rFr#_UnslothSFTTrainer._save_checkpointxsj 99 ! ! )dii22388J//55c:2>J *5  .rIrtagsc UR5(dg[URRS5(ac[R R URRR5(d!URRRnOSnUc [5nO$[U[5(aU1nO [U5n[URRS5(aURS5 S[R;aURS5 URUR5 [UUUR U[#U5[%5(a+[&R(b[&R(R*OS[-5SS9nUR/[R R1UR2R4S 55 g) a Creates a draft of a model card using the information available to the `Trainer`. Args: model_name (`str` or `None`, *optional*, defaults to `None`): Name of the model. dataset_name (`str` or `None`, *optional*, defaults to `None`): Name of the dataset used for training. tags (`str`, `list[str]` or `None`, *optional*, defaults to `None`): Tags to be associated with the model card. Nrunsloth_versionunslothJOB_IDhf_jobsSFT) base_modelrr rr  wandb_url comet_url trainer_namez README.md)is_world_process_zeror@r=rr,risdirrrrraddrBrrr$r rr(wandbrunurlr&savejoinrBr)rArrr r model_cards rFr$_UnslothSFTTrainer.create_model_cardsR"))++  4::$$o 6 6rww}}TZZM^M^MlMl?m?m**88JJ <5D c " "6Dt9D 4::$$&7 8 8 HHY  rzz ! HHY  DOO$(!!**%d'9';'; @Ueiimm[_.0    TYY%9%9;GHrI) rrrrrrr9rirrr6) NNNNNNNN)NNNNNN)FNr)NNN)-r[r\r]r^r_rr rr+Modulerrrrrrrrrrr rrrtupler1r Optimizer lr_schedulerLambdaLRtyperrrFboolrrrrfloatrrrrbrcrds@rFrfrf,sXtJ ?C04CGEIUY04FJ59jvaehl.2;?xS"))_45xuY(99:;x - x  g&> ?@ x uWd3<.@%@AB x#5)@.)P#QRx$H-x"(N+;T+A"BCxD12x(5;;#8#898EKKD\D\DeDe;ffgx#+5ekk6K6K1LdSVX[S[n1\+]"^x(0%,, 9UW\WcWc9c0d'exl+x"(D63;"78xxt Lw/0L  L "(D63;"78 LL w' (L\ w: $S%Z( $huo $QU $ $/%)&*,0 4ISM4Ism4ICcD() 4I4IrIrfcH^\rSrSrSrSU4SjjrSrU=r$)UnslothSFTTraineriad Trainer for Supervised Fine-Tuning (SFT) method. This class is a wrapper around the [`~transformers.Trainer`] class and inherits all of its attributes and methods. Example: ```python from datasets import load_dataset from trl import SFTTrainer dataset = load_dataset("roneneldan/TinyStories", split="train[:1%]") trainer = SFTTrainer(model="Qwen/Qwen2-0.5B-Instruct", train_dataset=dataset) trainer.train() ``` Args: model (`Union[str, PreTrainedModel]`): Model to be trained. Can be either: - A string, being the *model id* of a pretrained model hosted inside a model repo on huggingface.co, or a path to a *directory* containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `'./my_model_directory/'`. The model is loaded using `.from_pretrained` (where `` is derived from the model config) with the keyword arguments in `args.model_init_kwargs`. - A [`~transformers.PreTrainedModel`] object. If you're training a model with an MoE architecture and want to include the load balancing/auxilliary loss as a part of the final loss, remember to set the `output_router_logits` config of the model to `True`. args ([`SFTConfig`], *optional*, defaults to `None`): Configuration for this trainer. If `None`, a default configuration is used. data_collator ([`~transformers.DataCollator`] or `None`, *optional*): Function to use to form a batch from a list of elements of the processed `train_dataset` or `eval_dataset`. Will default to [`~trainer.sft_trainer.DataCollatorForLanguageModeling`] if the model is a language model and [`~trainer.sft_trainer.DataCollatorForVisionLanguageModeling`] if the model is a vision-language model. train_dataset ([`~datasets.Dataset`] or [`~datasets.IterableDataset`]): Dataset to use for training. SFT supports both [language modeling](#language-modeling) type and [prompt-completion](#prompt-completion) type. The format of the samples can be either: - [Standard](dataset_formats#standard): Each sample contains plain text. - [Conversational](dataset_formats#conversational): Each sample contains structured messages (e.g., role and content). The trainer also supports processed datasets (tokenized) as long as they contain an `input_ids` field. eval_dataset ([`~datasets.Dataset`], [`~datasets.IterableDataset`] or `dict[str, Union[Dataset, IterableDataset]]`): Dataset to use for evaluation. It must meet the same requirements as `train_dataset`. processing_class ([`~transformers.PreTrainedTokenizerBase`], [`~transformers.ProcessorMixin`] or `None`, *optional*, defaults to `None`): Processing class used to process the data. If `None`, the processing class is loaded from the model's name with [`~transformers.AutoProcessor.from_pretrained`]. A padding token, `tokenizer.pad_token`, must be set. If the processing class has not set a padding token, `tokenizer.eos_token` will be used as the default. compute_loss_func (`Callable` or `None`, *optional*, defaults to `None`): A function that accepts the raw model outputs, labels, and the number of items in the entire accumulated batch (batch_size * gradient_accumulation_steps) and returns the loss. For example, see the default [loss function](https://github.com/huggingface/transformers/blob/052e652d6d53c2b26ffde87e039b723949a53493/src/transformers/trainer.py#L3618) used by [`Trainer`]. compute_metrics (`Callable[[EvalPrediction], dict]` or `None`, *optional*, defaults to `None`): The function that will be used to compute metrics at evaluation. Must take a [`~transformers.EvalPrediction`] and return a dictionary string to metric values. When passing [`SFTConfig`] with `batch_eval_metrics` set to `True`, your `compute_metrics` function must take a boolean `compute_result` argument. This will be triggered after the last eval batch to signal that the function needs to calculate and return the global summary statistics rather than accumulating the batch-level statistics. callbacks (list of [`~transformers.TrainerCallback`] or `None`, *optional*, defaults to `None`): List of callbacks to customize the training loop. Will add those to the list of default callbacks detailed in [here](https://huggingface.co/docs/transformers/main_classes/callback). If you want to remove one of the default callbacks used, use the [`~transformers.Trainer.remove_callback`] method. optimizers (`tuple[Optional[torch.optim.Optimizer], Optional[torch.optim.lr_scheduler.LambdaLR]]`, *optional*, defaults to `(None, None)`): A tuple containing the optimizer and the scheduler to use. Will default to an instance of `AdamW` on your model and a scheduler given by [`~transformers.get_linear_schedule_with_warmup`] controlled by `args`. optimizer_cls_and_kwargs (`tuple[Type[torch.optim.Optimizer], Dict[str, Any]]`, *optional*, defaults to `None`): A tuple containing the optimizer class and keyword arguments to use. Overrides `optim` and `optim_args` in `args`. Incompatible with the `optimizers` argument. Unlike `optimizers`, this argument avoids the need to place model parameters on the correct devices before initializing the Trainer. preprocess_logits_for_metrics (`Callable[[torch.Tensor, torch.Tensor], torch.Tensor]`, *optional*, defaults to `None`): A function that preprocess the logits right before caching them at each evaluation step. Must take two tensors, the logits and the labels, and return the logits once processed as desired. The modifications made by this function will be reflected in the predictions received by `compute_metrics`. Note that the labels (second parameter) will be `None` if the dataset does not have them. peft_config ([`~peft.PeftConfig`] or `None`, *optional*, defaults to `None`): PEFT configuration used to wrap the model. If `None`, the model is not wrapped. formatting_func (`Callable` or `None`, *optional*, defaults to `None`): Formatting function applied to the dataset before tokenization. Applying the formatting function explicitly converts the dataset into a [language modeling](#language-modeling) type. c >Uc [5n[USS5n[U5[LaSn[USS5n[U5[LaSnSn[R R SS5S:HnU(d1[R R SS5S:Xa [S5 S n[R R S S 5n[URS S5=(d [URS S5nUcUR5RnSSK J n U"U5nU[R:HnU(dU(aU(a [S5eU(dU(dU(a [S5eU(a"SUlSUlS[R S'OCU(d<U(d5US :Xa/UUlU(+UlU(aSOS[R S'[USS5b-[USS5S:XaSUl[USS5cSUl[USS5nUb/US:a)SSKJn [-U5[-S5::a [S5 [USS5S:waL[USS5nUS:Xa!UR.U:aUR.Ul[US S5c UbUUl[US!S5n[U5[LaSn[US"S5n[U5[LaSnUR (aU(aSUlS UlUR"(aU(aS UlSUlU(aSUlSUlOc[R R S S 5S#:XaS UlSUlO0U(d)U(d"UR"UlUR UlSn[95R S$S5bS n[95R S%S5bS nU(aS[R S&'S'[95;a[;US'5(dOD[US'S5n[US'S5nUc'Ub$UR<n[;US'5(aUUlS([95;a[;US(5(dO[;US'5(aLUR<b?UR<S:a/[;US(5(aUR<UlUR>n O~[US'S5n!U!c [US(S5n!U!bU!UlUR>n OJ[;US(5(a'UR>bUR>n [AUS'U 5 O[S)5 S*UlUb![;US+5(aURC5 S,[95;a[;W"S-5(aS.U"l"S/[95;aU[;US-5(aS.Ul"[;US,5(a,[;URFS-5(aS.URFl"S/[95;aUOW"n#SS0K$J%n$ [MUU$5(dx[MU[N5(a(S1URP;a[SU#SS2[US3S5S49nO[MU[R5(a%S1URP;a[OU#[US3S5S59nOJ[;US65(aSUl*[;US75(aS8Ul+[;US95(a S:S 0Ul,[MUU$5(dx[;U#S;5(dg[;U#S,5(aV[MU[N5(a [OU#RF[US3S5S59nO![SU#RFSS2[US3S5S49n/n%SSS85R_S?5n'SS@K0J1n( SSAK2J3n) S,[95;aUn"U("UW"UU'SBSC9 U)"UU"U5 [USDS5[hRj:Xa(URlS:a[USES5S:waSUl7SF[95;a![;US+5(aURC5 [pT,U]"SMUUUUUUUUU U U U U SG. UD6 SF[95;a![;USH5(aURu5 [;USI5(a-URvRy5 [;USI5(aU?;[USJS5bURzUR5l=[;USK5(aVUR|R~n*Un+[;U+SF5(a&U*U+l@U+Rn+[;U+SF5(aM&U*U+l@[;USL5(a.[[URR5U5UlEg)NNrFrUNSLOTH_ENABLE_FULL_FINETUNINGrrUNSLOTH_FORCE_FLOAT32zKUnsloth: Switching to float32 training since model cannot work with float16TUNSLOTH_MIXED_PRECISIONr`rw torch_dtyper) _get_dtypezuUnsloth: Model is in float16 precision but you want to use bfloat16 precision. Set fp16 to `True` and bf16 to `False`zuUnsloth: Model is in bfloat16 precision but you want to use float16 precision. Set fp16 to `False` and bf16 to `True`rACCELERATE_MIXED_PRECISIONrkrrrrKrrZ) __version__z4.45.2z**** Unsloth: Please use our fixed gradient_accumulation_steps by updating transformers, TRL and Unsloth! `pip install --upgrade --no-cache-dir --force-reinstall --no-deps unsloth transformers trl unsloth_zoo`rrrrrxrnrrUNSLOTH_RETURN_LOGITSrr3zgUnsloth: We did not find `max_seq_length` or `max_length` in the model or args. We will set it to 1024.rXr>r padding_siderightrl)UnslothVisionDataCollatorrrRr7)rmlm_probabilityr7)r7rr.rQr/rr.)PatchRLStatistics sft_trainerUNSLOTH_IGNORED_TOKENIZER_NAMES )fix_untrained_tokens)fix_zero_training_lossgؗҜ<)eps parallel_mode_n_gpur=) r=rBrirjrkrlrmrnrorqrrrsrtr?neftune_hook_handler$ acceleratorrr=)Frrr#r$r,rBrCr>rget_input_embeddingsrwunsloth_zoo.utilsr-r1ryrrrrrr2r/r5rrrrrlocalsr@rr3setattrr>r2runsloth_zoo.vision_utilsr4rr8r+TransformersDataCollatorForLanguageModelingrr.r/unsloth_zoo.logging_utilsr6runsloth_zoo.tokenizer_utilsr:unsloth_zoo.training_utilsr;r9NOT_DISTRIBUTEDn_gpur>rErFr?r?remover$r@scaleraccelerator_scalerr=r:rLrGr)-rAr=rBrirjrkrlrmrnrorqrrrsrtrCuse_bf16use_fp16 force_float32full_finetuningmixed_precision_dtyperwr-ryga_stepstransformers_versioneval_bszrr_output_logitsmodel_max_seq_lengthargs_max_seq_lengthrr3model_max_lengthr_UnslothSFTTrainer__tokenizerr4 other_metricsr6IGNORED_TOKENIZER_NAMESr:r;rM current_modelrGs- rFrFUnslothSFTTrainer.__init__s" < 0 24/ > %%x4/ > %%x **..)I3OSVVBJJNN3JC$PTW$W _ ` M " /H) T gt4b m]a8b =%"<"<">"D"D%05!5==('hyJA@B:Bg(9NEDF>F DIDI7;BJJ3 481F)1SDI# DIAHvfBJJ3 4 4 . :wt_^b?cgk?k!(D t\408C$/4!>E  HqL H+,0AA@A 4$ /4 7t%A1EH1}!A!AH!LptqQqQdNmt6=E(J^@H`d`| '7?  t +e^ '7?  t +e^ 99u)<\`dFY 99t)<[`TEX "'D "'D  ZZ^^5y AZ O"&D "'D "&))D "&))D  8<<)4 0 J]aN 25BJJ. / 68 +GDBR4S4S #*52BD#I #*42BD#I "*/C/O!&!5!54!122.D4G vx 'l0K0K t-..43F3F3RW[WjWjmnWn4..&*&9&9DO!%J#*52BD#I #+|]a@b-=#/&6DO!%JT<00T__5P!%JE#3Z@DE&*DO  !?!?    &( "wy.'I'Idk9Ka  )'88Za:J:W'55'BRB\B\^l:m:mZaopJpJpW*<*H&i F-)BCC-)?@@XUbUoUoEo K&))07KT)R ! M+VWW\dhuiCiC]C 6)07KT)R! t455TYt7Qt122bD4Kt-..G]_cFd0C-)BCC;..7; 3T3Tm-CDD$:#---4T;OQU-V%M %P#--#*--4T;OQU-V %M  ?-7"$**..1RTV"W"]"]^b"cDF fh &4D UI}>U]bcui? 4$ /<3O3O OTXT^T^abTbtXq)Q. fh 75.#A#A      8))'/ 1-!'?,I%- 817 8 fh 75/#B#B    ! 4. / /  $ $ + + -t2339Q 4. 5 A?C?W?WE & & ( < 4 ' '%%,,F!M-1139 0 - 3 3 -1106M , 4 ! !#$=dnn>R>R$SUYZDJ rI)r) NNNNNNNNNNNN)r[r\r]r^r_rFrbrcrds@rFr'r's<Z| #'(,C C rIr' addFilterc \rSrSrSrSrSrg)HideLoggingMessageicXlgrrW)rArWs rFrFHideLoggingMessage.__init__sd)rIc<URUR5;$r)rW getMessage)rAxs rFfilterHideLoggingMessage.filtersalln)DErIrdN)r[r\r]r^rFrirbr=rIrFrbrbs2ErIrbz`use_cache=True`)Rr_r1rtorch.nnr+rFtypingrrrrr r r r trl.trainer.sft_trainerr rrrrrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r,r-r.r/r0r2 dataclassesr4packaging.versionr5numpynpr6r8rFtransformers.training_argsr9rJtypesr:rLtorch_compile_optionscompilerprar{rrrrfr'r@Filterrbr`r=rIrFrxs0 $IIIY Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y  (% "$3  4;PRS"|| \\ &,, %    \\ 6ellCELL Z-yZ- Z-v H IH IR_ *_ B  6;FW^^F  '(:;<  rI