oi<SSKrSSKrSSKJr SSKJrJrJr SSKJ r J r SSK J r SSK JrJrJrJrJrJr SSKJr SS KJr SS KJrJrJr "S S \ 5rg) N)Conv1D)is_bnb_4bit_availableis_bnb_availableis_gptqmodel_available) LoraConfig LoraModel)BaseTunerLayer)5TRANSFORMERS_MODELS_TO_ADALORA_TARGET_MODULES_MAPPING_freeze_adapter_get_submodulesget_auto_gptq_quant_linearget_gptqmodel_quant_linearget_quantization_config)gather_params_ctx)SVDQuantLinear) AdaLoraLayer RankAllocator SVDLinearc^\rSrSrSr\rU4SjrS\SS4U4Sjjr Sr \ SS j5r S r S rS rS rSrSrU=r$) AdaLoraModel%a Creates AdaLoRA (Adaptive LoRA) model from a pretrained transformers model. Paper: https://openreview.net/forum?id=lq62uWRJjiY Args: model ([`transformers.PreTrainedModel`]): The model to be adapted. config ([`AdaLoraConfig`]): The configuration of the AdaLora model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The AdaLora model. Example:: >>> from transformers import AutoModelForSeq2SeqLM >>> from peft import LoraConfig, AdaLoraModel, AdaLoraConfig >>> config = AdaLoraConfig( peft_type="ADALORA", task_type="SEQ_2_SEQ_LM", init_r=12, lora_alpha=32, target_modules=["q", "v"], lora_dropout=0.01, ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> model = AdaLoraModel(model, config, "default") **Attributes**: - **model** ([`transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`AdaLoraConfig`]): The configuration of the AdaLora model. c >[TU]"XU40UD6 SnURR5HnUR(aMUS- nM US:a [ S5eURUR(a[ URU5 gX0l[URURUUR5Ul g)NrrzAdaLoraModel supports only 1 trainable adapter. When using multiple adapters, set inference_mode to True for all adapters except the one you want to train.) super__init__ peft_configvaluesinference_mode ValueErrorr modeltrainable_adapter_namer rankallocator)selfr config adapter_namekwargstraininable_mode_counter __class__s S/home/james-whalen/.local/lib/python3.13/site-packages/peft/tuners/adalora/model.pyrAdaLoraModel.__init__Es  ??#$ &&--/F((((A-(0 $a '~    L ) 8 8 DJJ 5*6 '!.tzz4;K;KL;Y[_[v[v!wD r$returnNc>[TU]U5 SnURR5HnUR(aMUS- nM US:a"[ UR RS35eg)z A helper method to check the config when a new adapter is being added. Raise a ValueError if there is something wrong with the config or if it conflicts with existing adapters. rrz supports only 1 trainable adapter. When using multiple adapters, set inference_mode to True for all adapters except the one you want to train.N)r_check_new_adapter_configrrrrr(__name__)r#r$r'config_r(s r)r.&AdaLoraModel._check_new_adapter_configYs~ )&1#$ ''..0G)))(A-(1 $a '>>**+,%%  (r+c URURURURUR[ UR SS5[ UR SS5S.nUS(d US(a[5(d [S5e[UR SS 9nUbXS '[U[5(d|[UR S 5(aUR ROSn UR"XU4S U 0UD6n X R;aU R!S5 UR#XTX5 gUR%UURURURUR5 g) Nis_loaded_in_8bitFis_loaded_in_4bit)r lora_alpha lora_dropoutfan_in_fan_outinit_lora_weightsloaded_in_8bitloaded_in_4bitr:r;zTo use AdaLora with 8-bit quantization, please install the `bitsandbytes` package. You can install it with `pip install bitsandbytes`.gptq)methodgptq_quantization_config hf_device_map device_map)init_rr6r7r8r9getattrr r ImportErrorr isinstancerhasattrr?_create_new_moduleactive_adaptersrequires_grad__replace_module update_layer) r# lora_configr%target target_nameparent current_keyr&quantization_configr@ new_modules r)_create_and_replace AdaLoraModel._create_and_replacensY##%00'44)88!,!>!>%djj2EuM%djj2EuM  # $/?(@JZJ\J\F  6djjP  *1D- .&,//5"S 5 S=US 'Ul OQ[U [B5(a-US (d"[<R>"S5 S=US 'Ul O[ESUS35e[GX!40UD6n U $)Nrr)SVDLinear8bitLt) SVDLinear4bitr>)r@r:Fr;)has_fp16_weights thresholdindex) compute_dtypecompress_statistics quant_typer8zjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.TzTarget module zP is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.)$r bitsandbytesbnbrUrrVgetrrr poprDr get_base_layernn Linear8bitLtupdatestaterWrXrY Linear4bitcopyrZweightr[r\rtorchLinearwarningswarnr8rrr)rKr%rLr@r&r^rUrVr> QuantLinearr:r;target_base_layerrQfourbit_kwargss r)rFAdaLoraModel._create_new_modulesq    & , " " *#)::.H$#O ! # #45MeK45MNK$4e<$4e< fn - - & 5 5 7  &  j):CFF??5A5I#((4::c?1R01sxxX\XbXbcfXghijlXmOnC*4::s;LAq#]]<8BN#]]<8BN#]]<8HENnn\2G   &&((--  !8MM,/55nEMM,/55nEMM,/55nENN<066w? !+3*!s A?H## H3 cUR5HupE[U5nX4;a#SRURS5SS5O"SRURS5SS5nSH^nSUSUSU3n US:wa$XbU RS:waX)UOX)X)'M8XbU RS:wa X)SS2U4OX)X)'M` M U$) Nrrrr)rrsrtzbase_model.model.rtr)rrrrr}) r#r state_dictr%rrrprefixlayerrs r)!resize_state_dict_by_rank_pattern.AdaLoraModel.resize_state_dict_by_rank_patterns*002NDx=D8D8LSXXdjjoa34RURZRZ[_[e[efi[jklmo[pRqF7)&5'<.IH$59_=R=RST=U5U 1[e[jO 9=3@U@UVW@X8X 8 4^h^mO83r+c>URURnXRUR- :a7URR UR U5up4U(aXBlggXRUR- :XaHURR UR USS9up4XBlURR5 gXRUR- :a1URRUR UR5 gg)a This method updates Adalora budget and mask. This should be called in every training step after `loss.backward()` and before `zero_grad()`. `tinit`, `tfinal` and `deltaT` are handled with in the method. Args: global_step (`int`): The current training step, it is used to calculate adalora budget. Example: ```python >>> loss = model(**input).loss >>> loss.backward() >>> optimizer.step() >>> model.base_model.update_and_allocate(i_step) >>> optimizer.zero_grad() ``` T) force_maskN) rr! total_steptfinalr"update_and_allocater r reset_iptmask_using_rank_pattern)r# global_steprKrrs r)r AdaLoraModel.update_and_allocate.s*&&t'B'BC //+2D2DD D"00DDTZZQ\]OA+7(22[5G5GG G"00DDTZZQ\imDnOA(4 $    ( ( *11K4F4FF F    6 6tzz;C[C[ \r+cF[URRS35e)z;This method is not supported for AdaLoRA, use LoRA instead.z. does not support add_weighted_adapter method.) TypeErrorr(r/)r#rr&s r)add_weighted_adapter!AdaLoraModel.add_weighted_adapterXs 4>>2233abccr+)r"r!)N)r/ __module__ __qualname____firstlineno____doc__r target_module_mappingrrr.rR staticmethodrFryrrrr__static_attributes__ __classcell__)r(s@r)rr%si:Rx( t*+ZCCJ<@> (Tddr+r)rkritransformers.pytorch_utilsrpeft.import_utilsrrrpeft.tuners.lorarrpeft.tuners.tuners_utilsr peft.utilsr r r r rrpeft.utils.integrationsrr<rrrrrrr+r)rsD -]]236 99ud9udr+