oi'5SSKJr SSKrSSKrSSKJr SSKrSSKJr SSK J r J r SSK J r Jr SSKJrJrJrJrJr SS KJrJrJrJr "S S \ 5rg) ) annotationsN)replace)Conv1D)is_bnb_4bit_availableis_bnb_available) BaseTunerBaseTunerLayer)6TRANSFORMERS_MODELS_TO_IA3_FEEDFORWARD_MODULES_MAPPING1TRANSFORMERS_MODELS_TO_IA3_TARGET_MODULES_MAPPINGModulesToSaveWrapper_freeze_adapter_get_submodules)Conv2dConv3dIA3LayerLinearc^\rSrSr%SrSrS\S'\r\ S5r Sr \ SSj5r \ S 5r U4S jrSS jrSS jrS rU=r$)IA3Model$a Creates a Infused Adapter by Inhibiting and Amplifying Inner Activations ((IA)^3) model from a pretrained transformers model. The method is described in detail in https://huggingface.co/papers/2205.05638 Args: model ([`~transformers.PreTrainedModel`]): The model to be adapted. config ([`IA3Config`]): The configuration of the (IA)^3 model. adapter_name (`str`): The name of the adapter, defaults to `"default"`. low_cpu_mem_usage (`bool`, `optional`, defaults to `False`): Create empty adapter weights on meta device. Useful to speed up the loading process. Returns: `torch.nn.Module`: The (IA)^3 model. Example: ```py >>> from transformers import AutoModelForSeq2SeqLM, ia3Config >>> from peft import IA3Model, IA3Config >>> config = IA3Config( ... peft_type="IA3", ... task_type="SEQ_2_SEQ_LM", ... target_modules=["k", "v", "w0"], ... feedforward_modules=["w0"], ... ) >>> model = AutoModelForSeq2SeqLM.from_pretrained("t5-base") >>> ia3_model = IA3Model(config, model) ``` **Attributes**: - **model** ([`~transformers.PreTrainedModel`]) -- The model to be adapted. - **peft_config** ([`ia3Config`]): The configuration of the (IA)^3 model. ia3_strprefixc [5(a SSKnSSKJn [ 5(aSSKJn UR SS5nUR SS5nUR SS5n [U[5(aUR5n OUn U(a[U WRR5(adUR5n U RU RRU RRU R S .5 W"X!4SU 0U D6n U $U(a[U WRR 5(adUR5n U RU R"U R$R&U R$R(S .5 W"X!4SU 0U D6n U $[U[*RR,5(a[-X!4SU 0UD6n U $[U[*RR.5(a[/X!4SU 0UD6n U $[U [*RR05(a<US (a"[2R4"S 5 S=US 'Ul!4SU 0UD6n U $[U [85(a=US (d"[2R4"S 5 S=US 'Ul!4U SS.UD6n U $[;SUS35e)Nrr) Linear8bitLt) Linear4bitloaded_in_8bitFloaded_in_4bitis_feedforward)has_fp16_weights thresholdindex) compute_dtypecompress_statistics quant_typefan_in_fan_outzjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.zafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.T)ris_target_conv_1d_layerzTarget module zd is not supported. Currently, only `torch.nn.Linear`, `torch.nn.Conv2d`, and `Conv1D` are supported.)r bitsandbytesbnbrrrpop isinstancer get_base_layernncopyupdatestater r!r"r#weightr$r%torchrrrwarningswarnr&r ValueError) ia3_config adapter_nametargetkwargsr)rrrrrtarget_base_layereightbit_kwargs new_modulefourbit_kwargss O/home/james-whalen/.local/lib/python3.13/site-packages/peft/tuners/ia3/model.py_create_new_moduleIA3Model._create_new_moduleLs    & ) " " '$4e<$4e<$4e< fn - - & 5 5 7  &  j):CFFm]lmJJI +O>O P P#[[]N  ! !%6%D%D+<+C+C+W+W"3":":"E"E $Fjj[ijJ65 0 0^^^W]^J21 0 0^^^W]^J.-)588?? ; ;&' 7HML'(:+D^^^W]^J)6 2 2*+ wHLK'(:+D5C]aekJ  )de c URX5nURURU[URSS5[URSS5S.n[ U[ 5(aURUUR5 gUR"XU40UD6n X R;aU RS5 URXTX5 g)Nis_loaded_in_8bitFis_loaded_in_4bit)r&init_ia3_weightsrrr) _check_target_module_feedforwardr&rEgetattrmodelr+r update_layerr?active_adaptersrequires_grad__replace_module) selfr6r7r8 target_nameparent current_keyrr9r<s r>_create_and_replaceIA3Model._create_and_replaces>>zW)77 * ; ;,%djj2EuM%djj2EuM   fh ' '   ++  006\U[\J#7#77))%0  j IrAc^[UR[5(a,[[R "URT55nU$[ U4SjUR55nU$)z| A helper private method that checks if the target module `key` matches with a feedforward module specified in `ia3_config` c3F># UHnTRU5v M g7fN)endswith).0 target_keykeys r> .s kLjjj!9!9Ljs!)r+feedforward_modulesrboolre fullmatchany)r6rYrs ` r>rF)IA3Model._check_target_module_feedforwardsY j44c : :!",,z/M/Ms"STN! kJLjLj kkNrAcURc2US[;a [S5e[[US5UlURc2US[ ;a [S5e[[ US5UlU$)N model_typez0Please specify `target_modules` in `peft_config`z5Please specify `feedforward_modules` in `peft_config`)target_modulesr r5setr\r ) peft_config model_configs r>_prepare_adapter_config IA3Model._prepare_adapter_configs  % % -L)1bb !STT),A,|B\]*K &  * * 2L)1gg !XYY.1F|T`Gab/K +rAc>[URSS5(a [S5e[URSS5(a [S5e[TU]"U0UD6$)as This method merges the (IA)^3 layers into the base model. This is needed if someone wants to use the base model as a standalone model. Args: safe_merge (`bool`, `optional`, defaults to `False`): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. rCFz>Cannot merge ia3 layers when the model is loaded in 8-bit moderDz>Cannot merge ia3 layers when the model is loaded in 4-bit mode)rGrHr5super_unload_and_optionally_merge)rMargsr9 __class__s r>rl%IA3Model._unload_and_optionally_mergesZ 4::2E : :]^ ^ 4::2E : :]^ ^w3TDVDDrAc\^^TH"nUTR;dM[SUS35e TR5Vs/sHn[U[5(dMUPM nn[ U4SjU55(a [S5eTVs1sH%n[ TRUR5iM' nnTVs1sH%n[ TRUR5iM' nn[U5S:d[U5S:a [S5e[U;aSRU4SjT55nO[R"U4S jT56n[U;aSRU4S jT55nXx4$[R"U4S jT56nXx4$s snfs snfs snf) zw Helper function to check if the arguments to add_weighted_adapter are valid and compatible with the underlying model. zAdapter z does not existc3V>^# UHm[U4SjT55S:v M g7f)c3@># UHoTR;v M g7frU)modules_to_save)rWadapterwrappers r>rZAIA3Model._check_add_weighted_adapter...sK(w7222(srN)sum)rWruadapterss @r>rZ7IA3Model._check_add_weighted_adapter..s& _wT[CK(K Ka O_ws%)zLCannot add weighted adapters targeting the same module with modules_to_save.rzQAll adapter configs should have the same type for target and feedforward modules.|c3^># UH"nSTRURS3v M$ g7f()NrfrdrWrtrMs r>rZrys0)riq^eAd.>.>w.G.V.V-WWX*Yiq*-c3V># UHnTRURv M g7frUrrs r>rZrys%,nemZaT-=-=g-F-U-Uem&)c3^># UH"nSTRURS3v M$ g7fr|rfr\rs r>rZrys0/T\!D$$W-AAB!DT\rc3V># UHnTRURv M g7frUrrs r>rZrys"XxG$""7+??xr)rfr5modulesr+r r`typerdr\lenrjoinreunion) rMrxrtmodulemodules_to_save_wrapperstarget_module_typesfeedforward_module_typesnew_target_modulesnew_feedforward_moduless `` r>_check_add_weighted_adapter$IA3Model._check_add_weighted_adapters  Gd... 8G9O!DEE :>#tv:V\^rKsF #t  _w   kl l^ff]eRYtD$4$4W$=$L$LM]efgo#pgo\cD)9)9')B)V)V$Wgo #p " #a '3/G+H1+Lpq q % %!$)riq)r!r !$,nem,n!o  * *&)hh/T\/' #":: '*iiXxX' #"::7$ug#psFF ,F$=,F)cVU[URR55;agURUS9upE[ URUSUUS9URU'UR UR U5 [UR U5 UR R5VVs/sHupgURU;dMUPM nnnUHn[UR U5upyn[U [5(dM3X9R;aU RUn OMTU RR5U l[!X5HHupXR;aU RU n OM&U =RU RU -- slMJ M gs snnf)a This method adds a new adapter by merging the given adapters with the given weights. Args: adapters (`list`): List of adapter names to be merged. weights (`list`): List of weights for each adapter. adapter_name (`str`): Name of the new adapter. N)rxr)rdr\)listrfkeysrrinject_adapterrHr named_modulesrrr+ria3_ldatazero_zip)rMrxweightsr7rrrY_key_listr8 target_ia3_lrtr1current_adapter_ia3_ls r>add_weighted_adapterIA3Model.add_weighted_adapter ss" 4 0 0 5 5 78 8 6:6V6V7W7 3*1   Xa[ )- 7* & DJJ 5  L1&*jj&>&>&@[&@FCDKKWZDZC&@[C*4::s;LAq&(++<</#)<< #=L$0$5$5$;$;$= !'*8'=OG,,.06 W0E-  %%)>)C)Cf)LL% (>\s 1F% F%)returnr])rx list[str]rztuple[str, str])rxrrz list[float]r7rrNone)__name__ __module__ __qualname____firstlineno____doc__r__annotations__rtuner_layer_cls staticmethodr?rQrFrhrlrr__static_attributes__ __classcell__)rns@r>rr$s"HFCOAAFJ@    E,&;P1M1M1M 1M  1M1MrAr) __future__rr^r3 dataclassesrr2transformers.pytorch_utilsrpeft.import_utilsrrpeft.tuners.tuners_utilsrr peft.utilsr r r r rlayerrrrrrrrAr>rsA#  -E>43WMyWMrA