oi~SSKJr SSKrSSKrSSKJr SSKJrJrJ r SSK r SSK J r SSK J s Jr SSK Jr SSKJr SSKJr SSKJrJrJr SS KJrJrJrJr SS KJr SS K J!r! S S K"J#r#J$r$ S/r%"SS5r&"SS\5r'"SS\ RP\'5r)"SS\ RP\'5r*"SS\ RP\'5r+"SS\+5r,"SS\+5r-"SS\+5r."SS \ RP\'5r/"S!S"\ RP5r0S#r1"S$S%\ RP\'5r2S'S(S&jjr3g))) annotationsN)contextmanager)AnyOptionalUnion) svd_lowrank)Conv1D) BufferDict)BaseTunerLayer_get_in_out_featurescheck_adapters_to_merge)dequantize_module_weightgather_params_ctxget_bnb_param_typeskip_init_on_device) transpose) PeftWarning) ArrowConfig LoraConfig alora_offsetsc\rSrSrSr\S Sj5r\S Sj5r\S Sj5r\S Sj5r \S Sj5r Sr g ) LoraVariant,ac Base class for LoRA variants, e.g. DoRA. This class should be subclassed and the methods below should be implemented accordingly. The methods should be implemented as static methods, this makes it easier to combine variants. Note for developers: These methods are prone to change and should thus considered to be "private". Use at your own discretion. c[e)zKInitialization code for the LoRA variant, it's called within `update_layer`NotImplementedError)module adapter_names P/home/james-whalen/.local/lib/python3.13/site-packages/peft/tuners/lora/layer.pyinitLoraVariant.init7 "!c[e)zZSafe merging of the weights from `merge(..., safe_merge=True)`, should return a new tensorrractive_adapter orig_weights r merge_safeLoraVariant.merge_safe<r#r$cg)zdUnsafe merging of the weights from `merge(..., safe_merge=False)`, should modify the weight in-placeNr&s r merge_unsafeLoraVariant.merge_unsafeAr$cg)zFRemove the adapter weights from the original weights, then return themNr,r&s r unmergeLoraVariant.unmergeEr/r$c [e)a The forward pass of the LoRA variant, should return the overall result (not just the diff) Args: module (LoraLayer): The module on which the forward pass is called active_adapter (str): The name of the active adapter x (torch.Tensor): The input to the forward call result (torch.Tensor): The result from the base model **kwargs: Additional arguments passed from [`LoraLayer.forward`]. r)rr'xresultkwargss r forwardLoraVariant.forwardIs $"!r$r,N)r LoraLayerrstrreturnNone)rr9r'r:r( torch.Tensorr;r=)rr9r'r:r(r=r;r<) rr9r'r:r4r=r5r=r;r=) __name__ __module__ __qualname____firstlineno____doc__ staticmethodr!r)r-r1r7__static_attributes__r,r$r rr,s""""ssUU""" " "  ""r$rc&\rSrSr%SrS\S'SrS\S'SSSjjrSSjrSSS jjr S r S r S r Sr Sr\R "5S5rSSjrSSjrS SjrS!SjrS"S#SjjrSrS$SjrSrg )%r9^)lora_Alora_Blora_embedding_Alora_embedding_Bztuple[str, ...]adapter_layer_names)r lora_alphascaling lora_dropoutother_param_namesc hXl0Ul0Ul0Ul[R "05Ul[R "05Ul[R "05Ul[R"05Ul [R"05Ul SUl /Ul 0Ul0Ul0Ul["RR 5Ul0UlX lSUl0UlX0lUR15n[3U5upEX@lXPlg)NFT) base_layerrLrMrNnn ModuleDictrOrGrH ParameterDictrIrJ_disable_adaptersmerged_adaptersuse_dora use_rslora lora_biastorchlora_magnitude_vector_cachesephemeral_gpu_offloadcast_input_dtype_enabled lora_variantr6get_base_layerr in_features out_features)selfrRr^r6rbrcs r __init__LoraLayer.__init__ds$ MM"-mmB' mmB' " 0 0 4 " 0 0 4!&!)+ +-*,%*XX%8%8%:"') +@".2%46 ((* $8$D! &(r$c g)aReturn a matching LoRA variant for this layer type. Given the init arguments of this layer, return the correct LoRA variant, if any. E.g., if `use_dora=True`, this method should return the DoRA variant for the given layer. If `use_alora=True`, same for aLoRA. If there is no fitting variant, return None. Note: If this layer type does not support the LoRA variant at all, please raise an error during __init__ as is convention, and not here. Nr,)rdrXr6s r resolve_lora_variantLoraLayer.resolve_lora_variantsr$Nc T [5R5nUS US::a[SU35eU (a\[UR 5SS5c@[ R "S[UR 55RS3[5 URUUU U U S9nUbXRU'X RU'X0RU'US:a[R"US 9nO[R "5nUR"R%[R&"UU055 [R("UR*US S 9UR,U'[R("X R.U S 9UR0U'XR2U'U(a'U[4R6"U5- UR8U'OX2- UR8U'X`R:U'XpR<U'[?U[@5(aUURCS 5(a?[EUR 5RF5 URIX5 SSS5 GO[?U[@5(aUURCS 5(a?[EUR 5RF5 URKX5 SSS5 GOD[?U[@5(aRURM5S:Xa>[EUR 5RF5 UROU5 SSS5 OUS:Xa>[EUR 5RF5 URQU5 SSS5 OUS:Xa7[RRRUUR0URF5 O\US:Xa>[EUR 5RF5 URWU5 SSS5 OU(aURYX5 UR[U5 XR;a URURR"U40UD6 UR]UR^U S9 [aUS5(aYURHHnUURb;dMURbUReUR,UR05 MJ gg!,(df  N=f!,(df  N=f!,(df  N=f!,(df  GN=f!,(df  GN=f)Nrdr?`r` should be a positive integer value but the value passed is bias;`lora_bias=True` was passed but the targeted layer of type E has no bias. This means that merging LoRA weights won't be possible.)rX use_alora use_qaloraqalora_group_size arrow_configpFrlpissacordaoloraloftqeva orthogonalinference_mode lora_arrow)3localscopy ValueErrorgetattrrawarningswarntyper>rrhr`rLrMrSDropoutIdentityrOupdaterTLinearrbrGrcrHrZmathsqrtrNrYrX isinstancer: startswithrweight pissa_init corda_initlower olora_init loftq_initr!zeros_orthogonal_initreset_lora_parameters%_move_adapter_to_device_of_base_layer set_adapteractive_adaptershasattrron_adapter_change)rdrrLrMrOinit_lora_weightsrYrXrorprZrrrqr~r6r`lora_dropout_layeradapters r update_layerLoraLayer.update_layers$ 6N 6^_`^abc c '$"5"5"7FN MMMdSWSfSfShNiNrNrMstWW  00!/% 1   #.:  l + |(2 % # !#l!; !#     >P/Q!RS%'IId.>.>$N L!$&IIa1B1B$S L!'0|$ )3diil)BDLL &)3DLL &(2 %&. l# ' - -2C2N2Nw2W2W"4#6#6#8#?#?@ @A@ )3 / /4E4P4PQX4Y4Y"4#6#6#8#?#?@ @A@ )3 / /4E4K4K4MQX4X"4#6#6#8#?#?@ -A@ ' )"4#6#6#8#?#?@ -A@ % ' GGNN4;;|4;; < , ."4#6#6#8#?#?@$$\2A@   & &| G 22<@ ,, ,   l + 0 0 @ @ --nM 4 & &,,doo-OOG,>>t{{DKKX- 'AA@A@A@A@ A@s<S.S$S5TT S!$ S25 T T T'c>USLagXRR5;Ga8USLaJ[RR URUR [ R"S5S9 OiUR5S:XaF[RRURUR SURU- S9 O[SU<35e[RRURUR 5 URU(a6[RRURUR5 XR R5;a[RRUR U5 [RRUR"U5 URU(a7[RRUR"UR5 ggg) NFT)agaussianr)stdz)Unknown initialization init_lora_weights=)rGkeysrSr!kaiming_uniform_rrrrnormal_rLrrrHrZrlrIrJ)rdrrs r rLoraLayer.reset_lora_parameterss  %  ;;++- - D(((\)B)I)ITYYWX\(Z"((*j8 L 9 @ @a$&&Q]J^F^_ #M;L:N!OPP GGNN4;;|4;; <~~l+t{{<8==> 00557 7 GGNN400> ? GGOOD11,? @~~l+t44\BGGH, 8r$cUR5nURn[U5nURnU(a [ U5nOEU[ R [ R[ R4;aUnO[SUS35eURUnURUnUR[ R 5n[ RRUR5upU SS2SU24U SUpU R!5UR"URlU R!5UR$URlU=RXpR$UR-UR"UR--slUS:Xa[UR'UUR(UR*UR,UR.S9RUR05nXblgUS:XaEUR'UUR2UR4S9RUR05nXblgURU5nXbRlg)Nz.Unsupported data type for the base layer. Got .4bit) quant_type quant_storagecompress_statisticsr8bit) requires_gradhas_fp16_weights)rarrdtyperr[float32float16bfloat16 TypeErrorrNrLtolinalgqrdata contiguousrGrH __class__rrrrdevicerr) rdrrRr(bnb_param_typer weight_tensor scale_factorrLQRQrRrs r rLoraLayer.olora_inits((*  '' +K8!! 4Z@M u}}emmU^^D D'MLUGSTUV V||L1 FF< %((7 ||}1121bqb51Ra5B02  L!((-02  L!((-l[[-F-M-MMPTP[P[\hPiPpPppp V #'11&11)77$/$C$C")) 2 b##$ !.  v %'11)77!,!=!=2b##$  !. ),,U3M%2   "r$c @UR5RnURnU[R[R [R 4;a [S5e[UR[R5UR5nUS:Xay[RRURSS9upVnUSS2SURU24nUSURUn XRU-n USURUn O[!UR#S55S:Xab[%URURU['UR#S5S5S9upn XRU-n U R)5n O[+S US 35e[R,"[R."U 55U -n U[R,"[R."U 55-n XR0URl XR2URl URURUU -U -- n[URU5UR5nX0R5Rl g) NzPlease initialize PiSSA under float32, float16, or bfloat16. Subsequently, re-quantize the residual model to help minimize quantization errors.rwF) full_matrices_niter_)niterzLinit_lora_weights should be 'pissa' or 'pissa_niter_[number of iters]', got instead.)rarrr[rrrrrrfan_in_fan_outrsvdrrLrNlensplitrinttrdiagrrGrH)rdrrrrVSUhVrSrUhrUrrGrHs r rLoraLayer.pissa_init>s2$$&--   u~~F Fe 699U]]3T5H5HI  '||'' 5'IHA"1,|,,,-B)TVVL)*B ,,|, ,B+tvvl+,C "((3 4 9$ TVVL1=N=T=TU^=_`b=c9dJBB ,,|, ,B$$&C^_p^qqz{ EJJrN+c1ejjB0006 L!((-06 L!((-t||L9FBVKK699U+T-@-@A,2$$)r$c (UR5nURnURnU[R[R [R 4;a [S5eUR[R5nURRS5nURRS5n[US5(d [S5eURnURn URn UR n UR"Un [R$"U 5R'5(d)[R("U 5R'5(a [S5e[R$"U 5R'5(d)[R("U 5R'5(a [S5e[R$"U 5R'5(d)[R("U 5R'5(a [S5eU RS5U:wdU RS5U :wa#[S U R5S US U S 35eU RS5U :wa [S U R5S U S35eU RS5U:wdU RS5U :wa#[SU R5S US U S 35eXR*U-n U R-5R/U R15R3SS55R55n U R/U R155R55nXR6URl XR8URl URUR*UU-U -- nURU5nX@R5Rl U? g)NzPlease initialize CorDA under float32, float16, or bfloat16. Subsequently, re-quantize the residual model to help minimize quantization errors.rreigensz`eigens` attribute not found for layer, please run `preprocess_corda` first. More information can be found at examples/corda_finetuning/README.md.zdInvalid value found in matrix S. Please file an issue at https://github.com/huggingface/peft/issues.zdInvalid value found in matrix U. Please file an issue at https://github.com/huggingface/peft/issues.zdInvalid value found in matrix V. Please file an issue at https://github.com/huggingface/peft/issues.zMatrix U size mismatch: z vs. (z, z). Please make sure the `lora_config` and `model` argument of `preprocess_corda` is consistent with `get_peft_model`. If you're using cache in `preprocess_corda`, please make sure the cache is built with the same model and LoRA rank.zMatrix S size mismatch: z,). Please make sure the `lora_config` and `model` argument of `preprocess_corda` is consistent with `get_peft_model`. If you're using cache in `preprocess_corda`, please make sure the cache is built with the same model and LoRA rank.zMatrix V size mismatch: r)rarrr[rrrrrrsizerrrU_WCS_WCV_WCrLisnananyisinfrNrmulrviewrrGrH)rdrrlinearrrout_dimin_dimrUrrrLrGrHs r rLoraLayer.corda_initasV$$&   u~~F Fe 5==)++""1%!!!$vx((X  KK KK KK FF<  ;;q>    5;;q>#5#5#7#7v  ;;q>    5;;q>#5#5#7#7v  ;;q>    5;;q>#5#5#7#7v  66!9 166!9>*1668*F7)2aSIpp  66!9>*1668*F1#>YY  66!9 !&&)q.*1668*F6("QCHpp  \\, ''1668==Q/0;;=qvvx++-06 L!((-06 L!((-t||L9FBVKK5!,2$$) Mr$cSSKJn UR5RnURR SS5UR UURR SS5S.nU"U40UD6upVnXRR5;a:X`RURl XpRURl XRR5;a:X`RURl XpRURl XPR5Rl g)Nr)r loftq_bits loftq_iterr)num_bits reduced_ranknum_iter) peft.utils.loftq_utilsrrarr6getrLrGrrrHrIrJ)rdrrrr6qweightrGrHs r rLoraLayer.loftq_inits5$$&--  a8 FF<0  a8  #-V">v"> ;;++- -4:KK % , , 14:KK % , , 1 00557 7>D ! !, / 6 6 ;>D ! !, / 6 6 ;,3$$)r$cLURUnUS-S:wa[SUS35e[R"X"5n[RR U5upEUSSS2SS24nUSSS2SS24nUR 5RRn[R"URUS-5RU5RS- n [R"US-UR5RRU5S- n [R"U R5R!U55UR"Ul[R"U R5R!U55UR$Ulg)NrrzAOrthogonal initialization requires the LoRA rank to be even, got rrg$@)rLrr[randnrrrarrrbmmTrcrS ParameterrrrGrH) rdrrankXr_q_oddq_evenrrGrHs r rLoraLayer.orthogonal_initsSvvl# !8q=`ae`ffopq q KK #||q!!$Q$' 14a47##%,,22T--tqy9<3RUYU^U^_c_e_eft_uUv3vDLL037??>3RUYU[U[\jUk3kDLL0/3||N/Ke/S ^,3r$cURSS5nUcg[U5[U5:wa&S[U5S[U5S3n[U5eUR(a Sn[U5eUVs1sH ofS:wdM UiM nnUH1nURRUS5(dM&S n[U5e gs snf) MCheck if the arguments are compatible with the configs and state of the model adapter_namesNzNLength of `adapter_names` should be the same as the number of inputs, but got z and z respectively.z`Cannot pass `adapter_names` when there are merged adapters, please call `unmerge_adapter` first.__base__Fz1Cannot pass `adapter_names` when DoRA is enabled.)rrrmergedrX) rdr4argsr6rmsgnameunique_adaptersrs r _check_forward_argsLoraLayer._check_forward_argss ?D9   q6S' '`}%&eCF8>C S/ ! ;;uCS/ !-:PMDZ=O4MP+L}}  u55I o%,Qs 7 CCc [Vs0sHoUURUS5_M nnUR"U/UQ70UD6nURn[ U5n /n U H9n U R [ U5V V s/sHupX:XdM U PM sn n 5 M; URSS5n[ U 5GH;unnUS:XaMUURR5;aM/URUnURUnURUnURUnXURURR5nUUR;a7U"U"U"U555U-nXzU==URU5- ss'MUbXVs/sHnUUPM snUS'URUR "U4UUXzUS.UDUD6nURU5XzU'GM> U$s snfs sn n fs snf)Nrrr'r4r5)VARIANT_KWARG_KEYSr rRrsetappend enumeraterrGrrHrOrNrrr`r7)rdr4rrr6kvariant_kwargsr5torch_result_dtyper"sub_batch_indices_listrindexitemrir'rGrHdropoutrN sub_batch lora_outputjs r _mixed_batch_forwardLoraLayer._mixed_batch_forwards! ;MM:LQVZZ400:LM4T4V4#\\m,!#&G " ) )ImD\*pD\[U`d`o5D\*p q'&**?DA !*?!; A~+T[[%5%5%77[[0F[[0F''7Gll>2G3477 8K8KLIT%6%66$VGI,>%?@7J a01[^^DV5WW1 ,QgQj6kQjA}Q7GQj6kN?3"//?GG#1!";<  %    5@NNCU4Va019"<< ON+q*7lsG44 G9 G9 G?)r]rVrRr_r^rbr6rGrHrMrZrOrIrJr\r`rWrcrLrNrXrY)F)rR nn.Moduler^boolr;r<rXr9r;Optional[LoraVariant])FFFFN F)rXr9ror9rpr9rZr9rrrrqrr~r9)rr:rrr;r<)rr:r;r)rr:r float | intr;r<)rr=r;r<r)rzOptional[float | int]r;r< r4r=rrr list[str]r6rr;r=)r>r?r@rArK__annotations__rPrerhrrrrrrr[no_gradrr r rrrr#r6rDr,r$r r9r9^s+gg)WW)< , $(!#$fYfYfYfYfY"fYfYfYPI2+3Z!3FHT4* ]]_WW " W 2T&&6,,&),:C,OR, ,r$r9c^\rSrSrS S U4SjjjrS SjrS SSjjrSSjrSSjrSSjr SU4Sjjr S r U=r $)riWc >[TU]5 [R"X40UD6 X`lX lUR UUUUUU U U U U S9 Xplg)N)rMrOrrYrXrorZrr)superrer9r_active_adapterris_target_conv_1d_layer)rdrRrrLrMrOrrFrrYrXrorrrZr6rs r reLinear.__init__Ysh" 46v6,+   !%/!%  (?$r$c zUb SSKJn U"5$U(dU(dgSSKJnJn U(aU"5$U"5$)Nr)ArrowLinearVariant)ALoraLinearVariantDoraLinearVariant)variantsrIrJrK)rdrrrXror6rIrJrKs r rhLinear.resolve_lora_variant}s6  # 4%' ' C %' '$& &r$c[X5nU(dgUGHnX0RR5;dM#UR5nU(GaURR R 5nURnX0R;a%URU5nXWRU5- nOURURXU5n[R"U5R5(d[SUS35eXTRlUR U(a[#USS5c [%S5eUR&UR(UR&UR*U--n[R"U5R5(d[SUS35eURU5UR&lOX0R;a1URU5nUR=R U- slO)URUR-XUR5 UR U(a_[#USS5c [%S5eUR&=R UR(UR&UR*U-- slUR.R1U5 GM g) Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. N1NaNs detected in the merged weights. The adapter  seems to be brokenrlRImpossible to merge LoRA with `lora_bias=True` because the base layer has no bias.)r rGrrarrclonerr`get_delta_weightrr)r[isfiniteallrrZr RuntimeErrorrlrHrNr-rWr)) rd safe_mergerr'rRr( orig_dtype delta_weightnew_biass r merge Linear.mergesn0D  +N!1!1!33!002 #-"3"3"8"8">">"@K!,!2!2J%->->>'+'<'<^'L #z'BB &*&7&7&G&R&RSWit&u  >>+6::<<(OP^O__rs.9%%*~~n5":vt<D". t#$.??T[[5P5U5UX\XdXdesXt5t#t$~~h7;;==","STbSccv w#08{{:/F ,&->->>'+'<'<^'L "))..,>.)).9FFt]g]n]no~~n5":vt<D". t##,, N0K0P0PSWS_S_`nSo0oo,$$++N;_,r$c UR(d[R"S5 g[UR5S:GaLURR 5nXR R5;aUR5RnXR;aBURnURU5nU=RURU5-sl O%URURXU5nXRl UR U(aTUR5R"=RUR$UR"UR&U--sl [UR5S:aGMKggG This method unmerges all merged adapter layers from the base weights. Already unmerged. Nothing to do.NrrrrrrWr rGrrarr`rrTrrr1rZrlrHrNrdr'rrYrZunmergeds r r1Linear.unmerges- {{ MM< = $&&'!+!11557N!1!1!33,,.55!):)::!'J#'#8#8#HLKK>K#00@HH_efH"*K>>.1'')..33t{{>7R7W7WZ^ZfZfguZv7vv3$&&'!++r$cURURRnURURRnURS:H=(a- U[ R :H=(d U[ R:HnURURnURURnU(a UR5nUR5n[Xe-UR5URU-nU(aiURUS9nURU5URURlURU5URURlU$ Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. cpur)rHrrrrr[rrrGfloatrrrNrrrdrrr cast_to_fp32weight_Aweight_B output_tensors r rTLinear.get_delta_weights,W%,,33 G$++11 {{e+c%--1G1b5TYTbTbKb ;;w'..;;w'.. ~~'H~~'H!("5t7J7JKdll[bNcc ),,5,9M08{{5/ADKK ' ' ,/7{{5/ADKK ' ' ,r$c UR"U/UQ70UD6 URSS5n[Vs0sHoUURUS5_M nnUR(a9UR(aUR 5 UR "U/UQ70UD6nU$UbUR"U/UQ7SU0UDUD6nU$UR(aUR "U/UQ70UD6nU$UR "U/UQ70UD6nURnURR5n URHn X;aM URU n URU n URU n URU nURXR R5nXR";aX|"U "U "U555U--nMUR"U R$"U4U UUS.UDUD6nM UR'U5nU$s snfNrr&)r#r r'disable_adaptersrr1rRr6rrGrrrHrOrN_cast_input_dtyperr`r7r)rdr4rr6rr+r,r5r- lora_A_keysr'rGrHr2rNs r r7Linear.forward s   4T4V4 ?D9 :LM:LQVZZ400:LM  {{ __Q888FB A &..qq4q}qXfqjpqF> =[[__Q888F: 7__Q888F!' ++**,K"&"6"6!4^4^4++N;,,~6**1mm.A.AB!):)::#fVGAJ-?&@7&JJF!..~>FF'5%  )  ! F#7*YY12F MNsG;c*>[TU]5nSU-$Nlora.rD__repr__rdreprs r r|Linear.__repr__6g }r$)rErrF) rrrsFFTFFFNF)rr:rLrrMrrOrkrr9rFr9rUnion[bool, str]rYr9rXr9ror9rrrrZr9r;r<)rrrrXr9ror9r;r;FNrXr9rOptional[list[str]]r;r<r;r<r;r=r4r=rrr6rr;r=r;r:) r>r?r@rArerhr\r1rTr7r|rD __classcell__rs@r rrWs !$(-.2 $("?"?  "?  "?  "?"?"&"?,"?"?"?"?""?"? !"?"?H'*'6:'GK' '$AU (a&[SU SURRS35e[T U]5 [ R X5 X`lX lURUUUUUUU U U S9 g)Nz lora_bias=z is not supported for rrMrOrrYrXrZrr) rrr>rDrer9rrEr)rdrRrrLrMrOrrrYrXrrrZr6rs r reEmbedding.__init__=s z)4J4>>KbKbJccdef f 4,,+   !%/!%  r$c ,U(dgSSKJn U"5$)Nr)DoraEmbeddingVariant)rLr)rdrXr6rs r rhEmbedding.resolve_lora_variantas2#%%r$c  N[5R5n U S US::a[SU35eURXyS9n U bXRU'X R U'X0R U'US:a[R"US9n O[R"5n XRU'[R"X R45n[R"URU45n[R"U5UR U'[R"U5UR"U'XR$U'U(a'U[&R("U5- UR*U'OX2- UR*U'X`R,U'XpR.U'US:XaUR1U5 OU(aUR3X5 UR5U5 XR;a URUR6"U40U D6 UR9UR:U S9 g) NrdrrkrXrrrsrtrzr})rrrrhr`rLrMrSrrrOr[rrbrcrrIrJrZrrrNrYrXrrrr!rr)rdrrLrMrOrrYrXrZrrr~r6r`rrnros r rEmbedding.update_layeris 6N 6^_`^abc c00(0^  #.:  l + |(2 % # !#l!; !# *<,';;#3#345;; 1 1156.0ll8.Dl+.0ll8.Dl+'0|$ )3diil)BDLL &)3DLL &(2 %&. l#  ' OOL )   & &| G 22<@ ,, ,   l + 0 0 @ @ --nMr$cv[X5nU(dgUGHnX0RR5;dM#UR5nURR nU(aURR R5nX0R;a#X`RU5RU5- nOURURXU5n[R"U5R5(d[SUS35eXdRlOvX0R;a>UR=R URU5RU5- slO)URUR!XUR5 UR"R%U5 GM g)rONrPrQ)r rIrrarrrrSr`rTrr)r[rUrVrr-rWr))rdrXrr'rRrYr(s r r\Embedding.mergesm0D  +N!6!6!;!;!==!002 '..44 #-"3"3"8"8">">"@K%->->>#'<'<^'L'O'OPZ'[[ &*&7&7&G&R&RSWit&u  >>+6::<<(OP^O__rs.9%%*%->->>"))..$2G2G2W2Z2Z[e2ff.)).9FFt]g]n]no$$++N;1,r$cxUR(d[R"S5 g[UR5S:aURR 5nUR 5RRnXRR5;aUR 5RnXR;a4U=RURU5RU5-sl O%URURXU5nXCl [UR5S:aMggr_)rrrrrWr rarrrIrr`rrTrr1)rdr'rYrrds r r1Embedding.unmerges{{ MM< = $&&'!+!11557N,,.55;;J!6!6!;!;!==,,.55!):)::KK4#8#8#H#K#KJ#WWK#00@HH_efH"*K$&&'!+r$cZURURnURURnURS:H=(a- U[ R :H=(d U[ R:HnURUnURUnU(a UR5nUR5n[Xe-S5URU-nU(aKURUS9nURU5URU'URU5URU'U$)rhriTrj) rJrrIrrr[rrrkrrNrrls r rTEmbedding.get_delta_weights&&w/66%%g.44 {{e+c%--1G1b5TYTbTbKb ((1((1 ~~'H~~'H!("5t2G34Ikk)[9G% 3w>N&!/+..AUAU2V!V !, - ? -)"<, 1+qs  E !E c UR5n[R"UUURURUR UR URS9$)N) padding_idxmax_norm norm_typescale_grad_by_freqsparse)raF embeddingrrrrr)rdinputrrRs r rEmbedding._embed/sT((* {{  "..(( **)<<$$  r$cUR"U/UQ70UD6 URSS5n[Vs0sHoUURUS5_M nnUR(a9UR(aUR 5 UR "U/UQ70UD6nU$UbUR"U/UQ7SU0UD6nU$UR(aUR "U/UQ70UD6nU$UR "U/UQ70UD6nURnUR5n URHn XR;aMXR;aURU Rn URU Rn URU n UR!X5nX-U -nU bXR#UR5-nX-nMURU R$"U4U UUS.UDUD6nM UR#U5nU$s snfrs)r#r r'rtrr1rRr6rrrrIr`rrJrNrrr7)rdr4rr6rr+r,r5r-rr'rrrNrrs r r7Embedding.forward;s   4T4V4 ?D9 :LM:LQVZZ400:LM  {{ __Q888FP O &..q_4_}_X^_FL K[[__Q888FH E__Q888F!' //1K"&"6"6!)>)>>!):)::"&"7"7"G"I"IK"&"7"7"G"I"IK"ll>:G"kk!9G&-&;w%FN#.)7..I]I]:^)^#4F!..~>FF'5%  )  ! F##72YY12F YNsG<c*>[TU]5nSU-$ryr{r}s r r|Embedding.__repr__mrr$)rEr) rrrsFTFFNF)rRr8rr:rLrrMrrOrkrr9rrrYr9rXr9rrrrZr9r;r<r:NFrrrr~r9rrrrr>)rr=rr=r;r=rr)r>r?r@rArerhrr\r1rTr6rr7r|rDrrs@r rr;s !$.2 $(" " "   "  "  " " ," " " "" "  " " H&$%)$<N"<N<N|*[T U]5 [RX5 U RSS5(a [ S5eUR S:a[ R"S5 X1R -S:wa3[ SURRSUR S US 35eX l URR5Ul URUUUUUUUU U S 9 g) NroFz,aLoRA does not support adapting conv layers.rzMLoRA adapter added to ConvNd layer with groups > 1. Merging is not supported.rz Targeting a z with groups=z and rank z. Currently, support is limited to conv layers where the rank is divisible by groups. Either choose a different rank or do not target this specific layer.r)rDrer9rrgroupsrrrr>rErdim _kernel_dimr) rdrRrrLrMrOrrYrXrrrZr6rs r re_ConvNd.__init__ts 4, ::k5 ) )KL L   q MMi j   A %z33<<=]:K\K\J]]ghigjkWW  ,%,,002   !%/!%  r$c @[5R5n U S US::a[SU35eU(a\[UR 5SS5c@[ R "S[UR 55RS3[5 URXyS9n U bXRU'X RU'X0RU'US:a[R"US 9n O[R "5n XR"U'UR 5nUR$nUR&nUR(n[U5nS UR*S - -=nnU"UR,X/UUS S 9UR.U'U"X R0UUUR2US9UR4U'XR6U'U(a'U[8R:"U5- UR<U'OX2- UR<U'X`R>U'XpR@U'US:XaURCU5 OU(aUREX5 URGU5 XR;a URURH"U40U D6 URKURLU S9 g)NrdrrkrlrmrnrrsrtrrFrv)rrlrzr})'rrrrrarrrr>rrhr`rLrMrSrrrO kernel_sizestridepaddingrrbrGrcrrHrZrrrNrYrXrrrr!rr)rdrrLrMrOrrYrXrZrrr~r6r`rrRrrr conv_layer out_kernel out_strides r r_ConvNd.update_layersg 6N 6^_`^abc c '$"5"5"7FN MMMdSWSfSfShNiNrNrMstWW  00(0^  #.:  l + |(2 % # !#l!; !# *<,'((*  ,, ""$$*% "&$*:*:Q*>"?? Z$.t/?/?QWY`gl$m L!$.   *jARARYb%  L!(1|$ )3diil)BDLL &)3DLL &(2 %&. l#  ' OOL )   & &| G 22<@ ,, ,   l + 0 0 @ @ --nMr$c,SSURS- --$)N)rrr)rrds r _get_dora_factor_view_ConvNd._get_dora_factor_viewstt//!3444r$c[X5nU(dgUGHnX0RR5;dM#UR5nURR nUR S:a [S5eU(Ga{URRR5nX0R;a%URU5nXgRU5- nOURURXU5n[R"U5R!5(d[#SUS35eXdRlUR$U(a['USS5c [)S5eUR*UR,UR*UR.U--n[R"U5R!5(d[#SUS35eURU5UR*lOX0R;a@URU5nUR=RURU5- slO)URUR1XUR5 UR$U(a_['USS5c [)S5eUR*=RUR,UR*UR.U-- slUR2R5U5 GM g)a Merge the active adapter weights inside the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`list[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Nrz 1!rPrQrlrR)r rGrrarrrrrrSr`rTrr)r[rUrVrrZrrWrlrHrNr-rWr)) rdrXrr'rRrYr(rZr[s r r\ _ConvNd.merges0D  +N!1!1!33!002 '..44 $$q(-.lmm#-"3"3"8"8">">"@K%->->>'+'<'<^'L #z'BB &*&7&7&G&R&RSWit&u  >>+6::<<(OP^O__rs.9%%*~~n5":vt<D". t#$.??T[[5P5U5UX\XdXdesXt5t#t$~~h7;;==","STbSccv w#08{{:/F ,&->->>'+'<'<^'L "))..,//*2MM.)).9FFt]g]n]no~~n5":vt<D". t##,, N0K0P0PSWS_S_`nSo0oo,$$++N;i,r$c UR(d[R"S5 g[UR5S:GaLURR 5nXR R5;aUR5RnXR;aBURnURU5nU=RURU5-sl O%URURXU5nXRl UR U(aTUR5R"=RUR$UR"UR&U--sl [UR5S:aGMKggr_rbrcs r r1_ConvNd.unmerge5s-{{ MM< = $&&'!+!11557N!1!1!33,,.55!):)::!'J#'#8#8#HLKK>K#00@HH_efH"*K>>.1'')..33t{{>7R7W7WZ^ZfZfguZv7vv3$&&'!++r$cURURRnURURRnUR S:H=(a- U[ R:H=(d U[ R:HnURURnURURnU(a UR5nUR5nUR5RR5SSS:XapURS5RS5URS5RS5-RS5RS5URU-nOtURUR!SS5U5nUR5R"S:aXpRU-nO"UR!SS5URU-nU(aiUR%US9nUR%U5URURlUR%U5URURlU$) rhrirr)rrrrrj)rHrrrGrrr[rrrkrarsqueeze unsqueezerNconv_fnrrrrrls r rT_ConvNd.get_delta_weightKsW%,,33 G$++11 {{e+c%--1G1b5TYTbTbKb ;;w'..;;w'.. ~~'H~~'H    ' ' , , .q 3v =%--a088;h>N>Nq>Q>Y>YZ[>\\gghijtt W%&M!LL););Aq)A8LM""$++a/ - W0E E - 7 71 = W@U U ),,5,9M08{{5/ADKK ' ' ,/7{{5/ADKK ' ' ,r$c UR"U/UQ70UD6 URSS5n[Vs0sHoUURUS5_M nnUR(a9UR(aUR 5 UR "U/UQ70UD6nU$UbUR"U/UQ7SU0UD6nU$UR(aUR "U/UQ70UD6nU$UR "U/UQ70UD6nURnURHn XRR5;aM"URU n URU n URU n URU n URXR R5nXR";aX{"U "U "U555U --nMUR"U R$"U4U UUS.UDUD6nM UR'U5nU$s snfrs)r#r r'rtrr1rRr6rrrGrrHrOrNrurr`r7r)rdr4rr6rr+r,r5r-r'rGrHr2rNs r r7_ConvNd.forwardys   4T4V4 ?D9 :LM:LQVZZ400:LM  {{ __Q888F@ ? &..q_4_}_X^_F< ;[[__Q888F8 3__Q888F!' "&"6"6!)9)9);;^4^4++N;,,~6**1mm.A.AB!):)::#fVGAJ-?&@7&JJF!..~>FF'5%  )  ! F#7*YY12F INsG7c*>[TU]5nSU-$ryr{r}s r r|_ConvNd.__repr__rr$)rEr)rrrsTFFNF)rRr8rr:rLrrMrrOrkrrrYr9rXr9rrrrZr9r;r<rrrrrr)r4r=r;r=r)r>r?r@rArerrr\r1rTr7r|rDrrs@r rrrs !.2 $() ) )   )  )  ) ,) ) ) ") )  ) ) j%)$IN"ININV5F[TU]"U0UD6 URS:Xd[SUR35e[R Ulg)Nrz0Conv2d layer kernel must have 4 dimensions, not )rDrerrrconv2drrdrr6rs r reConv2d.__init__I $)&)1$OPTP`P`Oabc cxx r$c ,U(dgSSKJn U"5$)Nr)DoraConv2dVariant)rLr)rdrXr6rs r rhConv2d.resolve_lora_variant/ ""r$rr:r>r?r@rArerhrDrrs@r rr ##r$rc2^\rSrSrU4SjrSSjrSrU=r$)Conv1dic>[TU]"U0UD6 URS:Xd[SUR35e[R Ulg)Nrz0Conv1d layer kernel must have 3 dimensions, not )rDrerrrconv1drrs r reConv1d.__init__rr$c ,U(dgSSKJn U"5$)Nr)DoraConv1dVariant)rLr)rdrXr6rs r rhConv1d.resolve_lora_variantrr$rr:rrs@r rrrr$rc2^\rSrSrU4SjrSSjrSrU=r$)Conv3dic>[TU]"U0UD6 URS:Xd[SUR35e[R Ulg)Nrz0Conv3d layer kernel must have 5 dimensions, not )rDrerrrconv3drrs r reConv3d.__init__rr$c ,U(dgSSKJn U"5$)Nr)DoraConv3dVariant)rLr)rdrXr6rs r rhConv3d.resolve_lora_variantrr$rr:rrs@r rrrr$rc^\rSrSrSrSS U4Sjjjr\S!Sj5r\S"Sj5r\S"Sj5r \S#Sj5r \S!Sj5r \S$S j5r \S#S j5r \S!S j5r\S%S j5r\S%S j5r\S&Sj5r\S'Sj5r\S'Sj5rS(Sjr\S#Sj5rS)U4SjjrS*S+SjjrS)SjrS,SjrS-SjrU4SjrS.Sjr\S5rU4SjrU4Sjr S/U4Sjjr!Sr"U=r#$)0MultiheadAttentionia^LoRA implemented in a multihead attention layer This is currently only implemented for the case of `_qkv_same_embed_dim = True`, i.e. query, key, and value having the same dimension. Note: LoRA is applied to both the in_proj (query/key/value) and out_proj. There is currently no way to specify only one of them. Don't try to apply LoRA to the out_proj of MultiheadAttention by targeting that layer specifically, since the forward method of that layer is not being used, hence the LoRA adapter would be ignored. This is a little bit hacky because of the way that MultiheadAttention is implemented in PyTorch: There are no `nn.Linear` layers which we can hook onto or, in case of output projection, `.forward` is not used. This implementation works around these problems by merging the weights before the forward call and unmerging them after the forward call. c >[USS5(d#[SURRS35eU(a"[URRS35eU R SS5(a"[URRS35e[ T U]5 [R "X40U D6 [UR[R5(a.[URU4UUUUUUS .U D6URl O#[S URRS35eX l URX#XEXg5 g) N_qkv_same_embed_dimTz?Only same embed for query/key/value is supported as of now for rz: does not support DoRA (yet), please set use_dora to FalseroFz< does not support aLoRA (yet), please set use_alora to False)rLrMrOrrYrXz.out_proj must be an instance of nn.Linear for )rrrr>rrDrer9rout_projrSrrRrEr) rdrRrrLrMrOrrYrXr6rs r reMultiheadAttention.__init__sDz#8$??QRVR`R`RiRiQjjkl   7 788rst t ::k5 ) ) 7 788tuv v 46v6 j))299 5 5'-## (%)"3%! ( (DOO $MdnnNeNeMffghi i+ ,:EVcr$c6UR5R$r)ra embed_dimrs r rMultiheadAttention.embed_dim""$...r$c6UR5R$r)rakdimrs r rMultiheadAttention.kdim""$)))r$c6UR5R$r)ravdimrs r rMultiheadAttention.vdim rr$c6UR5R$r)rarrs r r&MultiheadAttention._qkv_same_embed_dim$s""$888r$c6UR5R$r)ra num_headsrs r r MultiheadAttention.num_heads(rr$c6UR5R$r)rar2rs r r2MultiheadAttention.dropout,s""$,,,r$c6UR5R$r)ra batch_firstrs r rMultiheadAttention.batch_first0s""$000r$c6UR5R$r)rahead_dimrs r rMultiheadAttention.head_dim4s""$---r$c6UR5R$r)rain_proj_weightrs r r!MultiheadAttention.in_proj_weight8s""$333r$c6UR5R$r)ra in_proj_biasrs r rMultiheadAttention.in_proj_bias<s""$111r$cRUR5RR5$r)rarrs r rMultiheadAttention.out_proj@s ""$--<<>>r$c6UR5R$r)rabias_krs r rMultiheadAttention.bias_kD""$+++r$c6UR5R$r)rabias_vrs r r MultiheadAttention.bias_vHrr$cBUR5R"U0UD6$r)ra merge_masks)rdrr6s r r#MultiheadAttention.merge_masksLs!""$00$A&AAr$c6UR5R$r)ra add_zero_attnrs r r& MultiheadAttention.add_zero_attnOs""$222r$cr>[TU]"U0UD6 URRR"U0UD6 gr)rDrrRrrs r rMultiheadAttention.update_layerSs2 d-f-   --t>v>r$c*[X5nU(dgUGHnX0RR5;dM#UR5nURR R nU(GaURRR5R5nX`RU5RU5- n[R"U5R5(d[!SUS35eURR RR5R5nXtRRU5RU5- n[R"U5R5(d[!SUS35eU?XdlURR5?XtRR5lURR#U/S9 OURU5RU5nURRR5U-n U?XlURRU5RU5nURR RR5U-n URR5?XRR5lURR#U/S9 UR$R'U5 GM g)a Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. NrPrQr)r rGrrarrrrrdetachrSrTrr[rUrVrr\rWr)) rdrXrr'rRrYorig_weight_inorig_weight_outrZ weight_mergeds r r\MultiheadAttention.mergeXs0D   ,N!1!1!33!002 '0077== &0%>%>%C%C%J%J%L%R%R%TN"&;&;N&K&N&Nz&ZZN >>.9==??(OP^O__rs '1&9&9&@&@&E&E&L&L&N&T&T&VO#':':'K'KN'['^'^_i'jjO >>/:>>@@(OP^O__rs #10>-"++::<CBQ''668?''--^$>$C$C$J$J$L|$[M"++::<CBO''668?''--^ Zgl@mn *22CCNSVVWab '00;;BBGG,V ''229##..AAbll:UK!$&&'!+( &&..0r$cU(aURX#S9 UR5nURnU?URS[R "UR URS95 URR5nURnU? URS[R "UR URS95 XdlU$)z Merging and unloading of the MultiheadAttention module This requires an extra step for MultiheadAttention, which is why there is this special method instead of relying on the normal merge_and_unload code path. rXrrr2r) r\rarr3rSrrrrr)rdr\rXrrRrout_proj_layers r "unload_and_optionally_merge_module5MultiheadAttention.unload_and_optionally_merge_modules  JJ*J J((* **  %%%&6 V[[`f`t`t8uv$,,;;=&&  !))(BLL\b\p\p4qr,r$cURURRnURURRnURS:H=(a U[ R :HnURURnURURnU(a UR5nUR5nXe-URU-nU(aiURUS9nURU5URURl URU5URURl U$rg) rHrrrrr[rrGrkrNrrrls r rT#MultiheadAttention.get_delta_weights W%,,33 G$++11 {{e+F0F ;;w'..;;w'.. ~~'H~~'H!, W0EE ),,5,9M08{{5/ADKK ' ' ,/7{{5/ADKK ' ' ,r$c>SU;a#[SURRS35e[TU]"U/UQ70UD6 g)Nrrzz( does not support mixed adapter batches.)rrr>rDr#rdr4rr6rs r r#&MultiheadAttention._check_forward_argssB f $eDNN$;$;#<rrGr\r) rdqueryrr6previous_dtyper5rcls_namerrs r r7MultiheadAttention.forwards   888  {{ __U*>S*>Qt{{BRq*>OS   9@@@ ),,~.PVWXPYPeq ^0Lu lrrsktu T s F#F,%FF0ctUR5nURnU?URS[R"UR UR S95 URR5nURnU?URS[R"UR UR S95 g)Nrr2r) rarr3rSrrrrr)rdrRrs r _restore_weights#MultiheadAttention._restore_weights/s((* **  %%%&6 V[[`f`t`t8uv ((779 ""  %%h V[[X^XlXl0mnr$cD>UR5 [TU]"U0UD6$r)rFrD state_dictrs r rIMultiheadAttention.state_dictDs$ w!42622r$cD>UR5 [TU]"U0UD6$r)rFrD named_modulesrs r rL MultiheadAttention.named_modulesHs$ w$d5f55r$c*>[TU]5nSU-$ryr{r}s r r|MultiheadAttention.__repr__Mrr$)rE)rrrsTFF)rr:rLrrMrrOrkrrrYr9rXr9r;r<)r;r)r;z Optional[int])r;r9)r;rk)r;z nn.Parameter)r;r8)r;zOptional[nn.Parameter])r;z,tuple[Optional[torch.Tensor], Optional[int]]rrr)r\r9rXr9rrr;znn.MultiheadAttentionr)rAr=rrr6rr;r=r)$r>r?r@rArBrepropertyrrrrr r2rrrrrrr r#r&rr\r1r9rTr#r7rrFrIrLr|rDrrs@r rrs &!.2 ,d,d  ,d  ,d  ,d,,d,d,d ,d,d\//****99//--11..4422??,,,,B33? E[TU]5 Xlgr)rDrerZ)rdrZrs r re_LoraParameterProxy.__init__Ws (r$c[RRR5 XR-sSSS5 $!,(df  g=fr)rSutils parametrizecachedrZ)rdWs r r7_LoraParameterProxy.forward[s0 XX ! ! ( ( *(((+ * *s A A)rZ) r>r?r@rArBrer7rDrrs@r rRrRRs)))r$rRc[U[R5(aURX5 gUR X5 gr)rrSrr3register_buffer)rr!rs r _register_parameter_or_bufferr]bs/!R\\""!!$*t'r$c^\rSrSrSrSSU4SjjjrSSSjjrSSSjjrSrSr \ SSj5r S r SSS jjr SS jrU4S jrSS jrSSjrSU4SjjrSrU=r$) ParamWrapperiiaA LoRA wrapper for `nn.Parameter`. This layer is dispatched if users target a parameter directly with `lora_config.target_parameters` Note: - When accessing the wrapped nn.Parameter directly, e.g. via `module.weight`, the LoRA weights are *not* applied. - It is currently not implemented to target multiple parameters on the same module. To achieve this, it is currently required to create a separate LoRA adapter (with another adapter name) and activate both at the same time. c >[TU]5 [R"X40U D6 X0lUR 5nUR S:Xa UR uUlUlUl O0SUR SUR SsUlUlUl UR S;a0[SURRSUR S35eU(a#[SURRS35eU(a#[SURRS 35eU (a#[SURRS 35eU (a#[SURRS 35eU(a#[SURRS 35eXpl X lURUUUUU U U U S 9 g)Nrrr)rrrzz was initialized with z9 dimensional Parameter, but only 2d and 3d are supported.& does not work with lora_dropout != 0.z# does not work with fan_in_fan_out.z# does not work with lora_bias=True.z" does not work with use_dora=True.z1 does not work with is_target_conv_1d_layer=True.)rMrOrrYrXrZ)rDrer9parameter_name get_paramndimshape num_expertsrbrcrrr>rrEr)rdrRrrbrLrMrOrrFrrYrXrZr6paramrs r reParamWrapper.__init__ts 46v6,  ::?DIKK AD d.0ADEu{{ST~W\WbWbcdWe AD d.0A ::V #//00FuzzlS((  uT^^%<%<$==cde e uT^^%<%<$==`ab b uT^^%<%<$==`ab b uT^^%<%<$==_`a a "uT^^%<%<$==nop p,+   !%/!  r$c  & [5R5n U S US::a[SU35eURXxU S9n U b#[SURR S35eX R U'X0RU'US:a#[SURR S35e[R"5nURR[R"X055 [R"URX R-S S 9UR U'[R"X R-UR"U S 9UR$U'XR&U'U(a'U[(R*"U5- UR,U'OX2- UR,U'X`R.U'XpR0U'[3U[45(aUUR7S 5(a?[9UR;5R<5 UR?X5 SSS5 GO[3U[45(aUUR7S 5(a?[9UR;5R<5 URAX5 SSS5 GOD[3U[45(aRURC5S :Xa>[9UR;5R<5 UREU5 SSS5 OUS:Xa>[9UR;5R<5 URGU5 SSS5 OUS:Xa7[RHRKUR$UR<5 O\US:Xa>[9UR;5R<5 URMU5 SSS5 OU(aUROX5 URQU5 XRR;a URRURH"U40U D6 URUURVU S9 g!,(df  Ni=f!,(df  Nz=f!,(df  N=f!,(df  N=f!,(df  N=f)Nrdrrk)rXrprqrzz, does not work with LoRA variants like DoRA.rsraFrvrwrxryrzr{r|r}),rrrrhrr>rLrMrSrrOrrTrrbrfrGrcrHrZrrrNrYrXrr:rrrarrrrrrr!rrrrr`rr)rdrrLrMrOrrYrXrprZrqr~r6r`rs r rParamWrapper.update_layers" 6N 6^_`^abc c00HY1   #uT^^%<%<$==ijk k |(2 % # uT^^%<%<$==cde e!#     /Q!RS%'IId.>.>DTDT@T[`$a L!$&IIa2B2B.BDDUDU\e$f L!'0|$ )3diil)BDLL &)3DLL &(2 %&. l# ' - -2C2N2Nw2W2W"4#6#6#8#?#?@ @A@ )3 / /4E4P4PQX4Y4Y"4#6#6#8#?#?@ @A@ )3 / /4E4K4K4MQX4X"4#6#6#8#?#?@ -A@ ' )"4#6#6#8#?#?@ -A@ % ' GGNN4;;|4;; < , ."4#6#6#8#?#?@$$\2A@   & &| G 22<@ ,, ,   l + 0 0 @ @ --nM1A@A@A@A@ A@s<P>Q+Q /Q10R> Q  Q Q.1 Q? Rc^UR5Rn[R"S5mUR5nURUR-Hn[ XS5n[ U[R[R[45(dMEX;aML[U4SjUR555(aMvURR(dURR(aXQR!X#RS9XQ'MXQR!U5XQ'M g)zm Move the adapter of the given name to the device of the base layer. Needs special handling for nn.Parameter metaNc3@># UHoRT:Hv M g7fr)r).0rurls r EParamWrapper._move_adapter_to_device_of_base_layer..sH-G88t#-Grj)rcrr[rKrPrrrSrTrUr r parametersris_floating_point is_complexr)rdrrrgadapter_layer_name adapter_layerrls @r r2ParamWrapper._move_adapter_to_device_of_base_layers!((||F# "&":":T=S=S"S #DdCMmbmmR=M=Mz-Z[[0H]-E-E-GHHH{{,, 0F0F.;.I.L.LV[f[f.L.g +.;.I.L.LV.T +#Tr$cN[UR5UR5nU$r)rrarb)rdrgs r rcParamWrapper.get_params"++-t/B/BC r$cBURS:Xa[R"X/UQ70UD6nOURURnUR URnUR URSURS5nUR URSSUR5n[R"SXe5URU-nUR5nUR5nURURUR5nU$)Nrrrzo r e, e r i -> e i o)rfrrTrGrrHreshaperer[einsumrNrarcrrr) rdrrr6rZrnrorRrgs r rTParamWrapper.get_delta_weights   q !224WWPVWL{{<077H{{<077H''(8(8"hnnR>PQH''q(92t?O?OPH <<(?TW[WcWcdpWqqL((*  #u||U[[A r$c#p^# U(a[U4SjU55(dSv gSnUH?nUTR;aMUcTRU5nM+UTRU5-nMA TR5nTR 5R n[ RRRUTR[U55 URTRRRU5 Sv TR5 g!TR5 f=f7f)Nc3@># UHoTR;v M g7fr)rG)rnrrds r ro.ParamWrapper._activate_lora...s)`P_WT[[*@P_rq)rrGrTrarcrrSrVrWregister_parametrizationrbrRparametrizationsoriginalrequires_grad__remove_parametrizations)rdrrZr'rRrequires_grad_befores` r _activate_loraParamWrapper._activate_lora,sc)`P_)`&`&`   -NT[[0##44^D +d.C.CN.SS .((* #~~/== 55 ++-@-N  ##D$7$78AAPPQef ,   ) ) +D ) ) +sDD6 D!D6!D33D6cUR5nURnX!R;a [S5eURUn[ U5S:Xa)[ R RRXSS9 g[[[ U555nUH nX5n[U[5(dMX5 g [R"SUS35 g)NzbSomething went wrong, please report this issue on PEFT: https://github.com/huggingface/peft/issuesrF)leave_parametrizedz+Could not find any LoRA parametrization on z], please open an issue on https://github.com/huggingface/peft/issues and report this warning.)rarbrrrrSrVrWremove_parametrizationsreversedrangerrRrr)rdrRrb param_listreversed_indicesr1rs r r%ParamWrapper._remove_parametrizationsHs((* ,, !>K >>+6::<<(OP^O__rs"-J$(#8#8#HLJJ,.J$$++N;/,r$cUR(d[R"S5 g[UR5S:aURR 5nXR R5;ae[UR5UR5nURnURU5nU=RURU5-sl [UR5S:aMgg)Nrar)rrrrrWr rGrrrarbrrTrr)rdr'rgrYrZs r r1ParamWrapper.unmerges{{ MM< = $&&'!+!11557N!1!1!33 3 3 5t7J7JK"[[ #44^D  looj99 $&&'!+r$c>URSS5(a#[SURRS35e[TU]"U/UQ70UD6 g)rrNrzz, does not support mixed adapter batches yet.)rrrr>rDr#r>s r r# ParamWrapper._check_forward_argssK ::ot , ,uT^^%<%<$==ijk k #A777r$cURnU(aXURX#S9 [U[5(a2URX#S9 URn[U[5(aM2U$UR 5nU$)Nr7)rRr\rr_ra)rdr\rXrrRs r r9/ParamWrapper.unload_and_optionally_merge_modulesr__  JJ*J JZ66  J T'22 Z66 ,,.Jr$c UR"U/UQ70UD6 URSS5nUR(a9UR(aUR 5 UR "U/UQ70UD6nU$Ub#[ SURRS35eUR(aUR "U/UQ70UD6nU$URUR5 UR "U/UQ70UD6nSSS5 U$!,(df  W$=f)Nrrzz' does not support mixed batch inference) r#r rtrr1rRrrr>rr)rdr4rr6rr5s r r7ParamWrapper.forwards   4T4V4 ?D9  {{ __Q888F  &uT^^%<%<$==def f [[__Q888F $$T%9%9:> D c>[TU]5nURS5S-nUSUSURSXS3nSU-$)N(rz parameter_name='z',rz)rDr|findrb)rdr~idxrs r r|ParamWrapper.__repr__sVg hhsmaTc /0C0C/DBs4ykR}r$)rErrbrfrcrb) rrrsFFTFFF)rr:rbr:rLrrMrrOrkrr9rFr9rrrYr9rXr9rZr9r;r<)FFFr<F) rXr9rpr9rZr9rqrr~r9r)rr:rzOptional[torch.device]r;r<)rr?rrr)r\r9rXr9rrrr)r>r?r@rArBrerrrcrTrrrr\r1r#r9r7r|rDrrs@r r_r_isK!$(-.2 6 6  6  6  6 6 6 "&6 ,6 6 6 6  6 6 @ !#$RNRNRNRNRNRNhU,$,,6@<@ :8 "r$r_c Sn[U[5(aUR5nOUnUb[X4SU0UD6nU$[U[R R 5(aKUR5nURSS5 URUR5 [ X40UD6nU$[U[R R5(a)URUR5 [X40UD6nU$[U[R R5(a)URUR5 [X40UD6nU$[U[ R5(a)URUR5 [X40UD6nU$[U[R R5(a)URUR5 [X40UD6nU$[U[R R5(aUUS(a"[ R""S5 S=US'UlURUR5 [X40UD6nU$[U[&5(aUUS(d"[ R""S5 S=US'UlURUR5 [X4SS0UD6nU$)Nrbrzjfan_in_fan_out is set to True but the target module is `torch.nn.Linear`. Setting fan_in_fan_out to False.Fzafan_in_fan_out is set to False but the target module is `Conv1D`. Setting fan_in_fan_out to True.TrF)rr rar_r[rSrrr r loftq_configrrrrrrrrr )targetr lora_configrbr6 new_moduletarget_base_layerembedding_kwargss r dispatch_defaultrsJ&.))"113"!!&`~`Y_` H G %uxx'9'9 : :!;;=-t4 8 89vH7GH > = %uxx 7 7 k../F;F; 8 7 %uxx 7 7 k../F;F; 2 1 %ryy 1 1 k../F;F; , + %uxx'B'B C C k../'GG & % %uxx 7 7 " # MM3 EJ IF# ${'A k../F;F;   %v . .&' MMs EI HF# ${'A k../FY$YRXY r$r) rztorch.nn.Modulerr:rrrbz Optional[str]r;zOptional[torch.nn.Module])4 __future__rrr contextlibrtypingrrrr[torch.nnrStorch.nn.functional functionalrrtransformers.pytorch_utilsr peft.tuners._buffer_dictr peft.tuners.tuners_utilsr r r peft.utils.integrationsrrrrpeft.utils.otherrpeft.utils.warningrconfigrrr'rr9ModulerrrrrrrrRr]r_rr,r$r rsH# %'' -/bb '*+&&/"/"dllraRYY aHt 9tn rbiirj #W#"#W#"#W#"uIup )")) ) (U299iUx %) 3 333" 3  3r$