oiJSSKJr SSKrSSKJrJrJr SSKrSSKJ r SSK J s J r SSK JrJr SSKJr "SS\ R$5r"S S \ R$5r"S S \5r"S S\ R$\5r"SS\ R$\5rSSjrg)) annotationsN)AnyOptionalUnion)BaseTunerLayercheck_adapters_to_merge) OFTConfigc6^\rSrSrSrSU4SjjrSrSrU=r$)MultiplicativeDropoutLayerz6 Implements the multiplicative dropout layer for OFT. c.>[TU]5 Xlg)z Initializes the multiplicative dropout layer. Parameters: p (float): The probability of dropping out a block. Defaults to 0.0. N)super__init__p)selfr __class__s O/home/james-whalen/.local/lib/python3.13/site-packages/peft/tuners/oft/layer.pyr#MultiplicativeDropoutLayer.__init__!s ctUR(Ga%URS:GaURSURS:wa [S5eURup#nUS:XaU$[ URU-5nX%- n[ R "[ R"XQRS9[ R"XaRS9/5nU[ R"U5RUSS5n[ R"X1RS9RUSS5nSU- U-Xx--nU$)a Applies multiplicative dropout to the input tensor. Parameters: x (Tensor): The input tensor of shape (D, H, H), where `D` represents the number of OFT blocks, and `H` is the size of the square blocks along the last two dimensions, the block size in OFT. rz4The last two dimensions of input should be the same!r device)trainingrshape ValueErrorinttorchcatonesrzerosrandpermvieweyerepeat) rxDH_num_to_replace num_zerosmask eye_matrixs rforward"MultiplicativeDropoutLayer.forward+s ===TVVaZwwr{aggbk) !WXXggGA!Av !_N*I99ejjI5;;W`iqiqKrstDq)*//1a8D1XX6==aAFJTQ!22Arr)) __name__ __module__ __qualname____firstlineno____doc__rr0__static_attributes__ __classcell__rs@rr r srr c^\rSrSrS U4SjjrSrSrSSSjjrSSjrSSjr Sr S r S r S r S rU=r$)OFTRotationModuleHc n>[T U]5 XlX lX0lX@l[ R"[R"X55Ul XPl X`l Xpl XlXlXl[R""X3S5upUR%SU SS9 UR%SU SS9 g)Nr rowsF) persistentcols)rrr n_elements block_size in_featuresnn Parameterr emptyweightcofteps block_share kernel_sizeuse_cayley_neumannnum_cayley_neumann_terms triu_indicesregister_buffer)rrCrDrErFrKrLrMrNrOrPr@rBrs rrOFTRotationModule.__init__Is $$&ll5;;q#=>  &&"4(@%'' B  VTe< VTe   ""6?FO%"PAttN##rcPURXR5nUS-[R"[R"UR S55- n[R "URS5URS54URURS9RS5RU5nX4- n[R"X4- SSS9nXb:*R5n[R"XsXBXV- --5nURXR5$)Nr rrU)r rbT)dimkeepdim)r[rEr sqrttensorrr#sizerrVrh expand_asnormboolwherer^) rrmrLoft_RIdiff norm_diffr.outs r_project_batch OFTRotationModule._project_batchs,,Q@Ag 5<< A#?@@ KKA 1 6u||SXS^S^ _ Yq\ Yu  yJJuyfdC  &&(kk$q$2B+C'CD//__EErcURSS:Xa[U5Vs/sHo1SPM nnO[U5Vs/sH o1US4PM nn[R"U6nU$s snfs snf)Nrr )r..)rrgr block_diag)rrrankiblocksAs r_block_diagonal!OFTRotationModule._block_diagonalsk ;;q>Q -24[9[Fm[F9F-24[9[AsFm[F9   f %:9s A&A+cURup#pE[UR[5(aURURpvOURupgS=pS=pUSU --U- U-S-n USU --U- U -S-n UR SXh5R SXy5nUR SSSSSS5R 5nURX,-U -S5nU$)z| Unfold with stride=1, padding=0 to preserve spatial dimensions. Only use kernel_size from base layer to define patch size. r rrbrcr)r isinstancerNrunfoldpermute contiguousr%)rr(rY in_channels in_heightin_width kernel_height kernel_widthstride_hstride_wpad_hpad_w out_height out_width x_unfoldeds r_unfoldOFTRotationModule._unfolds 89ww4  d&& , ,*.*:*:D...rc)rVrJrlrrKr no_gradcopy_rrLrvrErOrPlenrrMrFrCreshaper'einsumr) rr(required_dtyper orth_rotate folded_shaper batch_dimsrx_rotated_reshaped x_rotateds rr0OFTRotationModule.forwards  [[.. .T[[&&'AWW 99 !!$"5"5dkktxx"5"PQ!(( KK$*A*A4C`C` z?a  QAww 6:6F6Ft4??2DFFWWSb\ YYB BDB$//B   %,,T1a8K!&.@*!Z !&.@*!Z &.. = z?a  99I||N++7!s =>H## H2cURnUR(aR[R"5 UR XR S9nURR U5 SSS5 URXRURUR5nUR(d UROURUR-nURX#5$!,(df  N=f) Compute the delta weight for the given adapter. Args: adapter (str): The name of the adapter for which the delta weight should be computed. rN)rJrKr rrrLrrvrErOrPrMrCrFr)rrJrrs r get_weightOFTRotationModule.get_weights 99,,V,B !!&)!(( OOT%<%[>[ "--tvv43C3Ct3V##K66!s 5C(( C6) rMrErKrLrFrNrDrPrCrOrJ)FiUMu?F)rrTr)Tr) rm torch.TensorrErrOrrnrreturnr)gh㈵>)rrrrrr)r4r5r6r7rr[r^rvrrrrr0rr9r:r;s@rr=r=Hs !"=>kl&$&$+.&$DH&$dg&$ &$RF" 4>',R77rr=c\rSrSr%SrSrS\S'SrS\S'SSjr\ SS j5r S r SS jr SSS jjr SSSjjrSrSrSrg )OFTLayeri6z Implements the OFT layer. rztuple[str, ...]adapter_layer_names)rCoft_block_size oft_dropoutother_param_namesc Xl[R"05Ul0Ul0Ul0Ul[R"05UlSUl/UlSUl X l UR5n[U[R5(aURURpCGO[U[R 5(aUR"UR$pCGO['US5(a*['US5(aUR(UR*pCGO['US5(a*['US5(aUR,UR.pCGOb['US5(a3UR0R2S:XaURURpCGO['US 5(a2UR0R2S :XaURURpCOUR0R2S :XaURURpCO['US 5(a2UR0R2S :XaURURpCOf['US5(a)['US5(aURURpCOSup4[4R6"S[9U5S3[:5 X0lX@lg)z Initializes the OFT layer. Note, currently only support linear layer and convolutional layer, with further support for other layers to be added soon. Parameters: base_layer: the pretrained model layer FT infeatures outfeatures input_sizer codebooksQuantizedLinearw_bit WQLinear_GEMM EetqLinearW_q HQQLinearrF out_features)NNzUnsupported layer type 'z(' encountered, proceed at your own risk.N) base_layerrG ModuleDictrrrCr_disable_adaptersmerged_adapterscast_input_dtype_enabledkwargsget_base_layerrLinearrFrConv2dr out_channelshasattrrrrrrr4warningswarntype UserWarning)rrrrFrs rrOFTLayer.__init__@s3%]]2&   ==,!&!(,% ((* j")) , ,(2(>(> @W@W  BII . .(2(>(> @W@W Z . .7:}3U3U(2(=(=z?U?U Z . .7:}3U3U(2(=(=z?U?U Z - -*2F2F2O2OSd2d(2(>(> @W@W Z ) )j.B.B.K.K.^(2(>(> @W@W  ! ! * *l :(2(>(> @W@W Z ' 'J,@,@,I,I[,X(2(>(> @W@Wz=11gj.6Y6Y,6,B,BJD[D[\,6) MM*4 +;*<Unscaling operation for OFT not supported! Keeping scale to 1.rrs r unscale_layerOFTLayer.unscale_layers3"22NZZ__%66 MMZ [ 3rc US:a [US9n O[R"5n URR [R "X055 US:Xa|US:wavUR U-S:wdX0R :a;UnURUR U5n[R"SUSUS35 [UR U-5nOUS:wa|US:XavUR U-S:wdX R :a;UnURUR U5n[R"SUSUS35 [UR U-5nO [S 5eX3S - -S -n[U(dUOS UUUR UUUU U S 9 URU'URX5 X RU'X0R U'UR#U5 UR%UR&U S 9 g)zU Update the linear layer with trainable OFT weights. Override for other layer types. r3r2rInvalid `oft_block_size` (!)! Adjusted `oft_block_size` to (). Invalid `r` ()! Adjusted `r` to (ZSomething went wrong, please report this error: https://github.com/huggingface/peft/issuesr rb)rKrLrMrOrPinference_modeN)r rGIdentityrupdaterrFadjust_oft_parametersrrrrr=rreset_oft_parametersrCr%_move_adapter_to_device_of_base_layer set_adapterr)r adapter_namerCrmodule_dropoutrKrLrM init_weightsrOrPrroft_dropout_layerold_oft_block_sizeold_rrDs r update_layerOFTLayer.update_layers$  C  :^ L  "   |.O PQ 6n).0A5JZJZ9Z%3"!%!;!;D>? #H  r) rrrrFrrrrrrrC)r nn.ModulerNone)rzset[str])rfloatrr#rrr#Frr)r4r5r6r7r8r__annotations__rrpropertyrrrrrr r r9rrrrr6so ,65)OO9)v`d\& %QNQNf L!rrc^\rSrSrSrS S U4SjjjrS S SjjrSSjrSSjrSSjr SU4Sjjr S r U=r $)rizOFT implemented in Linear layerc >[TU]5 [R"X40UD6 XlX lUR UUUUUUUU U U S9 XlgN)rrrKrLrMrrOrP)rrrfan_in_fan_out_active_adapterris_target_conv_1d_layer)rrr rCrrrKrLrMrOrPr-rr/rrs rrLinear.__init__sh" $5f5,+   ))#%1%=  (?$rc4[X5nU(dgUGHnX0R;dMUR5nURRnU(aURR nUR U5n[R"USS5n[R"XvRUR55n[R"USS5n[R"U5R5(d[SUS35eUR5RU5URlOURR nUR U5n[R"USS5n[R"XvRUR55n[R"USS5nUR5RU5URlURR!U5 GM g)a Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If `True`, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If `None`, all active adapters will be merged. Defaults to `None`. Nrr z1NaNs detected in the merged weights. The adapter z seems to be broken)rrrrJrVdataget_delta_weightr rWmmrlisfiniteallrrrappendr safe_merge adapter_namesrr orig_dtype orig_weightsoft_mats rmerge Linear.merge6s0D  +N!9!99!002 '..44 #-#4#4#9#9L"33NCG#(??<A#FL#(88G__W]]5S#TL#(??<A#FL >>,7;;==(OP^O__rs.:-D-D-F-I-I*-UJ%%*#-#4#4#9#9L"33NCG#(??<A#FL#(88G__W]]5S#TL#(??<A#FL-9-D-D-F-I-I*-UJ%%*$$++N;7,rcUR(d[R"S5 gUR5nURR n[ UR5S:GakURR5nX0RR5;GaURU5nUR nU[R:waUR[R5nUR5RRn[R "USS5n[R""[R$R'U5RU5URU55n[R "USS5nURU5URl[ UR5S:aGMjggzG This method unmerges all merged adapter layers from the base weights. z Already unmerged. Nothing to do.Nrr )mergedrrrrJrVrrpoprrr3r float32rlr2rWr4rjinvrrr;rr=rpr<s runmergeLinear.unmergeesM{{ MM< = ((* &&,, $&&'!+!11557N!22//?!(!U]]2%jj7G#224;;@@ $|QB $xx (8(8(A(D(D^(TVbVeVeftVuv $|QB )5)D !!&$&&'!++rc<URUR5$rrrrr s rr3Linear.get_delta_weightzz,'2244rcHURnUR(a8UR(aUR5 UR"U/UQ70UD6nOUR(aUR"U/UQ70UD6nOUR H^nX`R R5;aM"UR UnURXRR5nU"U5nM` UR"URU5/UQ70UD6nURU5nU$r rVdisable_adaptersrBrGrrrr_cast_input_dtyperJrlrr(argsrrpresultrrs rr0Linear.forwards  {{ __Q888F [[__Q888F"&"6"6!):: >2**1ll.@.@A!H #7__QTT.%9KDKFKF>* rc*>[TU]5nSU-$Nzoft.r__repr__rreprs rrZLinear.__repr__g |r)r.r-r/) rr3FrFFrFTF)r strrCrrrrr$rKrrLr$rMrrOrrPrr-rrUnion[bool, str]r/rrr#FNr9rr:zOptional[list[str]]rr#r%rz!tuple[torch.Tensor, torch.Tensor])r(rrrrr`) r4r5r6r7r8rr>rGr3r0rZr9r:r;s@rrrs)  #!#(()$)-(-#?#?  #?  #?  #?#?#?#?!#?#&#?#?'#?"&#? !#?#?J-<^E4 5.rrc^\rSrSrSrS S U4SjjjrS SSjjrSSSjjrSSjrSSjr SSjr SU4S jjr S r U=r $)rizOFT implemented in Conv2d layerc >[TU]5 [RX5 XPlX lUR UUUUUUU U U U S9 gr,)rrrr-r.r)rrr rCrr-rrKrLrMrrOrPrrs rrConv2d.__init__s] $+,+   ))#%1%=  rc US:a [US9n O[R"5n URR [R "X055 UR 5nURSS:a [S5eURURS-URS-nUS:XaQUS:waKX-S:wdX?:a0UnURX5n[R"SUSUS35 [X-5nObUS:waQUS:XaKX-S:wdX/:a0UnURX5n[R"S US US35 [X-5nO [S 5eX3S- -S -n[U(dUOSUUUUUUURU U S 9 UR U'UR#X5 X R$U'X0R&U'UR)U5 UR+UR,U S9 g)z5 Update the conv2d layer with trainable OFT weights. r3r2rr z1Conv2d with dilation > 1 is not supported by OFT.rrrrrrrb)rKrLrMrNrOrPrN)r rGrrrrrdilationrrFrNr rrrr=rr rCrr r r)rr rCrrrKrLrMrrOrPrrrrconv_filter_dimrrrDs rrConv2d.update_layers& C  :^ L  "   |.O PQ((*   q !A %PQ Q**Z-C-CA-FFI_I_`aIbb 6n)/148X%3"!%!;!;O!\ 01C0DDefteuuwxO56A !V!+"a'1+>..B eW4H2NO !56Nl  $'9:a? #4 Aa   #"..1%= $  <  !!,= !|,:L) 22<@ --nMrc<[X5nU(dgUGHnX0RR5;dM#UR5nURR nU(GaLURR R5nURU5nURURURURS-URS-5n[R"USS5n[R"XvR!UR 55n[R"USS5nURURURURSURS5nUR#5R!U5URlGOJURU5nURR R5nURURURURS-URS-5n[R"USS5n[R"XvR!UR 55n[R"USS5nURURURURSURS5nUR#5R!U5URlUR$R'U5 GM g)a Merge the active adapter weights into the base weights Args: safe_merge (`bool`, *optional*): If True, the merge operation will be performed in a copy of the original weights and check for NaNs before merging the weights. This is useful if you want to check if the merge operation will produce NaNs. Defaults to `False`. adapter_names (`List[str]`, *optional*): The list of adapter names that should be merged. If None, all active adapters will be merged. Defaults to `None`. Nrr )rrrrrJrVr2cloner3r%rrFrNr rWr4rlrrr7r8s rr> Conv2d.merges0D  +N!22!002 '..44 $.#4#4#9#9#?#?#AL"33NCG#/#4#4))4+;+;j>T>TUV>W+WZdZpZpqrZs+s$L$)??<A#FL#(88G__W]]5S#TL#(??<A#FL#/#4#4))4+;+;Z=S=STU=VXbXnXnopXq$L.:-D-D-F-I-I*-UJ%%*"33NCG#-#4#4#9#9#?#?#AL#/#4#4))4+;+;j>T>TUV>W+WZdZpZpqrZs+s$L$)??<A#FL#(88G__W]]5S#TL#(??<A#FL#/#4#4))4+;+;Z=S=STU=VXbXnXnopXq$L.:-D-D-F-I-I*-UJ%%*$$++N;K,rcFUR(d[R"S5 gUR5nURR n[ UR5S:Ga9URR5nX0RR5;GaURU5nUR nU[R:waUR[R5nUR5RRR!5nUR#UR$UR&UR5R(S-UR5R(S-5n[R*"USS5n[R,"[R.R1U5RU5URU55n[R*"USS5nUR#UR$UR&UR5R(SUR5R(S5nURU5URl[ UR5S:aGM8ggrA)rBrrrrJrVrrrCrrr3r rDrlr2rnr%rrFrNrWr4rjrErFs rrGConv2d.unmergeSs{{ MM< = ((* &&,, $&&'!+!11557N!22//?!(!U]]2%jj7G#224;;@@FFH +00%%$$t':':'<'H'H'KKdNaNaNcNoNopqNrr   %|QB $xx (8(8(A(D(D^(TVbVeVeftVuv $|QB +00%%$$'')55a8'')55a8   *6)D !!&1$&&'!++rc<URUR5$rJrKrLs rr3Conv2d.get_delta_weightwrNrcHURnUR(a8UR(aUR5 UR"U/UQ70UD6nOUR(aUR"U/UQ70UD6nOUR H^nX`R R5;aM"UR UnURXRR5nU"U5nM` UR"URU5/UQ70UD6nURU5nU$rrPrSs rr0Conv2d.forwards  {{ __Q888F [[__Q888F"&"6"6!):: >2**1ll.@.@A!H #7__QTT.%9KDKFKF>* rc*>[TU]5nSU-$rXrYr[s rrZConv2d.__repr__r^r)r.r-) r_rFr3FrFTFr)rr"r r`rCrrrr-rrr$rKrrLr$rMrrrarOrrPrrr#r&r'rbrcr%rd)r(rrTrrrrrre)r4r5r6r7r8rrr>rGr3r0rZr9r:r;s@rrrs) $ #!)-#(()" " "   "  "  " " " " " '" !" #&"  " " ` %KNKNZ7rs#'' L))Xk7 k7\U!~U!pURYYUpuRYYup   r