oiwPSSKrSSKJrJrJrJrJrJrJrJ r "SS\RR5r SSK JrJr SSjrSSKJrJr SS jrSS KJrJr S r"S S \RR5rSSjr"SS\RR5rSr\R8R:r\R>S\R@S\R@4Sj5r!g)N) _maybe_fake_quantize_activationsfast_dequantize QUANT_STATEget_lora_parametersget_lora_parameters_bias matmul_loratorch_amp_custom_fwdtorch_amp_custom_bwdc\rSrSrSr\\S S\R4Sjj55r \\ S\R4Sj55r Sr g) LoRA_MLPa ### LoRA weights G = G + Ag @ Bg U = U + Au @ Bu W = W + Aw @ Bw ### SwiGLU(X) e = X @ G f = e * sigmoid(e) g = X @ U h = f * g i = h @ W ### Backpropagation chain rule See our blog post for more details df = sigmoid(e) * (1 - f) + f dC/dW = h.T @ dY dC/dU = X.T @ (D @ W.T * f) dC/dG = X.T @ (D @ W.T * df * g) ### Down projection LoRA weights dC/dAw = dC/dW @ B.T dC/dBw = A.T @ dC/dW dC/dAw = h.T @ dY @ B.T dC/dBw = A.T @ h.T @ dY ### Up projection LoRA weights dC/dAu = X.T @ (D @ W.T * f) @ B.T dC/dBu = A.T @ X.T @ (D @ W.T * f) ### Gate projection LoRA weights dC/dAg = X.T @ (D @ W.T * df * g) @ B.T dC/dBg = A.T @ X.T @ (D @ W.T * df * g) Don't forget to see our blog post for more details! Xc URn[XX4XV5n[XXX5nU"UU5n[UXXU5nUUUUUU U U UU4 UlURXEXXUUU5 UUlU$N)dtyper custom_saved_tensorssave_for_backwardinplace)ctxrgateW gateW_quantgateAgateBgateSupW upW_quantupAupBupSdownW downW_quantdownAdownBdownS_forward_function_backward_functionrreghis S/home/james-whalen/.local/lib/python3.13/site-packages/unsloth/kernels/fast_lora.pyforwardLoRA_MLP.forwardCs0 +e C   9 a # 5uU C           $   eCeAq!L dYc URu nnnnnnnn n n URu ppnnnnnURunnnURSURS5nURSURS5nURSURS5nURSURS5nURnU R U5U R U5UR U5UR U5UR U5UR U54uppnnU R 5U R 5UR 5UR 5UR 5UR 54uppnn[XR 5U UUU 5nU "UUU5unnnUUUnnn[R"U5n[R"U5n[R"U 5n[R"U 5n [R"U5n![R"U5n"URUR 5UUR 5-U SS9 URUR 5UR 5-XSS9 U!RUR 5UUR 5-USS9 U"RUR 5UR 5-UUSS9 URUR 5UU R 5-USS9 U RU R 5UR 5-UUSS9 [UR 5U5n[R"UUR 5UR(aUOSS9n#AU#RUUR 5-UR 5US9 [UR 5U5nU#RUUR 55 AU#RUU R 5-U R 5US9 U#RUUU5SSUR 5U R 5SSSU!R 5U"R 5SSSUR 5UR 5SSSS4$Nralphabeta)outr5)r saved_tensorsshapeviewrtotr torch empty_likeaddmm_rmatmulr)$rr0rrrrrr r!r"r%r'rrrrr#r$rr(r)batchseq_lenhdrDWr*dfded_downAd_downBd_gateAd_gateBd_upAd_upBdXs$ r,backwardLoRA_MLP.backwardrs  $ $          8;8I8I5cuaAWWw WWR" & FF2qwwr{ # FF2qwwr{ # FF2qwwr{ # HHUO HHUO FF5M FF5M HHUO HHUO 0 ,cu GGI GGI EEG EEG GGI GGI 0 ,cuWWY UE5 I%b!Q/Aq1r2""5)""5)""5)""5)  %  % qssub5779neAFuwwy1335("AF  QSSUBL#a @ SUUWqssu_b#a @ qssub5779neAFuwwy1335("eAFceegy1 \\"ceeg#++Q4 H  "suuw, 5 ;7 "eggi  "uwwy.%'')U ; GGE7B '   IIK IIK    GGI GGI    IIK IIK    '  r/NT __name__ __module__ __qualname____firstlineno____doc__ staticmethodr r>Tensorr-r rO__static_attributes__rQr/r,r r s`$L*)+ <<++Zq %,,q q r/r )swiglu_fg_kernelswiglu_DWf_DW_dfg_kernelc&[XR5n[UR5up4pVn[UR5uppn [UR5upnnn[ R UUUUUUUU U U U U UUUU[[U5nU$r) r gate_projrup_proj down_projr applyr\r]selfrrrrrrrrrrrr r!r"r#r$r%r7s r,apply_lora_mlp_swiglures(NN;A.A$...Q+Ee$7 $E!CCc.A$...Q+Eue ..              ' C* Jr/)geglu_exact_forward_kernelgeglu_exact_backward_kernelc&[XR5n[UR5up4pVn[UR5uppn [UR5upnnn[ R UUUUUUUU U U U U UUUU[[U5nU$r) rr_rr`rar rbrfrgrcs r,apply_lora_mlp_geglu_exactri s(NN;A.A$...Q+Ee$7 $E!CCc.A$...Q+Eue ..             "#' C* Jr/)geglu_approx_forward_kernelgeglu_approx_backward_kernelc"[XR5n[UR5up#pEn[UR5upxpn [UR5uppn[ R UUUUUUUUU U U U U UUU[[5nU$r) rr_rr`rar rbrjrk)rdrrrrrrrrrrr r!r"r#r$r%r7s r,apply_lora_mlp_geglu_approxrm+s(NN;A.A$...Q+Ee$7 $E!CCc.A$...Q+Ee ..             #$% C( Jr/cn\rSrSrSr\\SS\R4Sjj55r \\ S55r Sr g) LoRA_QKViGab ### LoRA weights Wq = Wq + Aq @ Bq Wk = Wk + Ak @ Bk Wv = Wv + Av @ Bv Q = X @ Wq = X @ Wq + X @ Aq @ Bq K = X @ Wk = X @ Wk + X @ Ak @ Bk V = X @ Wv = X @ Wv + X @ Av @ Bv ### Backpropagation chain rule See our blogpost for more details. dC/dWq = X.T @ D(Wq) dC/dWk = X.T @ D(Wk) dC/dWv = X.T @ D(Wv) We then sum them all find dC/dX ### Q projection LoRA weights dC/dAq = X.T @ D(Wq) @ B.T dC/dBq = A.T @ X.T @ D(Wq) ### K projection LoRA weights dC/dAk = X.T @ D(Wk) @ B.T dC/dBk = A.T @ X.T @ D(Wk) ### V projection LoRA weights dC/dAv = X.T @ D(Wv) @ B.T dC/dBv = A.T @ X.T @ D(Wv) rc  URnURnUnUR5S:XaURSURS5n[ UX#XEU5n[ UXxXU 5n[ UXXU5n[ U5S:XaKURUSUSS5nURUSUSS5nURUSUSS5nUUUUUU U U U4 UlURUUUU U UU5 UUlUUU4$)Nr3rr) rr:dimr;r lenrrr)rrQWQW_quantQAQBQSKWKW_quantKAKBKSVWVW_quantVAVBVSrr orig_shape X_for_matmulQKVs r,r-LoRA_QKV.forwardfs., WW  557a<66"aggbk2L  bBB ?  bBB ?  bBB ? z?a z!}jmR8Az!}jmR8Az!}jmR8A          $           !Qwr/c URu pEpgppn URun nnnnnnU RunnnURSURS5nUR SURS5nURSURS5nU RSU RS5n U R nUR U5UR U5UR U5UR U5UR U5UR U54upnnnnUR5UR5UR5UR5UR5UR54upnnnn[R"U5n[R"U5n[R"U5n[R"U5n[R"U5n[R"U5nURU R5XR5-USS9 URUR5U R5-XSS9 URU R5UUR5-U SS9 URUR5U R5-X)SS9 URU R5UUR5-U SS9 URUR5U R5-Xr?r@rrAr)rdQdKdVrtrurxryrzr}r~rrrrvrwr{r|rrrBrCrDrd_QAd_QBd_KAd_KBd_VAd_VBrNs r,rOLoRA_QKV.backwards@C?W?W<bhB"         WWw WWR" & ZZBHHRL ) WWR" & FF2qwwr{ # EE%L EE%L EE%L EE%L EE%L EE%L " BB"$!OBB###### ACCE2;1 = BDDFQSSUNB1 = ACCE2;1 = BDDFQSSUNB1 = ACCE2;1 = BDDFQSSUNB1 =RTTVX . \\"ddfA$ G  "ttv+rttvr 2RTTVX . "ddf  "rttv+rttvr 2RTTVX . "ddf  "rttv+rttvr 2 GGE7B '   FFH FFH    FFH FFH    FFH FFH  #  r/rQNrRrSrQr/r,roroGsS<&%< <<<<|l l r/roc [XR5n[UR5up4pVn[UR5uppn [UR5upnnn[ R UUUUUUUU U U U U UUUUU5unnnUUU4$r)rq_projrk_projv_projrorb)rdrrrtrurvrwrxryrzr{r|r}r~rrrrrrrs r,apply_lora_qkvrs(KK8A24;;?B""24;;?B""24;;?B"b"nn             #GAq!& a7Nr/c\rSrSrSr\\S\R4Sj55r \\ S\R4Sj55r Sr g) LoRA_Wi2a ### LoRA weights Wq = Wq + Aq @ Bq Wk = Wk + Ak @ Bk Wv = Wv + Av @ Bv Q = X @ Wq = X @ Wq + X @ Aq @ Bq K = X @ Wk = X @ Wk + X @ Ak @ Bk V = X @ Wv = X @ Wv + X @ Av @ Bv ### Backpropagation chain rule dC/dWq = X.T @ D(Wq) dC/dWk = X.T @ D(Wk) dC/dWv = X.T @ D(Wv) ### Q projection LoRA weights dC/dAq = X.T @ D(Wq) @ B.T dC/dBq = A.T @ X.T @ D(Wq) ### K projection LoRA weights dC/dAk = X.T @ D(Wk) @ B.T dC/dBk = A.T @ X.T @ D(Wk) ### V projection LoRA weights dC/dAv = X.T @ D(Wv) @ B.T dC/dBv = A.T @ X.T @ D(Wv) rcpURn[XX4XV5nUUU4UlURXEU5 U$r)rr rr) rrWW_quantABSrXWs r,r-LoRA_W.forwardNsE w1 0  $  aA& r/r0cURup#nURupVnURupn URSURS5nURSURS5nURn UR U 5UR U 5peUR 5UR 5pe[R"U5n [R"U5n U RUR 5XR 5-USS9 U RUR 5UR 5-XSS9 [UR 5U5nXR 5-nAURXR 5-UR 5US9 URXU 5SSU R 5U R 5S4$)Nr3rr4r8) rr9r:rrr<r=r>r?r@rr;)rr0rrrrrrrBrCrDrd_Ad_BrNs r,rOLoRA_W.backward[si00 A##aWW ZZBHHRL ) IIb!''"+ &ttE{ADDK1ssuacce1q!q! 1335"ssu*a : 13351335=" : ACCE7 + ##%Z  "ssu*acceQ /wwur*D$$NNr/rQNrSrQr/r,rr2sX6   !O%,,!O!Or/rc[XR5n[UR5up#pEn[R XX4XV5nU$r)ro_projrrrb)rdrOWOW_quantOAOBOSOs r, apply_lora_ors<(KK8A24;;?B"" QH"1A Hr/xreturnc[S5e)NzAUnsloth: Currently not supported yet - reshaping done incorrectly) NotImplementedError_check_forward_argspopdisable_adaptersmergedunmerge base_layer_mixed_batch_forwardrsactive_adapterslora_Akeys lora_dropout isinstanceIDENTITY_DROPOUTuse_doraweightlora_Bscalingrrbrcloner>is_autocast_enabledrr<nnIdentitytraininglora_magnitude_vectorget_base_layer)rdrargskwargs adapter_namesresultactive_adapterdropoutrrrrrequires_conversionexpected_dtype base_results r,fast_lora_forwardrs K r/rR)"r>utilsrrrrrr r r autogradFunctionr swiglur\r]regeglurfrgrirjrkrmrorrrrrr_disable_dynamorZrrQr/r,rs    I u~~&&I X?:K:M8M u~~&&M `6LOU^^ $ $LO^ 88$$Nu||NNNr/