oi,SrSSKJr SSKrSSKJr SSKJr SSKrSSK J r SSK J r SSKJ r SSKJr S S KJr S S KJr "S S \5rSSSjjrg)zC This module contains the implementation of the LoRA-FA optimizer. ) annotationsN)Iterable)Callable)is_bf16_available)autocast) Optimizer) PeftModel) infer_devicec^\rSrSrSrSSU4Sjjjr\R"5SS Sjj5rSr U=r $) LoraFAOptimizer#a Implements the LoRA-FA optimizer designed specifically for training Low-Rank Adaptation (LoRA) parameters efficiently. Note that LoraFAOptimizer is based on adamw-hf in transformers, with only LoRA part modified. Without LoRA it will fall back to adamw-hf. Args: params (Iterable[nn.parameter.Parameter]): Parameters to optimize. lr (float, optional): Learning rate (default: 1e-3). betas (Tuple[float, float], optional): Coefficients for computing running averages of gradient and squared gradient (default: (0.9, 0.999)). eps (float, optional): Term added to denominator to improve numerical stability (default: 1e-6). weight_decay (float, optional): Weight decay (L2 penalty) (default: 0.0). correct_bias (bool, optional): Whether to apply bias correction as in original Adam (default: True). Args in sub-function step: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Reference: - LoRA-FA: https://huggingface.co/papers/2308.03303 c>US:a[SUS35eSUSs=::aS:dO [SUSS35eSUSs=::aS:dO [SUSS35eSU::d[S US35eUUUUUS .n[TU] X5 g) NzInvalid learning rate: z - should be >= 0.0r?zInvalid beta parameter: z - should be in [0.0, 1.0)zInvalid epsilon value: )lrbetaseps weight_decay correct_bias) ValueErrorsuper__init__) selfparamsrrrrrdefaults __class__s P/home/james-whalen/.local/lib/python3.13/site-packages/peft/optimizers/lorafa.pyrLoraFAOptimizer.__init__9s 86rd:MNO OeAh$$7azA[\] ]eAh$$7azA[\] ]cz6se;NOP P((   *c l SnUbU"5nURGHnUSn/n/n[USUS5GH`upxSU;aURcMURn SU;a[URU5 URU5 [ U5S:XaUSUR S5S-n O[ U5S:XaMOUn UR W n [ U 5S:Xa[ U5S:Xa>SU S '[R"US5U S '[R"US5U S 'O7SU S '[R"U5U S '[R"U5U S '[ U5S:XGa'USn USn U RnSnXR-n[RRUU[R"U RS5RU R5--5n[!5n[#5(a0[%U[R&S9 SUS-- UU--nSSS5 OSUS-- UU--nWR(U RR(:wa%URU RR(5nU S U S nnUSunnU S ==S- ss'UR+U5R-USU- S9 UR+U5R/UUSU- S9 UR15R-US5nUSnUS(a2SUU S -- nSUU S -- nU[2R0"U5-U- nU R5UUU*S9 USS:aU R-XS*US-S9 /n/nGMcU S U S nnUSunnU S ==S- ss'UR+U5R-U SU- S9 UR+U5R/XSU- S9 UR15R-US5nUSnUS(a2SUU S -- nSUU S -- nU[2R0"U5-U- nUR5UUU*S9 USS:dGMGUR-XsS*US-S9 GMc GM U$!,(df  GNc=f)z Performs a single optimization step. Arguments: closure (`Callable`, *optional*): A closure that reevaluates the model and returns the loss. Nscaling_factorrnameslorar rrstep exp_avg_B exp_avg_sq_Bexp_avg exp_avg_sqg:0yE>) device_typedtyperr)alpha)valuerrrrr) param_groupszipgradappendlenfindstatetorch zeros_likeTlinalgpinveyeshapetodevicer rrbfloat16r,mul_add_addcmul_sqrtmathaddcdiv_)rclosurelossgroupr# param_list name_listpnr1namer5AB grad_B_orindeltaAA_TAA_T_invr+grad_Br'r(beta1beta2denom_B step_sizebias_correction1bias_correction2r)r*denoms rr&LoraFAOptimizer.stepSs  9D&&E"#34NJIE(OU7^<?qvv~vvQ;%%a($$Q':!+ !1166&>2V;ZA- .D 4(u:?:!+()f -2-=-=jm-Lk*050@0@A0On-()f +0+;+;A+>i(.3.>.>q.Al+z?a'"1 A"1 A"#&&K!Ess7D$||00 !''RS*@U@X@XYZYaYa@b8b1bcH"..K(**%+U^^T&'.!*;&; h@V%WFUT#$na&7"7K(LE5&MQ&MNN5)..vcEk.K %%e,55ffCRWK5X*//166uU|DG %d I^,+.%-1G+G(+.%-1G+G($- :J0K$KN^$^ JJy')JD^,s2q+n8M)MO!#J "I+0 *:E,LE5&MQ&MLL',,T#+,GOOE*33DcEk3R&OO-225<@E %d I^,+.%-1G+G(+.%-1G+G($- :J0K$KN^$^ JJwiZJ@^,s2q+n8M)MO_= 'j uUTs *R$$ R3 )gMbP?g?g+?gư>rT) rz Iterable[nn.parameter.Parameter]rfloatrztuple[float, float]rr_rr_rbool)N)rFr) __name__ __module__ __qualname____firstlineno____doc__rr6no_gradr&__static_attributes__ __classcell__)rs@rr r #sy0%1!!+0+ +# +  +  +++4 ]]_@@r!r cFUR5HupgSU;dM URS5 M U(aU[R"U5- OX!- nUR 5UUR5VV s/sHupiUPM sn nUSUS./n [ U 5$s sn nf)a Helper function to instantiate a lorafa optimizer specifically configured for a given model using the LoRA method. This function will: - Disable gradient updates for the "lora_A" parameters (these are typically frozen during LoRA training). - Compute the scaling factor based on provided `lora_alpha` and rank `r` for proper gradient projection. - Create and configure parameter groups for the optimizer including specified learning rate, weight decay, and additional optimizer options. For hyper-params, LoRA-FA uses the same hyper-params as AdamW, except for the LoRA hyper-params (r, lora_alpha, use_rslora). One can always use the same hyper-params such as lr and weight_decay, as AdamW in LoRA tuning. Args: model (PeftModel): The model containing LoRA-adapted parameters. r (int): Rank of the LoRA decomposition. lora_alpha (int): Scaling factor for LoRA parameterization. lr (float): Learning rate for optimizer updates. weight_decay (float): Weight decay for AdamW. use_rslora (bool): whether to use rslora. In rslora, the lora scaling factor becomes to lora_alpha / math.sqrt(r) instead of lora_alpha / r. Returns: Optimizer: Configured lorafa optimizer instance ready for training. lora_AFr^)rrr$r#rr)named_parametersrequires_grad_rDrC parametersr ) modelr lora_alpharr use_rslorarMparam lora_scaling_r/s rcreate_lorafa_optimizerrus8--/  t    '01;: ! , L&&(*/*@*@*BC*Bwtd*BC*!(   L < (( Ds:B )rF)rnr rointrprvrr_rr_rqr`returnr)re __future__rrDcollections.abcrtypingrr6torch.nnnnaccelerate.utils.importsrr torch.optimr peft_modelr utils.otherr r rur]r!rrs# $ 6!"&qiqjin*) *)*)*-*)38*)HM*)ae*)*)r!