6biVSrSSKJs Jr SSKJr SSKJr SSK J r \ "SSS/S9"S S \R55r "S S \R5r g) zAdam optimizer implementation.N)backend_config) optimizer_v2) keras_exportzkeras.optimizers.legacy.Adamzkeras.optimizers.Adam)v1c~^\rSrSrSrSrS U4SjjrSrU4SjrU4Sjr S Sjr S S jr U4S jr S r U=r$)Adama Optimizer that implements the Adam algorithm. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to [Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to `0.001`. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to `0.9`. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to `0.999`. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to `1e-7`. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to `"Adam"`. **kwargs: keyword arguments. Allowed arguments are `clipvalue`, `clipnorm`, `global_clipnorm`. If `clipvalue` (float) is set, the gradient of each weight is clipped to be no higher than this value. If `clipnorm` (float) is set, the gradient of each weight is individually clipped so that its norm is no higher than this value. If `global_clipnorm` (float) is set the gradient of all weights is clipped so that their global norm is no higher than this value. Usage: >>> opt = tf.keras.optimizers.legacy.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 Reference: - [Kingma et al., 2014](http://arxiv.org/abs/1412.6980) - [Reddi et al., 2018]( https://openreview.net/pdf?id=ryQu7f-RZ) for `amsgrad`. Notes: The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). Tc >>[TU]"U40UD6 URSURSU55 URSUR5 URSU5 URSU5 U=(d [ R "5UlXPlg)N learning_ratelrdecaybeta_1beta_2super__init__ _set_hyperget_initial_decayrepsilonamsgrad selfr rrrrnamekwargs __class__s ]/home/james-whalen/.local/lib/python3.13/site-packages/tf_keras/src/optimizers/legacy/adam.pyr Adam.__init__lsz (( D-)HI !4!45 &) &):."8"8":  cUHnURUS5 M UHnURUS5 M UR(aUHnURUS5 M ggNmvvhatadd_slotrrvar_listvars r _create_slotsAdam._create_slots~XC MM#s #C MM#s # << c6*  rc @>[T U]XU5 [R"URS-U5n[R "UR SU55n[R "UR SU55n[R"XT5n[R"Xd5nX1U4S[R"SU- 5SU- - -n X1U4R[U [R"URU5UUSU- UUSU- S95 gNrrlr_t)r rbeta_1_t beta_1_powerone_minus_beta_1_tbeta_2_t beta_2_powerone_minus_beta_2_t r_prepare_localtfcast iterationsidentity _get_hyperpowsqrtupdatedictconvert_to_tensorr r var_device var_dtype apply_state local_stepr1r4r2r5r rs rr8Adam._prepare_local zkBWWT__q0)< ;;txCD;;txCDvvh3 vvh3 i0 1& 9 GGA $ %\)9 :  +,33 ,,T\\9E!)#$x<!)#$x<  rc>URn[[U5S- S- 5n[U5SU-S-:XaUS[U5n[TU]U5 gNr/weightsintlenr set_weightsrrOparamsnum_varsrs rrRAdam.set_weightsWF a1,- w<1x  MM#s # MM#s #||:://JJ(((((8(8'":.":.$Y/ --0  ==f-D::::JJ(((([[(8(8'":.":.$Y/ --;  rcURURRpeU=(d 0RXV45=(d UR XV5nUR US5nXS-n [ RRRXUS-URS9n [ R"U /5 URXU 5n SSS5 UR US5n X-US-n [ RRRXUS-URS9n [ R"U /5 URXU 5n SSS5 UR(dl[ R"U 5n[ RRRUUSU -XS -- URS9n[ R "XU /6$UR US 5n[ R""UU 5n[ R"U/5 [ RRRUUURS9nSSS5 [ R"U5n[ RRRUUSU -UUS -- URS9n[ R "XU U/6$!,(df  GN=f!,(df  GN=f!,(df  N=f) Nr"r3r1)r^r#r6r4r rr$)r_r`rarrbrcr9compatrassignrgcontrol_dependencies_resource_scatter_addrr? assign_subgroupmaximum)rr]r)indicesrFrDrErir"m_scaled_g_valuesm_tr#v_scaled_g_valuesv_tv_sqrt var_updatev_hatv_hat_t v_hat_sqrts r_resource_apply_sparseAdam._resource_apply_sparses # CII,@,@I#)r..  # ?  ' ' >  MM#s # 0D#EEiill!! < ++9J9J"  $ $cU +,,Q9JKC, MM#s #![L9M,NNiill!! < ++9J9J"  $ $cU +,,Q9JKC,||WWS\F00T"S(F)5L,LM --1J 88js34 4MM#v.Ejj,G(('3)),,--70A0A.4)J00T" Y 779!-- 1J88jsG<= =G, +, +43s$=KK4K) K K&) K7c >[TU]5nURURS5URURS5URS5UR UR S.5 U$Nr rr)r r rrrrr get_configr@_serialize_hyperparameterrrrrconfigrs rrAdam.get_confign#% !%!?!?#",,88B88B<<<<   rrrgMbP?g?g+?gHz>FrN)__name__ __module__ __qualname____firstlineno____doc___HAS_AGGREGATE_GRADrr*r8rRrjr~r__static_attributes__ __classcell__rs@rrrsP IV $ + 0%&P/>brrc^\rSrSrSrSrSU4SjjrSrU4SjrU4Sjr \ R"SS9S 5r SS jr \ R"SS9S 5rSS jrU4S jrSrU=r$) NonFusedAdamiaK Optimizer that implements the Adam algorithm without fused kernels. Adam optimization is a stochastic gradient descent method that is based on adaptive estimation of first-order and second-order moments. According to the paper [Adam: A Method for Stochastic Optimization. Kingma et al., 2014](http://arxiv.org/abs/1412.6980), the method is "*computationally efficient, has little memory requirement, invariant to diagonal rescaling of gradients, and is well suited for problems that are large in terms of data/parameters*". For AMSGrad see [On The Convergence Of Adam And Beyond. Reddi et al., 5-8](https://openreview.net/pdf?id=ryQu7f-RZ). **If amsgrad = False**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{v_t} + \epsilon)$$ **If amsgrad = True**: initialize $m_0$ as 1st moment vector initialize $v_0$ as 2nd moment vector initialize $\hat{v}_0$ as 2nd moment vector The update rule for $\theta$ with gradient $g$ uses an optimization described at the end of section 2 of the paper: $$lr_t = \mathrm{learning\_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$ $$m_t = \beta_1 * m_{t-1} + (1 - \beta_1) * g$$ $$v_t = \beta_2 * v_{t-1} + (1 - \beta_2) * g^2$$ $$\hat{v}_t = \max(\hat{v}_{t-1}, v_t)$$ $$\theta_t = \theta_{t-1} - lr_t * m_t / (\sqrt{\hat{v}_t} + \epsilon)$$ The default value of 1e-7 for epsilon might not be a good default in general. For example, when training an Inception network on ImageNet a current good choice is 1.0 or 0.1. Note that since Adam uses the formulation just before Section 2.1 of the Kingma and Ba paper rather than the formulation in Algorithm 1, the "epsilon" referred to here is "epsilon hat" in the paper. The sparse implementation of this algorithm (used when the gradient is an IndexedSlices object, typically because of `tf.gather` or an embedding lookup in the forward pass) does apply momentum to variable slices even if they were not used in the forward pass (meaning they have a gradient equal to zero). Momentum decay (beta1) is also applied to the entire momentum accumulator. This means that the sparse behavior is equivalent to the dense behavior (in contrast to some momentum implementations which ignore momentum unless a variable slice was actually used). Usage: >>> opt = tf.keras.optimizers.legacy.Adam(learning_rate=0.1) >>> var1 = tf.Variable(10.0) >>> loss = lambda: (var1 ** 2)/2.0 # d(loss)/d(var1) == var1 >>> step_count = opt.minimize(loss, [var1]).numpy() >>> # The first step is `-learning_rate*sign(grad)` >>> var1.numpy() 9.9 Tc >>[TU]"U40UD6 URSURSU55 URSUR5 URSU5 URSU5 U=(d [ R "5UlXPlg)a*Construct a new Adam optimizer. Args: learning_rate: A `Tensor`, floating point value, or a schedule that is a `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable that takes no arguments and returns the actual value to use, The learning rate. Defaults to `0.001`. beta_1: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use. The exponential decay rate for the 1st moment estimates. Defaults to `0.9`. beta_2: A float value or a constant float tensor, or a callable that takes no arguments and returns the actual value to use, The exponential decay rate for the 2nd moment estimates. Defaults to `0.999`. epsilon: A small constant for numerical stability. This epsilon is "epsilon hat" in the Kingma and Ba paper (in the formula just before Section 2.1), not the epsilon in Algorithm 1 of the paper. Defaults to `1e-7`. amsgrad: Boolean. Whether to apply AMSGrad variant of this algorithm from the paper "On the Convergence of Adam and beyond". Defaults to `False`. name: Optional name for the operations created when applying gradients. Defaults to "Adam". **kwargs: keyword arguments. Allowed to be {`clipnorm`, `clipvalue`, `lr`, `decay`}. `clipnorm` is clip gradients by norm; `clipvalue` is clip gradients by value, `decay` is included for backward compatibility to allow time inverse decay of learning rate. `lr` is included for backward compatibility, recommended to use `learning_rate` instead. r r r rrNrrs rrNonFusedAdam.__init__`s{T (( D-)HI !4!45 &) &):."8"8":  rcUHnURUS5 M UHnURUS5 M UR(aUHnURUS5 M ggr!r%r's rr*NonFusedAdam._create_slotsr,rc @>[T U]XU5 [R"URS-U5n[R "UR SU55n[R "UR SU55n[R"XT5n[R"Xd5nX1U4S[R"SU- 5SU- - -n X1U4R[U [R"URU5UUSU- UUSU- S95 gr.r7rCs rr8NonFusedAdam._prepare_localrIrc>URn[[U5S- S- 5n[U5SU-S-:XaUS[U5n[TU]U5 grKrNrSs rrRNonFusedAdam.set_weightsrWr) jit_compilecURURRpTU=(d 0RXE45=(d UR XE5nUR US5nUR US5nUS[ R"SUS- 5-SUS- - n URX- SUS- -5 UR[ R"U5U- SUS- -5 UR(a9UR US 5n U R[ R"X55 U nURXy-[ R"U5US -- 5 g) Nr"r#r0r/r5r2r1r4r$r)r_r`rarrbrcr9r? assign_addsquarerrnrsrq) rr]r)rFrDrErir"r#alphar$s r_resource_apply_dense_impl'NonFusedAdam._resource_apply_dense_implsC # CII,@,@I#)r..  # ?  ' ' >  MM#s # MM#s #  gga,~667 8<// 1  dh1|J'?#?@A biio)a,z2J.JKL <<==f-D KK 4+ ,A  bggaj< 3J&JKLrcURXU5 [R"5(d9[RRR 5R 5S$gN)rr9executing_eagerlyrmrget_default_graphget_operations)rr]r)rFs rrj"NonFusedAdam._resource_apply_densesK '';?##%%99<<113BBDRH H&rcdURURRpeU=(d 0RXV45=(d UR XV5nUR US5nXS-n UR XS-5 UR[R"X55 UR US5n X-US-n U R XS-5 U R[R"X55 UR(d5URUSU-[R"U 5US-- 5 gUR US 5n U R [R"X55 URUSU-[R"U 5US-- 5 g) Nr"r3r1r#r6r4r rr$)r_r`rarrbrcrn scatter_addr9 IndexedSlicesrrqr?rs) rr]r)rtrFrDrErir"rur#rwr{s r_resource_apply_sparse_impl(NonFusedAdam._resource_apply_sparse_implsz # CII,@,@I#)r..  # ?  ' ' >  MM#s # 0D#EE *--. b&&'8BC MM#s #![L9M,NN *--. b&&'8BC|| NNT"Q&"''!*|I7N*NO MM#v.E LLE- . NNT"775>L$;;= rcURXX45 [R"5(d9[RRR 5R 5S$gr)rr9rrmrrr)rr]r)rtrFs rr~#NonFusedAdam._resource_apply_sparsesK ((GI##%%99<<113BBDRH H&rc >[TU]5nURURS5URURS5URS5UR UR S.5 U$rrrs rrNonFusedAdam.get_configrrrrr)rrrrrrrr*r8rRr9functionrrjrr~rrrrs@rrrsFP 0d + 0%[[T"M#M,I [[T"#>I rr)rtensorflow.compat.v2rmv2r9 tf_keras.srcrtf_keras.src.optimizers.legacyr tensorflow.python.util.tf_exportr OptimizerV2rrrrrse%!!'7:"!?@t< # #t tny<++yr